Projection system, projection system control device, projection method and program

Abstract

This projection system is provided with: a projector which projects a contents image and a pattern image; a camera which photographs a projection surface during projection of the pattern image; a camera deformation value calculation unit which, on the basis of a result of the photographing, calculates a texture deformation value indicative of a change in the position, orientation, or shape of the projection surface; a projector deformation value acquisition unit which acquires a projector deformation value for converting the contents image on the basis of the texture deformation value and the pattern image shown in the photographing result; and a transmission image acquisition unit which executes a process for deforming the contents image on the basis of the projector deformation value and a process for deforming the pattern image on the basis of the projector deformation value, wherein the pattern image satisfies a condition of containing multiple types of pattern-less regions of different luminance values.

Description

[Technical field]

[0001] The present invention relates to a projection system, a projection system control device, a projection method, and a program. This application claims priority on Japanese Patent Application No. 2019-156876, filed on August 29, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] Conventionally, in the projection mapping technology that makes an image follow a moving projection surface, a camera photographs the projection surface on which the image is projected, and the image (projection content) follows the movement of the projection surface based on the photographing result. Such projection mapping technology requires the alignment of the projector and camera and marking on the projection surface in advance, which makes the technology difficult for users to use. Therefore, for example, a method has been proposed in which the camera image obtained when the projection content is superimposed on the texture on the projection surface is simulated while tracking (Non-Patent Document 1). In this method, if the texture on the projection surface and the projection content on the projection surface can be tracked using the camera image, there is no need for prior alignment or marking. However, such a method requires a large amount of calculation, and it is difficult to make the content image follow the movement of the projection surface. In addition, a method has been proposed in which the projection surface is limited to a plain one and the shape of the projection surface is limited to a known one (Non-Patent Document 2), thereby making it possible to track the projection content without simulation. However, there is a problem in that the texture on the projection surface cannot be used as a clue for tracking.

[0003] In addition, both of these proposed methods have a problem that the content image must have a feature that allows the camera to track the movement of the content image. Therefore, when the projection surface is not plain, it has been proposed to hide a pattern image that is difficult for humans to perceive in the projected video, and to track the movement of the content image using the pattern image. However, this method requires the camera to capture multiple consecutive images (Patent Document 1, Non-Patent Document 3). Therefore, there is a problem that the limit at which the camera can track the movement of the projection surface depends on the shutter speed of the camera, and it does not work well when the movement of the projection surface is fast. Thus, when the projection surface is not plain, there are cases where the positional deviation on the projection surface of multiple images projected in sequence on the projection surface becomes large. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 002293 [Non-patent literature]

[0005] [Non-Patent Document 1] S. Audet, M. Okutomi, M. Tanaka, “Augmenting moving planar surfaces robustly with videoprojection and direct image alignment”, Virtual Reality, vol. 17, 2013. [Non-Patent Document 2] S. Kagami and K. Hashimoto, “Sticky projection mapping: 450-fps tracking projection onto a moving planar surface”, Proc. SIGGRAPH Asia 2015 Emerging Technologies, 2015. [Non-Patent Document 3] Ramesh Raskar, Greg Welch, Matt Cutts, Adam Lake, Lev Stesin, and Henry Fuchs “The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays”, Computer Graphics Proceedings, Annual Conference Series, 1998. Summary of the Invention [Problem to be solved by the invention]

[0006] In consideration of the above circumstances, the present invention aims to provide a technology for moving an image projected onto a projection surface in accordance with the movement of the projection surface, which suppresses deviations in the position on the projection surface of multiple images projected in sequence onto a moving projection surface. [Means for solving the problem]

[0007] One aspect of the present invention is a projection system comprising: a projector that projects a content image and a pattern image that satisfies predetermined conditions onto a projection surface that is not a plain surface; a camera that photographs the projection surface at a photographing timing when the pattern image is projected; a camera transformation value calculation unit that calculates a texture transformation value that represents a change in position, orientation or shape of the projection surface based on a camera image that is the photographing result by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image that corresponds to the state of the projection surface based on the pattern image and the texture transformation value captured in the camera image; and a transmission image acquisition unit that executes a process of image transformation of the content image based on the projector transformation value and a process of image transformation of the pattern image based on the projector transformation value, wherein the predetermined conditions include a condition that the projection surface has a plurality of types of plain areas with different brightness values.

[0008] One aspect of the present invention is a projection system control device comprising: a projector that projects a content image and a pattern image that satisfies predetermined conditions onto a projection surface that is not a plain surface; a camera that photographs the projection surface at a photographing timing when the pattern image is projected; a camera transformation value calculation unit that calculates a texture transformation value that represents a change in position, orientation or shape of the projection surface based on the camera image that is the photographing result by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image that corresponds to the state of the projection surface based on the pattern image and the texture transformation value captured in the camera image; and a transmission image acquisition unit that executes a process of image transformation of the content image based on the projector transformation value and a process of image transformation of the pattern image based on the projector transformation value, wherein the predetermined conditions include a condition that the projection system has a plurality of types of plain areas with different brightness values, and at least one of the plurality of types of plain areas has an area larger than a predetermined area.

[0009] One aspect of the present invention is a projection method comprising: a projection step of projecting a content image and a pattern image that satisfies predetermined conditions onto a projection surface that is not a plain surface; a photographing step of photographing the projection surface at a photographing timing when the pattern image is projected; a camera transformation value calculation step of calculating a texture transformation value that represents a change in position, orientation or shape of the projection surface based on a camera image that is the photographing result of the photographing step; a projector transformation value acquisition step of acquiring a projector transformation value of image transformation that converts an image projected in the projection step into an image that corresponds to the state of the projection surface based on the pattern image and the texture transformation value captured in the camera image; and a transmission image acquisition step of executing a process of image transformation of the content image based on the projector transformation value and a process of image transformation of the pattern image based on the projector transformation value, wherein the predetermined condition includes a condition that the projection surface has a plurality of types of plain areas with different brightness values.

[0010] One aspect of the present invention is a program for causing a computer to function as the above-described projection system.

[0011] One aspect of the present invention is a program for causing a computer to function as the above-mentioned projection system control device. Effect of the Invention

[0012] The present invention makes it possible to provide a technology for moving an image projected onto a projection surface in accordance with the movement of the projection surface, which suppresses the shift in position on the projection surface of multiple images projected in sequence onto a moving projection surface. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing an example of the system configuration of a projection system 100 according to a first embodiment. [Diagram 2] 5 is a timing chart showing an example of an order in which a content image and a pattern image are projected and an example of a predetermined image capturing timing in the first embodiment. [Diagram 3] FIG. 4 is a schematic diagram showing an example of information indicating a history of a deformation value in the first embodiment. [Figure 4] FIG. 1 is a first diagram showing an example of a pattern image in the first embodiment. [Diagram 5] FIG. 4 is a second diagram showing an example of a pattern image in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of the functional configuration of a control unit 30 in the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a target location in the first embodiment. [Figure 8] 5A and 5B are diagrams showing examples of experimental results illustrating a pattern image before acquiring a projector matrix in the first embodiment and a pattern image after performing homography transformation represented by the projector matrix. [Figure 9] 5 is a flowchart showing an example of the flow of processing executed by the projection system 100 in the first embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a system configuration of a projection system 100 according to a second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of the functional configuration of a control unit 30a in a second embodiment. [Figure 12] FIG. 11 is an explanatory diagram for explaining a first control point and a second control point in the second embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of processing executed by the projection system 100a in the second embodiment. [Figure 14] 11 is a first timing chart showing an example of an order in which a content image and a pattern image are projected and an example of a predetermined image capturing timing in a modified example. [Figure 15] 13 is a second timing chart showing an example of the order in which the content image and the pattern image are projected and an example of the predetermined image capturing timing in the modified example. [Figure 16] FIG. 13 is a diagram showing an example of a transmission image acquisition unit 308 in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] (First embodiment) FIG. 1 is a diagram showing an example of a system configuration of a projection system 100 according to a first embodiment. The projection system 100 projects a plurality of still images at a predetermined frame rate while switching between them onto a projection surface 90 that can be moved, rotated or deformed. The frame rate is the number of still images projected onto the projection surface 90 per unit time. The frame rate is, for example, 2400 fps (frames per second). The projection system 100 projects at least two types of still images onto the projection surface 90. One of the still images is a content image and the other is a pattern image. The surface of the projection surface 90 is not plain. Hereinafter, the pattern on the surface of the projection surface 90 is referred to as an object texture.

[0015] The pattern image is an image projected onto the projection surface 90 for correcting the content image according to changes in the position, orientation, or shape (hereinafter referred to as the "three-dimensional state") of the projection surface 90. Specifically, the pattern image is an image that satisfies the following first pattern image condition. The first pattern image condition includes a condition that the image has a plurality of types of plain areas with different luminance values, and at least one of the areas of the plurality of types of plain areas is larger than a predetermined area. The predetermined area is, for example, an area that is determined in advance according to the speed of change in the position, orientation, or shape of the projection surface 90 and the object texture. The predetermined area may be, for example, the area of ​​a surface that satisfies the first backprojection condition. The first backprojection condition is a condition that the shape is a rectangle in which the product of the backprojection vector and the frame time is equal to the length of the diagonal. The backprojection vector is a vector obtained by backprojecting the motion vector of the projection surface 90 onto the pattern image. The predetermined area may also be, for example, the area of ​​a surface that satisfies the second backprojection condition. The second backprojection condition is a condition that the area is equal to the square of the product of the backprojection vector and the frame time. In the case where at least one of the plain areas is elongated in one direction, the predetermined area may be, for example, the area of ​​a surface that satisfies the third backprojection condition. The third backprojection condition is a condition that the area is approximately 0. The predetermined area may be, for example, the area of ​​a surface that satisfies the fourth backprojection condition. The fourth backprojection condition is a condition that a plurality of plain areas of the pattern image are captured in an image captured by the camera 2 described later. Note that the predetermined area does not necessarily have to be the area of ​​a surface that satisfies only one of the first backprojection condition, the second backprojection condition, the third backprojection condition, or the fourth backprojection condition. The predetermined area may be, for example, the area of ​​a surface that satisfies more than one of the first backprojection condition, the second backprojection condition, the third backprojection condition, or the fourth backprojection condition. The predetermined area may be, for example, the area of ​​a surface that satisfies the third backprojection condition and the fourth backprojection condition.

[0016] Since the pattern image satisfies the first pattern image condition, the amplitude spectrum of the pattern image contains many high frequency components. Therefore, changes in the position, orientation, or shape of the pattern image are easier to detect than in an image that does not satisfy the first pattern image condition. A more detailed description of the pattern image will be provided later.

[0017] The content image is an image different from the pattern image that the user of the projection system 100 wants the spectators to see. Therefore, the content image is projected onto the projection surface 90 more times during a unit time than the pattern image. Since the content image is projected onto the projection surface 90 more times than the pattern image, the user can easily see the content image than the pattern image. The content image is projected, for example, 5 / 6 times the frame rate during a unit time, and the pattern image is projected, for example, 1 / 6 times the frame rate during a unit time. The multiple content images projected during a unit time do not necessarily have to all be the same. For example, they may be multiple different images that are seen by the spectators as the content image that the user of the projection system 100 wants the spectators to see due to the persistence of vision effect.

[0018] When the three-dimensional state of the projection surface 90 changes, the projection system 100 projects onto the projection surface 90 a content image whose position, orientation, or shape has been changed in the same way as the projection surface 90 .

[0019] The projection system 100 includes a projector 1, a camera 2, and a control device 3. The projector 1 acquires a plurality of image data items one by one in sequence. The projector 1 projects still images represented by image data input in chronological order onto a projection surface 90. The projector 1 is, for example, a DLP (Digital Light Processing) projector equipped with an LED (Light Emitting Diode) and a DMD (Digital Micromirror Device).

[0020] The camera 2 captures the projection surface 90 at a predetermined capture timing. The camera 2 transmits image data of the captured image (hereinafter referred to as the "camera image") to the control device 3. The capture timing is the timing when the pattern image is projected onto the projection surface 90. Since the camera 2 captures the projection surface 90 with the pattern image projected, the camera image shows an image in which the pattern image is superimposed on the object texture. The capture timing may be, for example, the timing when a predetermined signal (hereinafter referred to as the "capture instruction signal") is sent from the control device 3, or may be a predetermined timing. For simplicity, the projection system 100 will be described below using as an example a case in which the camera 2 captures the projection surface 90 at the timing when the capture instruction signal is received.

[0021] FIG. 2 is a timing chart showing an example of the order in which the content image and the pattern image are projected and an example of the predetermined image capturing timing in the first embodiment. Fig. 2 shows that when a content image is projected five times, a pattern image is projected once. In Fig. 2, the timing chart of the camera trigger shows the timing at which camera 2 captures an image. When the camera trigger is "Hi", camera 2 captures the projection surface 90. When the camera trigger is "Low", camera 2 does not capture the projection surface 90.

[0022] The control device 3 transmits image data to the projector 1. The control device 3 calculates a value (hereinafter referred to as a "transformation value") for transforming (hereinafter referred to as "image transformation") the reference content image so that the content image projected on the projection surface 90 has a position, orientation, or shape according to the three-dimensional state of the projection surface 90, based on one camera image. The image transformation is, for example, homography transformation. The image transformation may be, for example, radial basis function approximation. The transformation value is, for example, a matrix representing the homography transformation. The transformation value may be, for example, a coefficient of a radial basis function. The reference content image is a content image projected on the projection surface 90 when the projection surface 90 is in a first predetermined three-dimensional state (hereinafter referred to as an "initial state"). The control device 3 transforms the reference content image by the calculated transformation value. The control device 3 transmits image data of the reference content image after the image transformation (hereinafter referred to as a "transmitted content image") to the projector 1. The control device 3 also performs image transformation on the reference pattern image by the calculated transformation value. The reference pattern image is a pattern image that is projected onto the projection surface 90 when the projection surface 90 is in an initial state. The control device 3 transmits image data of the reference pattern image after image transformation (hereinafter referred to as a "transmission pattern image") to the projector 1 in place of the transmission content image at a predetermined timing. The number of times that the control device 3 transmits image data of the transmission pattern image during a unit time is less than the number of times that the control device 3 transmits image data of the transmission content image during a unit time.

[0023] The control device 3 includes a control unit 30 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected by a bus, and executes a program. The control device 3 functions as a device including the control unit 30, communication unit 31, and storage unit 32 by executing the program. More specifically, the processor 91 reads out a program stored in the storage unit 32, and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, and the control device 3 functions as a device including the control unit 30, communication unit 31, and storage unit 32.

[0024] The communication unit 31 includes a communication interface for connecting the device to the projector 1, the camera 2, and an external device. The communication unit 31 communicates with the projector 1, the camera 2, and the external device wirelessly or via wired communication. The communication unit 31 transmits, for example, image data of a transmission content image and a transmission pattern image to the projector 1. The communication unit 31 receives, for example, image data of a camera image transmitted by the camera 2. The communication unit 31 acquires, for example, image data of a reference content image output by an external device. The communication unit 31 outputs the acquired various information to the control unit 30.

[0025] The storage unit 32 is configured using a non-transitory computer-readable storage medium such as a magnetic hard disk device or a semiconductor storage device. The storage unit 32 stores various information related to the control device 3. The storage unit 32 stores, for example, image data of a reference content image. The storage unit 32 stores, for example, image data of a reference pattern image. The storage unit 32 stores, for example, image data of a transmission pattern image. The storage unit 32 stores, for example, image data of a transmission content image. The image data of the reference content image stored in the storage unit 32 may be information previously recorded in the storage unit 32, or may be information acquired via the communication unit 31 and then recorded in the storage unit 32. The storage unit 32 stores, for example, information indicating a history of a transformation value. More specifically, the storage unit 32 stores a history of a texture transformation value, a history of a pattern transformation value, and a history of a projector transformation value, which will be described later.

[0026] FIG. 3 is a schematic diagram showing an example of information indicating a history of a deformation value in the first embodiment. FIG. 3 shows an example of information indicating a history of transformation values ​​stored in the storage unit 32. In the information indicating the history of transformation values, the items "shooting timing", "texture transformation value", "pattern transformation value", and "camera transformation value" are associated with each other. "Shooting timing" indicates the shooting timing. "Texture transformation value" indicates the texture transformation value calculated based on a camera image captured at the corresponding shooting timing. "Pattern transformation value" indicates the pattern transformation value calculated based on a camera image captured at the corresponding shooting timing. "Camera transformation value" indicates the camera transformation value calculated based on a camera image captured at the corresponding shooting timing. For example, in FIG. 3, data D101 indicates that the texture transformation value calculated based on a camera image captured at the shooting timing of T2 is Vc2, the pattern transformation value is Vp2, and the camera transformation value is Vt2.

[0027] The storage unit 32 stores a reference texture image in advance. The reference texture image is an image of the projection surface 90 in a predetermined second predetermined three-dimensional state (hereinafter referred to as the "reference state"), which is illuminated with a predetermined light that illuminates the projection surface 90 almost uniformly, captured by the camera 2. The predetermined light is, for example, a single color light such as white light emitted by the projector 1. The predetermined light may be, for example, light of indirect lighting in a room in which the projection surface 90 is located. For simplicity, the following description will be given taking as an example a case in which the predetermined light is white light emitted by the projector 1. In this way, since the reference texture image is an image of the projection surface 90 in a state in which it is illuminated almost uniformly, the reference texture image is an image showing the distribution of reflectance on the projection surface 90. Note that the reference state and the initial state may be the same or different.

[0028] (About the pattern image) Here, the pattern image will be described in detail. The pattern image may be any pattern image as long as it satisfies the first pattern image condition. When the projection surface 90 is a flat surface, it is desirable for the pattern image to satisfy, for example, the following second pattern image condition in addition to the first pattern image condition.

[0029] The second pattern image condition includes a condition that an area within a predetermined distance from the periphery (hereinafter referred to as the "outer area") has more plain areas with different brightness values ​​than areas other than the outer area.

[0030] The pattern image may further satisfy the following third pattern image condition, in addition to the first pattern image condition and the second pattern image condition.

[0031] The third pattern image condition is that in an area other than the outer area, there are multiple types of plain areas with different brightness values, one or more for each type per unit area, and the area of ​​each plain area is larger than a predetermined area corresponding to the pattern of the object texture.

[0032] When the pattern image is a polygon, the pattern image may further satisfy the following fourth, fifth, and sixth pattern image conditions in addition to the first, second, and third pattern image conditions. A pattern image satisfying such conditions is desirably used when the projection surface 90 is a flat surface.

[0033] The fourth pattern image condition includes a condition that in a unit area in an area other than the outer area, there are one or more plain areas of each type, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of type.

[0034] The fifth pattern image condition includes a condition that, in a unit area of ​​the area other than the corner area of ​​the outer area, there are one or more plain areas of each type, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of the type. The corner area is an area of ​​a predetermined size that includes a vertex of a polygon and is within the outer area.

[0035] The sixth pattern image condition includes a condition that the corner area is a plain area of ​​one type.

[0036] When the pattern image is a circle or an ellipse, the pattern image may satisfy, for example, a fourth pattern image condition and the following seventh pattern image condition in addition to the first, second, and third pattern image conditions. A pattern image satisfying such conditions is desirably used when the projection surface 90 is a flat surface.

[0037] The seventh pattern image condition includes a condition that there are one or more plain areas of each type within a unit area in the region within the outer region, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of type.

[0038] 4 and 5 are diagrams showing an example of a pattern image that satisfies the pattern image condition in the first embodiment. More specifically, Fig. 4 is a first diagram showing an example of a pattern image in the first embodiment. FIG. 4 shows the pattern image A1. FIG. 4 shows that the pattern image A1 is a square. FIG. 4 shows that the outer region of the pattern image A1 is a region with a thickness E from the outer periphery of the pattern image A1 toward the inside. FIG. 4 shows that the outer region of the pattern image A1 has white regions and black regions arranged alternately along the outer periphery of the pattern image A1. The white regions are regions with a predetermined luminance value, and the black regions are regions with a lower luminance value than the white regions. FIG. 4 shows that a square region with one side including a vertex of the square and E is a white region. In FIG. 4, the square white region with one side including a vertex of the square and E is a corner region of the pattern image A1. When the pattern image A1 is projected onto the projection surface 90, the white region is a region irradiated with light of a predetermined intensity. The black region is a region irradiated with light of an intensity weaker than the intensity of the light irradiated to the white region. The intensity of the light irradiated to the black region may be any intensity as long as it is weaker than the intensity of the light irradiated to the white region, and may be, for example, 0.

[0039] Fig. 4 shows that the white areas in the outer region of pattern image A1 are rectangles with a length of L and a thickness of E, or squares with a side length of E. The length L is longer than E. Fig. 4 shows that the black areas in the outer region of pattern image A1 are rectangles with a length of L and a thickness of E. Fig. 4 shows that in regions other than the outer region of pattern image A1, square white areas with a side length of L and square black areas with a side length of L are arranged alternately.

[0040] FIG. 5 is a second diagram showing an example of the pattern image in the first embodiment. FIG. 5 shows pattern image A2. Pattern image A2 is a pattern image in which the white and black areas in pattern image A1 shown in FIG. 4 are reversed. That is, pattern image A1 and pattern image A2 have the same shape. The sum of the luminance value of each pixel in pattern image A1 (one side) and the luminance value of each corresponding pixel in pattern image A2 (the other side) is approximately the same regardless of the pixel. In FIG. 5, the square black area with one side of E including the vertex of the square is the corner area of ​​pattern image A2. This is the end of the detailed description of the pattern image.

[0041] 6 is a diagram showing an example of the functional configuration of the control unit 30 in the first embodiment. The control unit 30 includes a reference content image acquisition unit 301, a camera transformation value calculation unit 302, a projector transformation value acquisition unit 303, a transmission content image acquisition unit 304, a transmission pattern image acquisition unit 305, and a communication control unit 306.

[0042] The reference content image acquiring unit 301 acquires image data of the reference content image. The reference content image acquiring unit 301 may read out image data of the reference content image recorded in the storage unit 32, or may acquire image data of the reference content image input via the communication unit 31.

[0043] The camera deformation value calculation unit 302 calculates a deformation value (hereinafter referred to as a "texture deformation value") representing a change in the position, orientation, or shape of the projection surface 90 as seen from the camera 2 based on the camera image every time a camera image is acquired. More specifically, the camera deformation value calculation unit 302 calculates a texture deformation value based on an image of an object texture (hereinafter referred to as an "object texture image") captured in the camera image every time a camera image is acquired. The camera deformation value calculation unit 302 may calculate the texture deformation value by any method as long as the method can calculate the texture deformation value. For example, the camera deformation value calculation unit 302 may calculate the texture deformation value by a convergence calculation in which a predetermined deformation value is used as an initial condition and the deformation value is updated in accordance with a predetermined rule. In this case, the convergence condition of the convergence calculation is a condition that the degree of agreement at a target location between the object texture image captured in the camera image and the reference texture image after image deformation is equal to or greater than a predetermined degree. The target location is a location that would be captured in the camera image if the reference texture image after image deformation were projected onto the projection surface 90. The degree of agreement is, for example, the smallness of the sum of squared differences in luminance values ​​between the object texture image and the reference texture image after image deformation. The camera deformation value calculation unit 302 may adjust the luminance of the object texture during the convergence calculation. Specifically, the luminance of the object texture may be divided by the average value of the luminance values ​​of all pixels of the reference texture image during the convergence calculation. In this way, the difference in luminance values ​​between the reference texture image and the object texture is reduced, thereby reducing the possibility that the convergence calculation will converge at an inappropriate saddle point or local solution, or will diverge without converging. The calculated texture deformation value may be recorded in the storage unit 32.

[0044] For example, when the pattern image is an image having two types of regions, a white region and a black region, the camera deformation value calculation unit 302 may calculate the texture deformation value using each position in the region that satisfies the following target region condition as the target region, instead of the entire surface of the projection surface 90. The target region condition is a condition that the white region of the previous pattern image is located in the region at the previous shooting timing. The previous pattern image is a pattern image that appears in the camera image captured at the previous shooting timing. For example, the target region does not have to be the entire position in the white region of the previous pattern image, and may be, for example, a part of the white region of the previous pattern image. More specifically, the target region may be, for example, a region in the white region of the previous pattern image, and may be a position in the region located at a position away from the outer periphery of the white region by a distance according to the speed of movement or rotation of the projection surface 90 and the interval of the shooting timing. The distance is, for example, v × Δt, where v is the speed of movement of the projection surface 90 and Δt is the interval of the shooting timing. The target region may be a black region instead of a white region, or may be a part of the black region. The target area may be a white area and a black area, or a part of the white area and a part of the black area.

[0045] At the time when the camera transformation value calculation unit 302 acquires the texture transformation value, it has not been determined where the pattern image is located in the just acquired camera image. Therefore, when the target area is the entire plain area, it is difficult to determine whether the area represented by each pixel in the target area captured in the camera image is a white area or a black area of ​​the pattern image. On the other hand, when the target area is a part of the plain area, the target area is a white area of ​​the pattern image. Therefore, the texture transformation value can be calculated more accurately than when the target area is the entire plain area.

[0046] Fig. 7 is a diagram showing an example of a target location in the first embodiment. Fig. 7 shows an example of a camera image of an experimental result. In Fig. 7, the target locations are positions within region R1. Distance d is the distance between the perimeter of region R1 and the perimeter of the white region.

[0047] The image transformation using the texture transformation value is a transformation that causes the object texture image of the target location in the camera image to match the image of the target location in the reference texture image to a degree of matching that is equal to or greater than a predetermined degree of matching, and therefore the texture transformation value indicates the correspondence between pixels in the camera image and the reference texture image.

[0048] The projector transformation value acquisition unit 303 acquires a projector transformation value based on the camera image and the texture transformation value. The projector transformation value is a transformation value that represents an image transformation for converting a reference content image into a transmission content image. For example, the projector transformation value acquisition unit 303 first executes a pattern image extraction process. The pattern image extraction process is a process for extracting a pattern image from a camera image. For example, the pattern image extraction process is the following process. The projector transformation value acquisition unit 303 first executes an image transformation on the camera image using the texture transformation value. Next, the projector transformation value acquisition unit 303 divides the luminance value of each pixel of the camera image after image transformation by the luminance value of the corresponding pixel of the reference texture image, using the correspondence between pixels indicated by the texture transformation value. The reason why a pattern image is extracted by division is as follows.

[0049] The distribution of reflectance on the projection surface 90 is superimposed on the pattern image captured in the camera image. Therefore, the luminance value of the image captured in the camera image is approximately proportional to the sum of the luminance value of the irradiated pattern image and the intensity of the indoor background light multiplied by the reflectance. The indoor background light is assumed to be approximately uniform regardless of position on the projection surface 90. On the other hand, the reference texture image is approximately proportional to the product of the intensity of the uniform light irradiated when it was acquired and the reflectance. Therefore, if the luminance value of each pixel of the camera image corresponding to the luminance value of each pixel of the reference texture image is divided by the luminance value of the corresponding pixel of the camera image, the reflectance term disappears, and the magnitude relationship between the pixels of the division result coincides with the magnitude relationship between the luminance values ​​of the pixels of the pattern image. This is the reason why the pattern image is extracted by division.

[0050] Next, the projector transformation value acquisition unit 303 acquires a transformation value representing an image transformation that makes the degree of match between the reference pattern image and the extracted pattern image equal to or greater than a predetermined degree. For example, the projector transformation value acquisition unit 303 acquires a transformation value representing an image transformation that minimizes the sum of squared differences between the luminance values ​​of the reference pattern image and the luminance values ​​of the extracted pattern image. The acquired transformation value is the pattern transformation value.

[0051] Next, the projector transformation value acquisition unit 303 acquires a projector transformation value based on the acquired pattern transformation value and the projector transformation value at the previous shooting timing. To acquire the projector transformation value, the projector transformation value acquisition unit 303 first acquires the positions of the camera reference point and the projector reference point. The camera reference point is a point where a plurality of reference points (hereinafter referred to as "pattern reference points") predefined on the pattern image are projected onto the camera image based on the pattern transformation value. The projector reference point is a point where the pattern reference point is projected onto the projector image based on the projector transformation value at the previous shooting timing. The projector image is an image projected by the projector 1. The camera reference point and the projector reference point are associated with a different projector reference point for each camera reference point in advance.

[0052] The projector deformation value acquisition unit 303 acquires a deformation value (hereinafter referred to as "target deformation value") representing an image deformation (hereinafter referred to as "target image deformation") in which the projection destination of each camera reference point coincides with the corresponding projector reference point. The projector deformation value acquisition unit 303, for example, minimizes the sum of squares of the distance between the projection destination of the camera reference point and the corresponding projector reference point by convergence calculation. The image deformation that minimizes the sum of squares is the target image deformation, and the deformation value representing the image deformation that minimizes the sum of squares is the target deformation value. The target image deformation is, for example, a homography transformation. The target image deformation may be, for example, an affine transformation for each mesh region when an image is divided into triangular meshes.

[0053] By using the image transformation from the camera image to the projector image thus obtained, the texture image appearing on the camera image can be mapped onto the projector image. The projector transformation value acquired by the projector transformation value acquisition unit 303 represents an image transformation that matches a predetermined position of the content image with a predetermined position of the texture image mapped onto the projector image. The projector transformation value acquisition unit 303 acquires the projector transformation value in the same procedure as for the target transformation value.

[0054] The transmission content image acquisition unit 304 acquires image data of the transmission content image based on the reference content image and the projector transformation value. Specifically, the transmission content image acquisition unit 304 executes image transformation represented by the acquired projector transformation value on the reference content image. The reference content image after the image transformation based on the projector transformation value is performed is the transmission content image.

[0055] The transmission pattern image acquisition unit 305 acquires image data of the transmission pattern image based on the reference pattern image and the projector transformation value. Specifically, the transmission pattern image acquisition unit 305 executes the image transformation represented by the projector transformation value on the reference pattern image. The reference pattern image after the image transformation represented by the projector transformation value is performed is the transmission pattern image.

[0056] FIG. 8 is an example of an experiment result showing a pattern image before acquiring a projector transformation value in the first embodiment and a pattern image after image transformation represented by the projector transformation value is performed. Fig. 8(A) shows the pattern image captured in the camera image before acquiring the projector transformation value. Fig. 8(B) shows the pattern image captured in the camera image after image transformation represented by the projector transformation value is performed. Fig. 8 shows that the pattern image moves in response to the movement of the projection surface 90.

[0057] The communication control unit 306 transmits the transmission content image and the transmission pattern image to the projector 1 via the communication unit 31. The communication control unit 306 transmits a shooting instruction signal to the camera 2 at the same time as transmitting the transmission pattern image.

[0058] FIG. 9 is a flowchart showing an example of the flow of processing executed by the projection system 100 in the first embodiment. The projector 1 projects still images represented by the image data input in chronological order onto the projection surface 90 in the order of input (step S101). The camera 2 captures the projection surface 90 at the capture timing (step S102). Next, the camera transformation value calculation unit 302 calculates a texture transformation value based on the camera image captured by the camera 2 (step S103). Next, the projector transformation value acquisition unit 303 acquires a pattern transformation value based on the camera image and the texture transformation value (step S104). Next, the projector transformation value acquisition unit 303 acquires a projector transformation value based on the pattern transformation value acquired in step S104 and the projector transformation value at the previous capture timing (step S105). Next, the transmission content image acquisition unit 304 acquires image data of the transmission content image based on the reference content image and the projector transformation value (step S106). Next, the transmission pattern image acquisition unit 305 acquires image data of the transmission pattern image based on the reference pattern image and the projector transformation value (step S107). Next, the communication control unit 306 transmits image data of the transmission content image and image data of the transmission pattern image to the projector 1 in a predetermined order via the communication unit 31 (step S108). The projector 1 projects the transmission content image or the transmission pattern image onto the projection surface 90 in the order in which the image data was received (step 109). After step S109, the processes from step S102 to step S109 are repeated until a predetermined end condition is satisfied. The predetermined end condition is, for example, a condition that power is not supplied to the control device 3.

[0059] The projection system 100 configured in this manner controls the content image to be projected based on the pattern image that satisfies the first pattern image condition, and therefore can project a content image according to the movement, rotation, or deformation of the projection surface 90 based on one pattern image. Therefore, in a technology in which an image projected on a projection surface is moved in accordance with the movement of the projection surface, the projection system 100 can suppress deviation of the positions of a plurality of images projected in sequence onto a moving projection surface.

[0060] Second embodiment When the performance of the projector 1 or the camera 2 included in the projection system 100 does not meet a predetermined performance, the positional deviation of the image on the projection surface becomes larger than when the performance of the device meets the predetermined performance. For example, when the lens of the projector 1 has a distortion of a predetermined magnitude or more, or when there is a delay of a predetermined time or more between when the camera 2 takes an image and when the projector 1 projects the image (hereinafter referred to as "delay time Δt"), the positional deviation becomes large. In addition, when the projection surface 90 is not flat, the positional deviation becomes large. Specifically, the positional deviation is the deviation between the virtual image and the image projected on the projection surface 90 (hereinafter referred to as "image deviation"). The virtual image is an image displayed on the projection surface 90 when the projection surface 90 is flat and the performance of the projector 1 or the camera 2 meets the predetermined performance. Therefore, in order to correct the image deviation, the control unit 30 may further include a correction unit 307.

[0061] 10 is a diagram showing an example of a system configuration of a projection system 100a according to the second embodiment. Hereinafter, components having the same functions as those in the projection system 100 are given the same reference numerals as those in FIG. 1, and description thereof will be omitted.

[0062] The projection system 100a differs from the projection system 100 in that it includes a control device 3a instead of the control device 3. The control device 3a differs from the control device 3 in that it includes a control device 30a instead of the control device 30. The control device 30a differs from the control device 30 in that it further includes a correction unit 307 in addition to the functional units included in the control device 30.

[0063] Fig. 11 is a diagram showing an example of the functional configuration of a control unit 30a in the second embodiment. Hereinafter, components having the same functions as those of the control unit 30 are given the same reference numerals as those in Fig. 6, and description thereof will be omitted.

[0064] The correction unit 307 changes the value of the projector deformation value to a value that reduces image misalignment. Each time a camera image is captured, the correction unit 307 acquires a value (hereinafter referred to as a "projector deformation correction value") for correcting the projector deformation value acquired by the projector deformation value acquisition unit 303 based on the projector deformation value at the capture timing two shots before and the projector deformation value at the capture timing one shot before. The projector deformation correction value is a value that reduces image misalignment. Specifically, the correction unit 307 acquires, as the projector deformation correction value, a value that brings four predetermined positions (hereinafter referred to as a "second control point") on the pattern image projected on the projection surface 90 closer to four predetermined positions (hereinafter referred to as a "first control point") on the projection surface 90. The projector deformation value at the capture timing two shots before is the camera deformation value acquired based on the camera image captured at the capture timing two shots before. The projector deformation value at the immediately previous shooting timing is the camera deformation value obtained based on the camera image captured at the immediately previous shooting timing.

[0065] Fig. 12 is an explanatory diagram for explaining the first control point and the second control point in the second embodiment. In Fig. 12, the arrow indicates that the second control point moves to the first control point. Fig. 12 shows that multiple second control points do not move to one first control point.

[0066] 13 is a flowchart showing an example of the flow of processing executed by the projection system 100a in the second embodiment. Hereinafter, processing similar to that executed by the projection system 100 is denoted by the same reference numerals as in FIG. 9, and description thereof will be omitted.

[0067] After step S105, the correction unit 307 acquires a projector deformation correction value based on the projector deformation value at the shooting timing two shots before and the projector deformation value at the shooting timing one shot before (step S110). Next, the correction unit 307 corrects the projector deformation value acquired in step S104 by the acquired projector deformation correction value (step S111). The correction is to execute a predetermined calculation on the projector deformation value that depends on the definition of the projector correction value. The predetermined calculation may be, for example, multiplying the projector deformation value by the projector deformation correction value, or adding the projector deformation correction value to the projector deformation value. After step S111, the process of step S105 is executed using the corrected projector deformation value.

[0068] The projection system 100a of the second embodiment thus configured acquires a projector deformation correction value and corrects the projector deformation value using the acquired projector deformation correction value. Therefore, the projection system 100a can project an image while suppressing image shifts caused by the projector 1 or the camera 2 not satisfying a predetermined performance or the projection surface 90 being not flat, more than the projection system 100.

[0069] (Modification) The transmission content image acquisition unit 304 does not necessarily need to extract the pattern image and then acquire the projector matrix based on the extracted pattern image. The transmission content image acquisition unit 304 may calculate the projector deformation value using the edge of the pattern image.

[0070] The correction unit 307 may be, for example, a Smith predictor. Furthermore, when acquiring the projector deformation correction value, the correction unit 307 does not necessarily need to use four points on the projection surface 90. It may be five or more points or less than four points.

[0071] The content image and the pattern image do not necessarily have to be one type of pattern image projected in the order shown in Fig. 2. For example, two types of pattern images having white and black areas, in which the white and black areas are inverted, may be projected consecutively once each. The two types of pattern images are, for example, the pattern image shown in Fig. 4 (hereinafter referred to as the "first pattern image") and the pattern image shown in Fig. 5 (hereinafter referred to as the "second pattern image").

[0072] FIG. 14 is a first timing chart showing an example of the order in which the content image and the pattern image are projected and an example of the predetermined image capturing timing in the modified example. Fig. 14 shows that when the content image is projected five times, the first pattern image is projected once, and then the second pattern image is projected once. Fig. 14 shows that the camera 2 captures the first pattern image, but does not capture the second pattern image. By projecting the first pattern image and the second pattern image in this manner, it appears to the user that an average image of the first pattern image and the second pattern image is projected onto the projection surface 90. Therefore, it becomes difficult for the user to visually recognize the patterns of the first pattern image and the second pattern image.

[0073] In addition, when two types of pattern images, a first pattern image and a second pattern image, are projected, the first pattern image and the second pattern image may be switched and projected each time a pattern image is projected, as shown in Figure 15 below.

[0074] FIG. 15 is a second timing chart showing an example of the order in which the content image and the pattern image are projected and an example of the predetermined image capturing timing in the modified example. 15 shows that when the content image is projected five times, the first pattern image is projected once, and then when the content image is captured five times, the second pattern image is projected once. By alternately projecting the first pattern image and the second pattern image in this way, it appears to the user that an average image of the first pattern image and the second pattern image is projected on the projection surface 90. Therefore, it becomes difficult for the user to visually recognize the patterns of the first pattern image and the second pattern image.

[0075] Note that the projection systems 100 and 100a may project, for example, a red image, a green image, and a blue image of one content image on the projection surface 90 while switching in sequence. The red image is an image that represents the red luminance value of each pixel of the content image and is projected onto the projection surface with red light. The green image is an image that represents the green luminance value of each pixel of the content image and is projected onto the projection surface with red light. The blue image is an image that represents the blue luminance value of each pixel of the content image and is projected onto the projection surface with blue light.

[0076] The first pattern image and the second pattern image may be any pattern image as long as they satisfy the following two conditions. One of the conditions is that the shapes are the same. The other condition is that the sum of the brightness value of each pixel in the first pattern image and the brightness value of the corresponding pixel in the second pattern image is approximately the same regardless of the pixel.

[0077] Even if the position, orientation, or shape of the projection surface 90 does not necessarily change, as long as the positional relationship between the projection surface 90, the projector 1, and the camera 2 changes, the projection systems 100 and 100a can project a content image onto the projection surface 90 according to the change in the positional relationship. The positional relationship between the projection surface 90, the projector 1, and the camera 2 changes, for example, when the position or orientation of the projector 1 or the position or orientation of the camera 2 changes.

[0078] The pattern image on the projection surface 90 is located within the area where the object texture is formed or drawn. The size of the pattern image on the projection surface 90 is larger than the area on the projection surface 90 that is viewed by a spectator. The content image may be a color image or a binary image.

[0079] The light emitted by projector 1 to project an image may be visible light, X-rays, ultraviolet light, infrared light, or terahertz waves. Camera 2 may be capable of receiving not only visible light, but also X-rays, ultraviolet light, infrared light, or terahertz waves.

[0080] All or part of the functions of the control devices 3 and 3a may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0081] The control devices 3 and 3a may be implemented using a plurality of information processing devices connected to each other so as to be able to communicate with each other via a network. In this case, each functional unit of the control device 3 may be distributed and implemented in a plurality of information processing devices. For example, the reference content image acquisition unit 301, the camera transformation value calculation unit 302, the projector transformation value acquisition unit 303, the transmission content image acquisition unit 304, the transmission pattern image acquisition unit 305, the communication control unit 306, and the correction unit 307 may each be implemented in a different information processing device.

[0082] The transmission content image acquisition unit 304 is an example of a first conversion unit. The transmission pattern image acquisition unit 305 is an example of a second conversion unit. The control device 3 is an example of a projection system control device.

[0083] Although the embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and designs within the scope of the gist of the present invention are also included. For example, the transmission content image acquisition unit 304 and the transmission pattern image acquisition unit 305 do not need to be different functional units. For example, the control unit 30 or 30a may include a transmission image acquisition unit 308 instead of the transmission content image acquisition unit 304 and the transmission pattern image acquisition unit 305. The transmission image acquisition unit 308 executes the process executed by the transmission content image acquisition unit 304 and the process executed by the transmission pattern image acquisition unit 305.

[0084] Fig. 16 is a diagram showing an example of a transmission image acquisition unit 308 in a modified example. For ease of explanation, Fig. 16 illustrates a control unit 30a including the transmission image acquisition unit 308, but the control unit 30 may include the transmission image acquisition unit 308.

[0085] The first pattern image condition does not necessarily include both the condition of having multiple types of plain areas with different brightness values ​​and the condition that at least one of the multiple types of plain areas has an area larger than a predetermined area. The first pattern image condition may be a condition that includes only the condition of having multiple types of plain areas with different brightness values. [Explanation of symbols]

[0086] 100, 100a...projection system, 1...projector, 2...camera, 3, 3a...control device, 30, 30a...control unit, 31...communication unit, 32...storage unit, 301...reference content image acquisition unit, 302...camera deformation value calculation unit, 303...projector deformation value acquisition unit, 304...transmission content image acquisition unit, 305...transmission pattern image acquisition unit, 306...communication control unit, 307...correction unit, 308...transmission image acquisition unit

Claims

1. a projector that projects a content image and a pattern image that satisfies a predetermined condition onto a projection surface that is not plain; a camera that captures an image of the projection surface at a capture timing that is a timing when the pattern image is projected; a camera deformation value calculation unit that calculates a texture deformation value that represents a change in a position, a direction, or a shape of the projection plane based on a camera image that is a result of shooting by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image corresponding to a state of the projection surface, based on the pattern image captured in the camera image, the texture transformation value, and a projector transformation value at the immediately previous shooting timing; a transmission image acquisition unit that executes a process of image-deforming the content image based on the projector transformation value, and a process of image-deforming the pattern image based on the projector transformation value; Equipped with the predetermined condition includes a condition that an area of ​​at least one of a plurality of plain areas having a plain area of ​​a predetermined luminance value and a plain area of ​​a luminance value lower than the predetermined luminance value is larger than a predetermined area; the predetermined area is an area according to a speed of change in a position, a direction, or a shape of the projection surface and a pattern on the surface of the projection surface; the camera deformation value calculation unit calculates the texture deformation value based on a pattern on the surface of the projection surface captured in a region captured in the camera image, the region being within the plain region and located at a position away from an outer periphery of the plain region by a distance corresponding to a speed of movement or rotation of the projection surface and an interval between timings of capturing images by the camera; Projection system.

2. The predetermined condition further includes a condition that an outer region, which is a region within a predetermined distance from the outer periphery, has more plain regions with different brightness values ​​than regions other than the outer region.

10. The projection system of claim 1 .

3. The predetermined condition further includes a condition that, in an area other than the outer area, a plurality of types of the plain area having different brightness values ​​are present per unit area, one or more for each type, and the area of ​​each of the plain areas is larger than a predetermined area corresponding to a surface pattern on the projection surface.

3. The projection system of claim 2.

4. The predetermined conditions further include a condition that the pattern image is a polygon; a condition that one or more of the plain areas of each type are present in a unit area in an area other than the outer area, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of type; a condition that an area of ​​a predetermined size within the outer area that includes a vertex of the polygon is defined as a corner area, a condition that one or more of the plain areas of each type are present in a unit area in an area other than the corner areas in the outer area, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of type; and a condition that the corner area is a plain area of ​​one type.

4. The projection system of claim 3.

5. The predetermined conditions further include a condition that the pattern image is a circle or an ellipse, a condition that one or more of the plain areas of each type are present in a unit area in the region other than the outer region, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of the type, and a condition that one or more of the plain areas of each type are present in a unit area in the region within the outer region, and the total area of ​​the plain areas of each type is approximately the same per unit area regardless of the type.

4. The projection system of claim 3.

6. the projector deformation value acquisition unit further acquires the projector deformation value based on an image showing a distribution of reflectance on the projection surface.

6. A projection system according to any one of claims 1 to 5.

7. Two types of pattern images having the same shape are projected onto the projection surface, and a sum of a luminance value of each pixel in one pattern image and a luminance value of a corresponding pixel in the other pattern image is substantially the same regardless of the pixel.

7. A projection system according to any one of the preceding claims.

8. a correction unit which regards an image displayed on the projection surface when the projection surface is a plane and the projector 1 or the camera 2 satisfies a predetermined performance as a virtual image, regards a deviation between the virtual image and the image projected by the projector as an image deviation, and corrects the projector deformation value acquired by the projector deformation value acquisition unit so as to reduce the image deviation based on a projector deformation value acquired based on a camera image captured at the shooting timing two shots ago and a projector deformation value acquired based on a camera image captured at the shooting timing one shot ago; The projection system of claim 1 , further comprising:

9. a projector that projects a content image and a pattern image that satisfies a predetermined condition onto a projection surface that is not plain; a camera that captures an image of the projection surface at a capture timing that is a timing when the pattern image is projected; a camera deformation value calculation unit that calculates a texture deformation value that represents a change in a position, a direction, or a shape of the projection plane based on a camera image that is a result of shooting by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image corresponding to a state of the projection surface, based on the pattern image captured in the camera image, the texture transformation value, and a projector transformation value at the immediately previous shooting timing; a transmission image acquisition unit that executes a process of image-deforming the content image based on the projector transformation value, and a process of image-deforming the pattern image based on the projector transformation value; Equipped with the predetermined condition includes a condition that an area of ​​at least one of a plurality of plain areas having a plain area of ​​a predetermined luminance value and a plain area of ​​a luminance value lower than the predetermined luminance value is larger than a predetermined area; the predetermined area is an area according to a speed of change in a position, a direction, or a shape of the projection surface and a pattern on the surface of the projection surface; the camera deformation value calculation unit calculates the texture deformation value based on a pattern on the surface of the projection surface captured in a region captured in the camera image, the region being within the plain region and located at a position away from an outer periphery of the plain region by a distance corresponding to a speed of movement or rotation of the projection surface and an interval between timings of capturing images by the camera; Projection system control device.

10. a projection step of projecting the content image and a pattern image that satisfies a predetermined condition onto a projection surface having a non-plain surface; an imaging step of imaging the projection surface with a camera at an imaging timing when the pattern image is projected; a camera deformation value calculation step of calculating a texture deformation value representing a change in a position, a direction, or a shape of the projection surface based on a camera image that is a result of the photographing step; a projector transformation value acquisition step for acquiring a projector transformation value of an image transformation for converting the image projected in the projection step into an image corresponding to the state of the projection surface, based on the pattern image captured in the camera image, the texture transformation value, and a projector transformation value at the immediately previous shooting timing; a transmission image acquisition step of executing a process of image-deforming the content image based on the projector deformation value, and a process of image-deforming the pattern image based on the projector deformation value; having the predetermined condition includes a condition that an area of ​​at least one of a plurality of plain areas having a plain area of ​​a predetermined luminance value and a plain area of ​​a luminance value lower than the predetermined luminance value is larger than a predetermined area; the predetermined area is an area according to a speed of change in a position, a direction, or a shape of the projection surface and a pattern on the surface of the projection surface; the camera transformation value calculation step calculates the texture transformation value based on a pattern on the surface of the projection surface captured in a region captured in the camera image, the region being within the plain region and located at a distance away from an outer periphery of the plain region by a distance corresponding to a speed of movement or rotation of the projection surface and an interval between image capture timings by the camera; Projection method.

11. A computer including a projector that projects a content image and a pattern image that satisfies a predetermined condition onto a projection surface having a non-plain surface, and a camera that captures an image of the projection surface at a capture timing that is a timing when the pattern image is projected, a camera deformation value calculation unit that calculates a texture deformation value that represents a change in a position, a direction, or a shape of the projection plane based on a camera image that is a result of shooting by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image corresponding to a state of the projection surface, based on the pattern image captured in the camera image, the texture transformation value, and a projector transformation value at the immediately previous shooting timing; a transmission image acquisition unit that executes a process of image-deforming the content image based on the projector transformation value, and a process of image-deforming the pattern image based on the projector transformation value; Equipped with the predetermined condition includes a condition that an area of ​​at least one of a plurality of plain areas having a plain area of ​​a predetermined luminance value and a plain area of ​​a luminance value lower than the predetermined luminance value is larger than a predetermined area; the predetermined area is an area according to a speed of change in a position, a direction, or a shape of the projection surface and a pattern on the surface of the projection surface; the camera deformation value calculation unit calculates the texture deformation value based on a pattern on the surface of the projection surface captured in a region captured in the camera image, the region being within the plain region and located at a position away from an outer periphery of the plain region by a distance corresponding to a speed of movement or rotation of the projection surface and an interval between timings of capturing images by the camera; A program to operate as a projection system.

12. A computer including a projector that projects a content image and a pattern image that satisfies a predetermined condition onto a projection surface having a non-plain surface, and a camera that captures an image of the projection surface at a capture timing that is a timing when the pattern image is projected, a camera deformation value calculation unit that calculates a texture deformation value that represents a change in a position, a direction, or a shape of the projection plane based on a camera image that is a result of shooting by the camera; a projector transformation value acquisition unit that acquires a projector transformation value of an image transformation that converts an image projected by the projector into an image corresponding to a state of the projection surface, based on the pattern image captured in the camera image, the texture transformation value, and a projector transformation value at the immediately previous shooting timing; a transmission image acquisition unit that executes a process of image-deforming the content image based on the projector transformation value, and a process of image-deforming the pattern image based on the projector transformation value; Equipped with the predetermined condition includes a condition that an area of ​​at least one of a plurality of plain areas having a plain area of ​​a predetermined luminance value and a plain area of ​​a luminance value lower than the predetermined luminance value is larger than a predetermined area; the predetermined area is an area according to a speed of change in a position, a direction, or a shape of the projection surface and a pattern on the surface of the projection surface; the camera deformation value calculation unit calculates the texture deformation value based on a pattern on the surface of the projection surface captured in a region captured in the camera image, the region being within the plain region and located at a position away from an outer periphery of the plain region by a distance corresponding to a speed of movement or rotation of the projection surface and an interval between timings of capturing images by the camera; A program for operating as a projection system control device.

Images