Projection image control device, method and program

Abstract

To provide a projection video control apparatus, method and program which enable high-precision alignment without depending on size of an opening of an invisible light marker or an optical sensor.SOLUTION: A projection object 10 includes: a plurality of illuminance sensors 102 for sensing illuminance of irradiation light; an optical mask 101 for blocking a part of the irradiation light incident on a light receiving unit of each illuminance sensor 102; and a sensor read unit 103 which reads sensor signals output from the illuminance sensors 102 and converts them into illuminance values. A video projection control unit 20 includes a first position identifying unit 201, a second position identifying unit 202, and a control unit 203, specifies, in stages, positions of the illuminance sensors 102 in a pixel coordinate system controlling a projection video, based on the illuminance values output by the sensor read unit 103, and perform various controls for keystone correction or dimming correction of a specific region on a video projected by a video projection unit 30 on the basis of the positions of the illuminance sensors 102. The video projection unit 30 projects corrected video data on a video screen for visualization.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a projection image control device, method, and program for projecting an image from a projector onto a predetermined area of ​​a projection object with high accuracy. [Background technology]

[0002] In recent years, projectors have become smaller and more powerful, and their usage scenarios are expanding. In an environment where the projector faces the projection object (screen) directly and both are fixed, there is little need for automatic control of the projected image. However, when at least one of the projector and the projection object moves, when you want to control the image according to the movement of a moving object within the projection area, or when you want to project the image accurately onto a specific area within the projection area, a system for controlling the projected image is required.

[0003] Patent document 1 discloses a technology in which a light transmission hole and a light sensor are installed in the housing of a projector, and the angle of the optical axis relative to the screen is estimated by detecting reflected light from the screen with the light sensor, thereby automatically performing keystone correction.

[0004] Patent document 2 discloses a technology that automatically detects a presenter in front of the screen by installing optical sensors in multiple places on the screen and protects the presenter's eyes by reducing the amount of light projected near the presenter's eyes.

[0005] Patent Document 3 discloses a technique for using a three-dimensional sensor to detect a flat area suitable for a screen in a three-dimensional space and automatically controlling the projection area.

[0006] Patent Document 4 discloses a technique in which a plurality of optical sensors are provided on a screen, and an alignment pattern is projected from a projector onto the optical sensors, thereby accurately aligning a projected image at a predetermined position.

[0007] Patent Document 5 discloses a technology in which multiple reflective materials are placed on a screen, the reflected light is captured by a camera, and the movement of the screen is detected to cause the projected image to follow the movement.

[0008] Patent Document 6 discloses a technology in which a plurality of invisible light markers are placed on a moving object, the invisible light markers are detected by a sensor, the position of the moving object is detected, and a projected image is made to track the moving object.

[0009] Patent Document 7 discloses a technique for accurately detecting the position of an optical sensor that is installed to automatically adjust the display brightness of a projected image, and for allowing an image for evaluation to enter the optical sensor with high accuracy. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2005-159426 A [Patent Document 2] JP 2013-25014 A [Patent Document 3] JP 2017-73705 A [Patent Document 4] Japanese Patent Application Publication No. 03-259188 [Patent Document 5] JP 2019-102931 A [Patent Document 6] JP 2011-254411 A [Patent Document 7] JP 2009-175355 A Summary of the Invention [Problem to be solved by the invention]

[0011] In any of the above conventional techniques, it is difficult to project an image from a projector onto a predetermined area (projection target area) of a projection target with high accuracy.

[0012] For example, the projection image control systems disclosed in Patent Documents 1 to 3 are all capable of automatically controlling projected images (keystone correction, dimming specific areas, and projection control onto flat areas), but they are not technologies that can be applied to applications such as high-precision image projection onto a target projection area.

[0013] The projection image control systems disclosed in Patent Documents 4 to 7 all aim to project an image with high accuracy onto a projection target area or an evaluation optical sensor, but the accuracy of the projection control is insufficient.

[0014] For example, in Patent Document 4, an optical sensor is used to align the projection target area on the screen, and the alignment accuracy depends on the size of the aperture of the optical sensor. In this method, the alignment accuracy is increased by making the aperture as small as possible within the range where the alignment light pattern can be detected. However, the smaller the aperture is made, the less light is transmitted, making it more difficult to detect the light pattern, so there is a limit to how small the aperture can be (Patent Document 4 assumes the aperture size to be 5 mm square), and the alignment accuracy could not be increased sufficiently.

[0015] The same is true for the projection image control technologies disclosed in Patent Documents 5 and 6, which align the projection target area by placing a reflective material or invisible light marker on the screen and capturing the reflected light with a sensor such as a camera. However, the accuracy of alignment depends on the size of the reflective material or invisible light marker, and the smaller the reflective material or invisible light marker is, the more difficult it is to detect the reflective material or invisible light marker itself, so it was not possible to achieve a sufficiently high accuracy of alignment.

[0016] The same is true of the projection image control technology disclosed in Patent Document 7, in which the position detection accuracy of the optical sensor depends on the size of the aperture of the optical sensor, and therefore it is not possible to achieve a sufficiently high alignment accuracy.

[0017] An object of the present invention is to provide a projection image control device, method, and program that solves the above technical problems and enables highly accurate alignment regardless of the size of the opening of the invisible light marker or the optical sensor. [Means for solving the problem]

[0018] In order to achieve the above-mentioned object, the present invention provides a projection image control device that provides a plurality of illuminance sensors on a projection object, whose relative positions with respect to a projection object area are known, and controls the projected image based on the position of each illuminance sensor specified in the pixel coordinate system of the projected image, and is characterized in that it has the following configuration.

[0019] (1) Equipped with an optical mask that blocks incident light to each illuminance sensor with a predetermined mask pattern, and a means for identifying the position of each illuminance sensor in a pixel coordinate system based on the output of each illuminance sensor when a mask pattern image having the same pattern structure as the mask pattern is scanned on a projection target object.

[0020] (2) The mask pattern has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the ratio of change in the amount of light transmission to change in the relative position of each pattern is greater in ranges where the amount of deviation of each pattern is relatively small than in ranges where the amount of deviation is large.

[0021] (3) The means for identifying the position of each illuminance sensor includes a first position identification means for identifying a rough position of each illuminance sensor in a pixel coordinate system, a means for calculating a transformation matrix for converting the relative position of each illuminance sensor on the projection object to the roughly identified position of each illuminance sensor, a means for deforming the mask pattern image with the transformation matrix, and a second position identification means for identifying the position of each illuminance sensor based on a change in output of each illuminance sensor when the deformed mask pattern image is scanned on the projection object.

[0022] Furthermore, the present invention can be realized not only as a projection image control device having such a characteristic configuration, but also as a projection image control method having a procedure that includes such a characteristic configuration, and further as a projection image control program that causes a computer to execute such a procedure. Effect of the Invention

[0023] According to the present invention, the following effects can be achieved.

[0024] (1) Since a mask pattern image having the same pattern structure as the mask pattern of the optical mask that blocks the light incident on the illuminance sensor is scanned onto the projection target, the sensor output value of the illuminance sensor can be made significantly different between scanning positions where the mask patterns match and scanning positions where they do not match, regardless of the size of the invisible light marker or the aperture of the optical sensor. Therefore, the position of the illuminance sensor in the pixel coordinate system can be accurately identified.

[0025] (2) The mask pattern of the optical mask has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission with respect to the change in the relative position of each pattern is greater in a range where the amount of deviation of each pattern is relatively small than in a range where the amount of deviation is large, so that the sensor output value of the illuminance sensor can be made to differ specifically and significantly between scanning positions where each mask pattern coincides and scanning positions where it does not coincide. This makes it possible to more accurately identify the position of the illuminance sensor in the pixel coordinate system.

[0026] (3) A first position specifying means and a second position specifying means are provided, and the first position specifying means is roughly within a wide range. N While performing a position search, the second position specifying means performs a precise position search only in the vicinity of the rough position specified by the first position specifying means, so that the position of each illuminance sensor can be specified precisely in a short period of time. Moreover, in this embodiment, the second position identification means projects a mask pattern image that has been deformed by homography transformation so as to cancel out the distortion, so that it becomes possible to detect the coincidence of the central coordinates of the optical mask with the true value as a peak in the illuminance value. [Brief description of the drawings]

[0027] [Figure 1] 1 is a functional block diagram of a projection image control device according to an embodiment of the present invention; [Diagram 2]13 is a diagram showing the relative positional relationship between a projection target area R1 and a plurality of illuminance sensors and their optical masks. [Diagram 3] FIG. 13 is a diagram showing an example of an optical mask. [Figure 4] 11 is a diagram showing the relationship between the amount of light transmission and the amount of shift of a projected mask pattern image relative to a mask pattern of an optical mask. [Diagram 5] FIG. 4 is a functional block diagram of a sensor reading unit. [Figure 6] 1 is a diagram showing the relationship between a pixel coordinate system that controls a projected image and a projection target object. [Figure 7] 10 is a flowchart showing the operation of an image projection control unit. [Figure 8] 13 is a diagram showing an example of an image projected to roughly identify the position of an illuminance sensor. FIG. [Figure 9] 13 is a diagram showing an example of a pattern structure image in which a mask pattern image is arranged at a corresponding position of each illuminance sensor, which is projected in order to precisely specify the position of the illuminance sensor. FIG. [Figure 10] FIG. 13 is a diagram showing a method for calculating a homography matrix that converts the position of each light mask on the projection object into each rough position specified in a pixel coordinate system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a functional block diagram showing the configuration of the main parts of a projection image control device according to an embodiment of the present invention, and the main components are a projection object 10, an image projection control unit 20, and an image projection unit 30.

[0029] The projection object 10 is equipped with a plurality of illuminance sensors 102 that detect the illuminance of the illuminance of the illuminance light, an optical mask 101 that blocks a part of the illuminance light incident on the light receiving section of each illuminance sensor 102, and a sensor reading section 103 that reads the sensor signal output by each illuminance sensor 102 and converts it into an illuminance value. Each illuminance sensor 102 is configured integrally with or separately from an image screen or the like that visualizes the projected image.

[0030] The image projection control unit 20 is equipped with a first position identification unit 201, a second position identification unit 202, and a control unit 203, and based on the illuminance value output by the sensor reading unit 103, it gradually identifies the position of each illuminance sensor 102 in a pixel coordinate system that controls the projected image, and based on the position of each illuminance sensor 102, performs various controls such as geometric transformations such as keystone correction and dimming correction of specific areas on the image projected by the image projection unit 30 onto the projection object 10.

[0031] Such an image projection control unit 20 can be configured by implementing applications (programs) that realize the functions described below in detail on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

[0032] The image projection unit 30 visualizes the corrected image data by projecting it onto an image screen. As the image projection unit 30, any general-purpose projector device such as DLP (Digital Light Processing), liquid crystal, laser, etc. can be used.

[0033] 2 is a diagram showing the relative positional relationship between a projection target area R1 onto which an image for viewing is irradiated on the projection target object 10 and a plurality of illuminance sensors 102. Note that in this embodiment, the optical mask 101 is provided on the light receiving portion of the illuminance sensor 102, and the position of each optical mask 101 as viewed from the image projection unit 30 corresponds to the position of each corresponding illuminance sensor 102. Therefore, for convenience, the position of each optical mask 101 may be regarded as the position of each illuminance sensor 102 in the description.

[0034] In this embodiment, four illuminance sensors 102 and their optical masks 101 are fixed in positions within a gap area outside the projection target area R1, within the physically projectable area on the image screen, with a known relative relationship to the projection target area R1.

[0035] Each illuminance sensor 102 detects light transmitted through the optical mask 101 and outputs a sensor signal such as a current value according to the intensity of the light to the sensor reading unit 103. Each illuminance sensor 102 can be implemented using any device that converts light into a sensor signal and outputs it, generally called a photodiode or phototransistor, or a device such as a photoresistor whose electrical resistance decreases according to the intensity of light.

[0036] In this embodiment, when a mask pattern image L having the same pattern structure as the mask pattern is projected at the position of the optical mask 101, the pattern structure of each mask pattern is designed so that the smaller the positional deviation of each pattern, the greater the amount of light transmitted.

[0037] 3 is a diagram showing an example of a mask pattern of the optical mask 101 in this embodiment, where black parts represent masked areas that do not transmit light and white parts represent non-masked areas that transmit light. In the area of ​​each optical mask 101 corresponding to the light receiving surface of each illuminance sensor 102, a plurality of masked areas and a plurality of non-masked areas are arranged in a predetermined pattern structure.

[0038] In this embodiment, in order to accurately identify the position of each illuminance sensor 102 in the pixel coordinate system based on the output of each illuminance sensor 102 when a mask pattern image having the same pattern structure as the mask pattern of the optical mask 101 is scanned on the projection target, the mask pattern of the optical mask 101 has a pattern structure in which the amount of light transmission is sensitive to positional deviation from the mask pattern image.

[0039] More specifically, as will be described in detail later with reference to Figure 4, the mask pattern of the optical mask 101 has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission relative to the change in the relative position of each pattern is greater in ranges where the amount of deviation of each pattern is relatively small than in ranges where the amount of deviation is large.

[0040] In Fig. 1A, a plurality of rectangular masked and non-masked areas are arranged in a staggered pattern. In other words, a plurality of non-masked areas are discontinuously arranged in a masked area corresponding to the light receiving surface of the illuminance sensor 102.

[0041] In Fig. 1B, a plurality of rectangular masked and non-masked areas are irregularly arranged. In other words, the non-masked areas are arranged in a manner that allows them to be interconnected and integrated into the masked area corresponding to the light receiving surface of the illuminance sensor 102, resulting in the configuration of a plurality of complex non-masked areas.

[0042] In Fig. 1C, multiple annular masked and non-masked regions of different sizes are alternately arranged in a concentric pattern. Note that the shape of each non-masked region is not limited to a rectangle or annular shape, and may be any shape.

[0043] Figures 4(a)-(c) are three-dimensional graphs calculated to show the relationship between the amount of shift of the mask pattern image relative to the mask pattern of each optical mask 101 and the amount of light transmission. For comparison, Figure 4(d) shows the relationship in the case of a conventional optical mask adopted in Patent Document 4, that is, a simple pattern structure in which one non-masked area is surrounded by one masked area.

[0044] When an optical mask 101 having a complex mask pattern and a pattern structure sensitive to misalignment is used as in this embodiment, the amount of light transmitted becomes specifically high and sharp in the range where the misalignment between the mask pattern and the mask pattern image is small, compared to when an optical mask having a simple mask pattern and a pattern structure insensitive to misalignment is used as in the same embodiment (d), and it is found that highly accurate position detection is possible. If the effects of ambient light, measurement noise, and mismatch between the angle of the projected image and the optical mask are ignored, it is expected that the illuminance value will ideally be sensed as a two-dimensional graph of a similar shape.

[0045] In addition, in a regular mask pattern as shown in Fig. 1(a), multiple maximum values ​​occur, and the influence of ambient light and measurement noise may cause erroneous detection of the peak position. In contrast, by arranging multiple masked and non-masked areas irregularly as shown in Fig. 1(b), it is possible to reduce the illuminance value when a pattern image is projected at an incorrect position, enabling highly accurate and robust position detection.

[0046] Such an irregular mask pattern can be generated by dividing the area corresponding to the light receiving part of the illuminance sensor 102 into a two-dimensional grid, generating pseudo-random numbers such as M-sequence codes corresponding to the grid numbers, and designating grids with a code of 1 as non-masked areas.

[0047] Here, for example, an M-sequence code with a length equal to the number of columns in a two-dimensional grid may be generated and assigned to the first row, and codes obtained by shifting the code by a few bits at a time may be assigned to the second and subsequent rows of the grid. In this case, the rows and columns may be reversed. This makes it possible to impart the properties of the M-sequence code not only in the row (column) direction but also in the column (row) direction.

[0048] Generally, the autocorrelation of an M-sequence code reaches a peak value when the amount of deviation is 0 and is low otherwise. Therefore, compared to the case where general pseudorandom numbers are used, the illuminance value when an image is projected at an incorrect position is reduced, enabling more accurate and robust position detection.

[0049] Furthermore, by adopting a concentric mask structure as shown in FIG. 1(c), it is possible to reduce the illuminance value when there is a positional misalignment, as in the case of the irregular pattern in FIG. 1(b). In addition, the effect on the illuminance value caused by the mismatch in angle between the projected image and the optical mask 101 is reduced, making it possible to perform position detection that is robust against the mismatch in angle between the projected image and the optical mask 101.

[0050] It is desirable that the size of the range in which the non-masked regions are arranged in optical mask 101 be approximately the same as the size of the light receiving portion of illuminance sensor 102 (for example, 5 mm square) or smaller. This allows illuminance sensor 102 to detect all light that has passed through optical mask 101, making it possible to detect the peak illuminance value when the most light has passed through the optical mask.

[0051] The sensor reading unit 103 reads the sensor signals output by each illuminance sensor 102, converts them into illuminance values, and outputs them to the image projection control unit 20. Fig. 5 is a functional block diagram of the sensor reading unit 103, and here, a case will be described where the illuminance sensor 102 is a photodiode as an example.

[0052] The sensor reading unit 103 converts analog sensor signals such as current values ​​at time t output by each illuminance sensor 102 into digital signals, converts them into corresponding illuminance values, corrects errors, and outputs the illuminance values ​​L of each illuminance sensor 102. 1,t ,L 2,t …L K,t to the image projection control unit 20 in sequence.

[0053] The image projection control unit 20 mainly includes the first position identification unit 201, the second position identification unit 202, and the control unit 203, and detects the illuminance value L of each illuminance sensor 102 at time t obtained from the sensor reading unit 103. 1,t ,L 2,t …L K,t The position of each optical mask 101 is specified in stages based on the above, and the position represents the position of each illuminance sensor 102. Furthermore, a projection target area R1 is specified based on the specified position of each illuminance sensor 102, and the image projected by the downstream image projection unit 30 and the projection target area R1 are controlled.

[0054] As shown in FIG. 6, when the projection target region R1 of the projection target object 10 is a rectangular region, K (K≧3: 4 in this embodiment) illuminance sensors 102 are arranged near the periphery of the rectangular region, and the optical mask 101 is fixedly arranged on the same plane as the projection target region R1, the central coordinate P M,1 ,P M,2,P M,3 ,P M,4 and the coordinates P of the four corners that define the projection target area R1 T,1 ,P T,2 ,P T,3 ,P T,4 can be defined in the pixel coordinate system (u, v) of the image projected by the image projection unit 30.

[0055] Then, the center coordinates P M,1 ~P M,4 The coordinates P of the four corners of the projection target area R1 T,1 ~P T,4 The homography matrix H MT By using the homography matrix H, it is possible to identify the projection target region R1 from the central coordinates of each optical mask 101. Note that the optical mask 101 and the projection target region R1 have a known positional relationship, so MT In other words, the central coordinates P of each optical mask 101 can be calculated in advance. M,1 ,P M,2 ,P M,3 ,P M,4 By specifying the projection target region R1, the projection target region R1 can be specified.

[0056] Therefore, in this embodiment, the position of each optical mask 101 is taken to represent the position of each illuminance sensor 102, the position of each optical mask 101 in the pixel coordinate system is identified, and the position of the projection target region R1 in the pixel coordinate system is identified based on the position of each optical mask 101.

[0057] Next, the operation of the image projection control unit 20 will be described in detail with reference to the flowchart of Fig. 7. In step S1, the first position identification unit 201 identifies a specific coordinate (u m ,v m ) is a circular area of ​​a predetermined radius around the center of the projection target 10, and a spot light having white (maximum brightness) is projected onto the projection target 10. Then, the illuminance value L 1,t ~L 4,tThe coordinates of each scanning position of the spot light where each of the maximum values ​​is reached are roughly expressed as P ' M,1 ~P ' M,4 Identify as:

[0058] Alternatively, as a method that can be expected to be executed in a shorter time, a method of scanning a white straight line having a predetermined line width in the u direction and v direction instead of a circular area may be adopted.Other methods that can be adopted include dividing the image projection area into a grid, assigning an ID to each grid, and specifying each optical mask position by switching between black and white image projection with a specific pattern according to the ID, or projecting a sine wave pattern with multiple phase shifts using a method called a phase shift method that is generally used for distance measurement, and estimating the phase of the sine wave with sub-pixel accuracy from the illuminance value.

[0059] The position identification process of the optical mask 101 by the first identification unit 201 alone can identify the position of the illuminance sensor with a certain degree of accuracy. However, in either method, the change in the illuminance value when the projected image is slightly shifted from the true value at the center of the mask is similarly small, and it is difficult to identify the position of the optical mask with robustness against measurement noise and with high accuracy. Therefore, in this embodiment, the rough position P ' Based on this, the second position determination unit 202 performs more precise position determination.

[0060] 9, the second position specifying unit 202 stores a mask pattern image CP having the same pattern structure as each optical mask 101 at a position corresponding to the optical mask 101 provided on the light receiving section of each illuminance sensor 102 on the projection object 10, and stores a pattern structure image in which the remaining area is all black. In this embodiment, a 3×3 houndstooth pattern is assumed as the optical mask 101, and therefore each mask pattern image CP also has a 3×3 houndstooth pattern. The relative positions of each mask pattern image CP1 to CP4 in the pixel coordinate system are equal to the relative positions of each optical mask 101 on the projection object 10.

[0061] FIG. 10 shows rough positions P of the optical masks 101 on the projection target 10, which are represented by the positions of the mask pattern images CP1 to CP4, specified in a pixel coordinate system. ' 13 is a diagram showing a method for calculating a homography matrix for performing coordinate transformation to

[0062] In step S2, as shown in FIG. 10, the center coordinates p M,1 ,p M,2 ,p M,3 ,p M,4 and the rough position P of each optical mask 101 specified by the first position specifying unit 201. ' M,1 ,P ' M,2 ,P ' M,3 ,P ' M,4 Based on the point correspondence, the center coordinate p M,1 ~p M,4 , the rough position P of each corresponding optical mask 101. ' M,1 ~P ' M,4 The homography matrix H pM →P M Calculate.

[0063] In step S3, the control unit 203 applies the transformation matrix H pM →P M In the transformed pattern structure image, the pattern structure of each mask pattern image CP is also transformed by the homography transformation H pM →P M The shape of the object is changed according to its position.

[0064] In step S4, the second position specifying unit 202 locates the area of ​​each mask pattern image CP in the transformed pattern structure image at the corresponding rough position P ' M,1 ~P ' M,4The illuminance value of each illuminance sensor 102 is acquired while shifting (scanning) within a rectangular or circular area of ​​a predetermined range centered on the illuminance value L 1,t ~L 4,t The coordinates at which each of the optical masks 101 has a maximum value are used as the precise position P M,1 ~P M,4 Identify as:

[0065] The second position specifying unit 202 determines the position P of each optical mask 101 by shifting the mask pattern image CP with sub-pixel accuracy. M,1 ~P M,2 Alternatively, the illuminance value can be upsampled by applying parabolic fitting or a bicubic method using the illuminance values ​​around the maximum value of the illuminance value, and the position P M,1 ~P M,2 may be calculated with sub-pixel accuracy.

[0066] As a method for calculating a homography matrix from point correspondences, a commonly known method is to estimate initial values ​​by the Direct Linear Transform (DLT) method using at least four pairs of point correspondences, and then perform optimization by the Gauss-Newton method, the Levenberg-Marquardt method, etc. When using this method, it is necessary to use at least four optical masks 101 and detect their center coordinates.

[0067] Another known method for calculating a homography matrix from point correspondences is to estimate the external parameters of a projector by a Perspective-n-Point (PnP) method using three or four sets of point correspondences, and calculate the homography matrix by multiplying the external parameters by the internal parameters of the projector acquired in advance by calibration. When using this method, it is necessary to use at least three optical masks and detect their center coordinates. In the present invention, there is no particular limitation on the means for calculating the homography matrix, and any method can be used.

[0068] In step S5, the control unit 203 uses the homography matrix HMT to calculate a specific position P M,1 ~P M,4 The coordinates of the four corners of the projection target area R1 are P T,1 ~P T,4 If the projection target area R1 can be specified, the vertex coordinates P C,1 ~P C,4 P T,1 ~P T,4 The homography matrix H PC →P M By calculating and applying the above, it becomes possible to control and project an image to be projected over the entire projection area so that it fits within the projection target area R1.

[0069] In addition, each specific position P M,1 ~P M,4 is still considered as a rough position, and the transformation matrix homography matrix H pM →P M The second position identification unit 202 may re-calculate the transformation matrix, re-transform the pattern structure image using the re-calculated transformation matrix, and re-identify the position of each optical mask 101 using the re-transformed mask pattern image CP, repeating this process until a predetermined convergence condition is satisfied.

[0070] The control unit 203 determines the specific position P M,1 ~P M,4 The image projection unit 30 projects the image controlled by the image projection control unit 20 onto the projection object 10.

[0071] According to this embodiment, the projection target is scanned with a mask pattern image having the same pattern structure as the mask pattern of the optical mask 101 that blocks the light incident on the illuminance sensor 102, so that the sensor output value of the illuminance sensor can be made significantly different between scanning positions where the mask patterns match and scanning positions where they do not match. Therefore, the position of the illuminance sensor 102 in the pixel coordinate system can be accurately identified.

[0072] Furthermore, according to this embodiment, the mask pattern of the optical mask 101 has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission relative to the change in the relative position of each pattern is greater in a range where the amount of deviation of each pattern is relatively small than in a range where the amount of deviation is large, so that the sensor output value of the illuminance sensor can be made to differ specifically and significantly between scanning positions where the mask patterns match and scanning positions where they do not match. This makes it possible to more accurately identify the position of each optical mask 101 in the pixel coordinate system.

[0073] Furthermore, according to this embodiment, a first position specifying unit 201 and a second position specifying unit 202 are provided, and the first position specifying unit 201 roughly specifies a position within a wide range. N While performing position search, the second position identification unit 202 performs precise position search only in the vicinity of the rough position identified by the first position identification unit 201, making it possible to precisely identify the position of each optical mask 101 in the pixel coordinate system in a short period of time.

[0074] Furthermore, simply projecting a mask pattern image having the same pattern structure as the optical mask 101 makes it impossible to accurately calculate the peak illuminance value due to the mismatch in angles between the projected image and the optical mask 101. In this embodiment, however, a mask pattern image that has been deformed by homography transformation to offset distortion is projected, so that the matching of the central coordinates of the optical mask 101 with the true value can be detected as the peak illuminance value.

[0075] In the above embodiment, the position of each illuminance sensor 102 is estimated by performing homography transformation on the mask pattern image and then projecting it, but the present invention is not limited to this. The mask pattern image may be deformed to fit the optical mask 101 by physically controlling (moving) the position, direction, and angle at which the image is projected, or may be controlled to fit within the projection target area R1.

[0076] Also, in the present embodiment, one optical mask 101 is described as covering the light receiving portion of one illuminance sensor 102, but the present invention is not limited to this, and a configuration may be adopted in which multiple illuminance sensors 102 are arranged closely together, all the light receiving portions of the multiple illuminance sensors 102 are covered with one large optical mask 101, and light transmitted through the optical mask 101 is detected without omission by the light receiving portions of the multiple illuminance sensors 102. In this case, the sensor reader 103 can add up the illuminance values ​​of the multiple illuminance sensors 102 corresponding to one optical mask 101, thereby treating them as if one large illuminance sensor 102 was used.

[0077] Furthermore, in the above embodiment, it has been described that the mask patterns of all the optical masks 101 are the same, the first position identification unit 201 projects an image that scans a spot light having no pattern structure, and only the second position identification unit 201 scans an image that has the same pattern structure as the mask pattern.

[0078] However, the present invention is not limited to this, and if a large number (e.g., eight) of illuminance sensors 102 can be provided, optical masks 101a that are insensitive to pattern misalignment may be applied to four of them, and optical masks 101b that are sensitive to pattern misalignment may be applied to the remaining four.

[0079] Generally, a mask is more sensitive to pattern misalignment when the proportion of masked areas is higher than the proportion of non-masked areas, so the proportion of masked areas in optical mask 101a is made lower than the proportion of non-masked areas, and the proportion of masked areas in optical mask 101b is made higher than the proportion of non-masked areas.

[0080] The first position specifying unit 201 performs rough position specifying that prioritizes speed over accuracy by scanning a mask pattern image having the same pattern structure as the optical mask 101a for the illuminance sensor 102 masked with the optical mask 101a. The second position specifying unit 202 estimates a rough position of the optical mask 101b based on the rough position of the optical mask 101a, and performs precise position specifying that prioritizes accuracy by scanning a mask pattern image having the same pattern structure as the optical mask 101b for the illuminance sensor 102 masked with the optical mask 101b.

[0081] Furthermore, the above-described embodiment makes it possible to project images with high viewing quality, which makes it possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure and promote inclusive and sustainable industrialization." [Explanation of symbols]

[0082] 10: projection object, 20: image projection control unit, 30: image projection unit, 101: light mask, 102: illuminance sensor, 103: sensor reading unit, 201: first position determination unit, 202: second position determination unit, 203: control unit

Claims

1. A projection image control device is provided with a plurality of illuminance sensors, the relative positions of which with respect to a projection object area are known, on a projection object, and controls a projection image based on the position of each illuminance sensor specified in a pixel coordinate system of the projection image, an optical mask for blocking light incident on each of the illuminance sensors with a predetermined mask pattern; and a means for identifying the position of each illuminance sensor in a pixel coordinate system based on the output of each illuminance sensor when a mask pattern image having the same pattern structure as the mask pattern is scanned on an object to be projected.

2. The projection image control device according to claim 1, characterized in that the mask pattern has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission relative to the change in the relative position of each pattern is greater in ranges where the amount of deviation of each pattern is relatively small than in ranges where the amount of deviation is large.

3. The means for identifying the position of each of the illuminance sensors includes: a first position specifying means for specifying a rough position of each of the illuminance sensors in a pixel coordinate system; a means for calculating a transformation matrix for transforming the relative positions of the respective illuminance sensors on the projection object into the roughly determined positions of the respective illuminance sensors; means for transforming the mask pattern image with the transformation matrix; 3. The projection image control device according to claim 1, further comprising a second position identification means for identifying the position of each illuminance sensor based on a change in output of each illuminance sensor when the deformed mask pattern image is scanned on an object to be projected.

4. recalculating the transformation matrix by regarding the positions of the illuminance sensors identified by the second position identification means as rough positions; The mask pattern image is transformed again using the newly calculated transformation matrix; the second position specifying means repeats a process of specifying a position of each illuminance sensor again based on a change in output of each illuminance sensor when the newly deformed mask pattern image is scanned on a projection target until a predetermined condition is satisfied; 4. The projection image control device according to claim 3, wherein the means for controlling the projection image controls the projection image based on the newly specified positions of each illuminance sensor.

5. 4. The projection image control device according to claim 3, wherein the first position identification means identifies a rough position of each illuminance sensor based on a change in output of each illuminance sensor when an image in which a spot light is scanned on a projection target is projected.

6. the means for transforming the mask pattern image transforms, by the transformation matrix, a pattern structure image in which the same number of mask pattern images as the number of illuminance sensors are arranged at positions corresponding to the respective illuminance sensors and an area other than the mask pattern images is black; 4. The projection image control device according to claim 3, wherein the second position specifying means scans each mask pattern image portion of the deformed pattern structure image in the vicinity of an approximate position of a corresponding illuminance sensor.

7. 3. The projection image control device according to claim 1, wherein the mask pattern has a pattern structure in which a plurality of mutually separated non-masked areas are arranged within a masked area corresponding to a light receiving surface of an illuminance sensor.

8. The projection image control device as described in claim 1 or 2, characterized in that the mask pattern has a pattern structure in which a plurality of non-masked areas are arranged within a mask area corresponding to the light receiving surface of the illuminance sensor, allowing each non-masked area to be interconnected and integrated.

9. 3. The projection image control device according to claim 1 or 2, characterized in that the pattern structure of the mask pattern is any one of a first pattern structure in which a plurality of masked areas and non-masked areas are arranged in a houndstooth pattern, a second pattern structure in which a plurality of masked areas and non-masked areas are arranged irregularly, and a third pattern structure in which a plurality of annular masked areas and non-masked areas of different sizes are arranged alternately in a concentric pattern.

10. 10. The projection image control device according to claim 9, wherein in the second pattern structure, the area in which the light is masked is divided into a two-dimensional grid, and an M-sequence code corresponding to each grid number is generated, and a non-masked area is provided according to each code.

11. 11. The projection image control device according to claim 10, characterized in that in the second pattern structure, an M-sequence code having a length equal to the number of columns or rows of the two-dimensional grid is generated and assigned to the first row or column, and a code obtained by shifting the code by a predetermined number of bits is assigned to grids in the second row or column and thereafter.

12. A projection image control method in which a projection object is provided with a plurality of illuminance sensors, the relative positions of which are known with respect to a projection object area, and an optical mask for blocking light incident on the illuminance sensors, and a computer controls a projection image based on the position of each illuminance sensor specified in a pixel coordinate system of the projection image, comprising: A projection image control method characterized by identifying the position of each illuminance sensor in a pixel coordinate system based on the output of each illuminance sensor when a mask pattern image having the same pattern structure as the mask pattern of the optical mask is scanned onto a projection object.

13. The projection image control method of claim 12, characterized in that the mask pattern has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission relative to the change in the relative position of each pattern is greater in a range where the amount of deviation of each pattern is relatively small than in a range where the amount of deviation is large.

14. When identifying the position of each of the illuminance sensors in a pixel coordinate system, Identify the approximate location of each light sensor in the pixel coordinate system, calculating a transformation matrix that transforms the relative positions of each of the illuminance sensors on the projection object into the roughly determined positions of each of the illuminance sensors; Transforming the mask pattern image with the transformation matrix; 14. The projection image control method according to claim 12, wherein the position of each illuminance sensor is specified based on a change in output of each illuminance sensor when the deformed mask pattern image is scanned on the projection object.

15. A projection image control program is provided on a projection object with a plurality of illuminance sensors, the relative positions of which are known with respect to a projection object area, and an optical mask for blocking light incident on the illuminance sensors, and the projection image is controlled based on the position of each illuminance sensor specified in a pixel coordinate system of the projection image, A projection image control program that causes a computer to execute the following steps: identify the position of each illuminance sensor in a pixel coordinate system based on the output of each illuminance sensor when a mask pattern image having a mask pattern with the same pattern structure as the mask pattern of the optical mask is scanned on an object to be projected.

16. The projection image control program of claim 15, characterized in that the mask pattern has a pattern structure in which the amount of light transmission increases as the amount of deviation from the projected mask pattern image decreases, and the rate of change in the amount of light transmission relative to the change in the relative position of each pattern is greater in a range where the amount of deviation of each pattern is relatively small than in a range where the amount of deviation is large.

17. When identifying the position of each of the illuminance sensors in a pixel coordinate system, Identify the approximate location of each light sensor in the pixel coordinate system, calculating a transformation matrix that transforms the relative positions of each of the illuminance sensors on the projection object into the roughly determined positions of each of the illuminance sensors; Transforming the mask pattern image with the transformation matrix; 17. The projection image control program according to claim 15, wherein the position of each illuminance sensor is specified based on a change in output of each illuminance sensor when the deformed mask pattern image is scanned on an object to be projected.

Images