Projection control method and projection control apparatus

Abstract

To provide a projection control method and the like capable of associating positions of a projector and a camera with each other with a simple configuration.SOLUTION: A projection control method includes a first projection control step of causing a projector (20) to project a projection image including a plurality of projection indices, a detection control step of causing a camera (40) to capture the projection image, a display control step of displaying, on a display, an instruction to cause a user to arrange, in an image capturing range, a plurality of camera indices indicating relative positions with respect to the projection indices in a camera coordinate system obtained by image capturing, a camera coordinate detection step of detecting coordinates of the plurality of arranged camera indices, and a control step of determining a correspondence between camera coordinates and projection coordinates in accordance with the projection indices or a relative positional relationship between the camera indices.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a projection control method and a projection control device. [Background technology]

[0002] Patent Document 1 discloses a calibration device that, after calibrating a projector, acquires multiple pieces of position information indicating feature points of a screen, corrects the positions of the feature points of a test image based on the differences between the acquired position information, and recalibrates the projector. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-207373 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional calibration devices have a problem in that they are unable to establish correspondence between the positions of a projector and a camera with a simple configuration.

[0005] An object of the present disclosure is to provide a projection control method and the like that can associate the positions of a projector and a camera with a simple configuration. [Means for solving the problem]

[0006] In order to achieve the above object, a projection control method according to one aspect of the present disclosure is a projection control method in which a processor controls the projection of an image by a projector, and includes: a first projection control step of causing the projector to project a projection image including a plurality of projection indices located within a projection range of the projector; a detection control step of causing a camera to capture the projection image so that it fits within an imaging range; a display control step of causing a display to display instructions for a user to place a plurality of camera indices in the imaging range, the camera indices indicating relative positions to the projection indices in a camera coordinate system obtained by the imaging; a camera coordinate detection step of detecting coordinates of the placed camera indices; and a control step of determining a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera, depending on the relative positional relationship between the projection indices and the plurality of camera indices.

[0007] Furthermore, a projection control device according to one aspect of the present disclosure is a projection control device that controls the projection of an image by a projector, and includes: a first projection control unit that causes the projector to project a projection image including a plurality of projection indices located within a projection range of the projector; a detection control unit that causes a camera to capture the projection image so that it fits within an imaging range; a display control unit that causes a display to display instructions to a user to place a plurality of camera indices in the imaging range, the camera indices indicating relative positions to the projection indices in a camera coordinate system obtained by the imaging; a camera coordinate detection unit that detects the coordinates of the placed camera indices; and a control unit that determines a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera, depending on the relative positional relationship between the projection indices and the plurality of camera indices. [Effects of the Invention]

[0008] A projection control method according to an aspect of the present disclosure can associate the positions of a projector and a camera with a simple configuration. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a projection control device according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing an overview of a high-speed projection mapping system. [Figure 3] FIG. 3 is a diagram showing the equipment of a high-speed projection mapping system. [Figure 4] FIG. 4 is a diagram illustrating the issues with projection control in a high-speed projection mapping system. [Figure 5] FIG. 5 is a diagram showing an outline of the projection control method according to the first embodiment. [Figure 6] FIG. 6 is a flowchart of the projection control method according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an outline of image projection using the projection control method according to the first embodiment. [Figure 8A] FIG. 8A is a diagram showing a sensing area for image projection using the projection control method according to the first embodiment. [Figure 8B] FIG. 8B is a diagram showing calibration for image projection using the projection control method according to the first embodiment. [Figure 8C] FIG. 8C is a diagram showing parameter settings for image projection using the projection control method according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating the configuration of a projection control device according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating another configuration of the projection control device according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing projection of projection targets by the projection control device according to the second embodiment. [Figure 12] FIG. 12 is a diagram showing an outline of a projection control method according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing a projection control method according to the second embodiment. [Figure 14] FIG. 14 is a diagram showing a projection method for a projection image. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

[0011] The drawings are schematic diagrams and are not necessarily drawn to scale. Therefore, for example, the scales of the drawings do not necessarily match. In addition, the same reference numerals are used in the drawings to denote substantially the same components, and redundant explanations will be omitted or simplified.

[0012] A projection control method and a projection control device according to the following embodiments will be described.

[0013] (Embodiment 1) [Projection control device configuration] First, the configuration of the projection control device will be described. Fig. 1 is a block diagram of a projection control device 10 according to the first embodiment. The projection control device 10 includes a first projection control unit 11, a second projection control unit 12, a detection control unit 13, a display control unit 14, a camera coordinate detection unit 15, and a control unit 16. The projection control device 10 is connected to a projector 20, a display 30, and a camera 40 via wired or wireless connections. The projection control device 10 causes the camera 40 to capture an image of a marker 50, which is used for calibration during video projection.

[0014] The projection control device 10 is realized by a processor and a memory. The processor is, for example, a CPU (Central Processing Unit). The memory is a memory such as a DRAM (Dynamic Random-Access Memory) or an SRAM (Static Random-Access Memory). It may also be a non-volatile memory such as a ROM (Read-Only Memory) or a flash memory.

[0015] The first projection control unit 11 causes the projector 20 to project a projection image including multiple projection indices positioned within the projection range of the projector 20. The first projection control unit 11 generates a projection image including multiple projection indices and causes the projector 20 to project the projection image. The first projection control unit 11 acquires the coordinates of the multiple projection indices at the time of projection. In other words, the first projection control unit 11 acquires the coordinates of the positions of the projection indices in the coordinate system at the time of generation of the projection image. Note that the coordinates of the projection indices at the time of projection refer to the coordinates in the coordinate system set by the projection control device 10 when projecting the projection indices, and are distinguished from the coordinates of the positions of the projection indices in the coordinate system on the projection surface (the coordinate system in the camera image, which will be described later).

[0016] The detection control unit 13 causes the camera 40 to capture the projected image so that it fits within the imaging range. The detection control unit 13 may generate a camera image from the image captured by the camera 40. In other words, the camera image includes multiple projected indices projected as content of the projected image.

[0017] The display control unit 14 causes the display to display instructions for the user to place multiple camera indices, each indicating a relative position to the projected indices in the camera coordinate system obtained by the image capture, in the image capture range. Specifically, for example, the display control unit 14 may superimpose multiple camera indices, each indicating a relative positional relationship to the projected indices in the projected camera image, on the camera image displayed on the display 30. In this embodiment, the number of camera indices is the same as the number of projected indices. Here, the projection positions of the multiple displayed camera indices may be the positions where the corresponding projected indices should be in the camera image.

[0018] The camera coordinate detection unit 15 detects the coordinates of the multiple camera indices that have been placed. That is, the camera coordinate detection unit 15 detects the coordinates of the multiple camera indices that the user has placed on the screen or in the space where the projector 20 and the projection control device 10 are present. Specifically, for example, the camera coordinate detection unit 15 may accept an operation by the user to move any of the projected indices and the multiple camera indices included in the camera image displayed on the display 30, so as to adjust the positional relationship between each of the projected indices included in the camera image and each of the multiple camera indices.

[0019] The control unit 16 determines a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera, according to the relative positional relationship between the projection indices and the multiple camera indices. Specifically, the control unit 16 may determine a correspondence relationship between projection coordinates located within the projection range of the projector 20 and camera coordinates located within the imaging range of the camera 40, according to the manner in which either the projection indices or the multiple camera indices included in the camera image move on the display 30. For example, the control unit 16 may determine a correspondence relationship between the coordinates of the projection indices at the time of projection and the camera coordinates, according to the amount of movement of either one of the multiple projection indices or the multiple camera indices.

[0020] The control unit 16 may also cause the camera 40 to capture an image of the marker 50 present in the projection range. Additionally, the control unit 16 may calculate the camera coordinates of the marker 50 in the camera coordinate system of the marker 50 captured by the camera 40, or may convert the camera coordinates of the marker 50 into projection coordinates using the determined correspondence. The control unit 16 may also cause the projector 20 to project an image at the position of the marker 50 captured by the camera 40, using the converted projection coordinates of the marker 50.

[0021] The second projection control unit 12 causes the projector 20 to project an image using projection coordinates according to the determined correspondence.

[0022] Projector 20 projects a projection image including multiple projection indices onto a projection surface under the control of projection control device 10. The projector may be a CRT (Cathode-Ray Tube) projector that enlarges and projects an image displayed on a CRT using an optical system such as a lens, or a liquid crystal projector that enlarges and projects an image onto a screen using a lens, where light from a light source lamp using discharge light is transmitted through a built-in liquid crystal panel. Alternatively, the projector may be a DLP (Digital Light Processing) projector, an LCOS (Liquid Crystal On Silicon) projector, a GLV (Grating Light Valve) projector, or a laser projector.

[0023] Display 30 displays the camera image and the camera index under the control of projection control device 10. Display 30 may be a CRT display, a liquid crystal display, an organic EL (Electro-Luminescence) display, or a plasma display.

[0024] Camera 40, under the control of projection control device 10, captures an image of an imaging area including the projected image. Camera 40 is, for example, a visible light camera or an infrared camera. There may be multiple cameras 40, and the multiple cameras 40 may include a visible light camera and an infrared camera. Furthermore, instead of an infrared camera, an ultraviolet camera may be used as camera 40, and camera 40 may include a camera that can capture invisible light.

[0025] The marker 50 is an object that serves as a landmark and is captured by an infrared camera or the like. The marker 50 has an infrared reflecting member and reflects infrared light emitted from the infrared light 81. Because the marker 50 reflects infrared light, it appears in an infrared image captured by the infrared camera. The marker 50 is not connected to the projection control device 10 by wire or wirelessly.

[0026] In addition, the infrared reflecting member may be a retroreflective member. In the projection control device 10, an infrared light 81 is installed near the infrared camera, and the retroreflective member provided on the marker 50 reflects the infrared light emitted from the infrared light 81 toward the infrared camera.

[0027] The marker 50 may be provided with an LED (Light Emitting Diode) instead of an infrared reflecting member. The marker 50 may also emit infrared light or visible light by itself.

[0028] [High-speed projection mapping system] Next, a high-speed projection mapping system in which the projection control device 10 according to the first embodiment is used will be described. Fig. 2 is a diagram showing an overview of the high-speed projection mapping system. The high-speed projection mapping system is a system in which a high-speed projector projects an image by tracking the movement of a marker 50 detected by an infrared camera.

[0029] In a high-speed projection mapping system, a high-speed projector projects an image sent from a content server configured with a PC or the like onto a screen. For example, a person stands in front of the screen holding a marker 50. Then, the high-speed projection mapping system emits infrared light from an infrared light 81 and reflects the infrared light off the marker 50.

[0030] Next, the high-speed projection mapping system uses an infrared camera to capture an image of the marker 50 reflecting infrared light. Then, the high-speed projection mapping system uses a high-speed projector to project an image onto the screen at a position that overlaps with the marker 50. By repeating the above series of processes, the high-speed projection mapping system can project an image onto the screen in accordance with the movement of the marker.

[0031] For example, the frame rate of a high-speed projector may be 240 fps (frames per second), the frame rate of an infrared camera may be 240 fps, and the frequency of a content server may be 240 Hz.

[0032] The markers may be located behind a screen as viewed from the high-speed projector, and the infrared camera may capture the infrared light reflected by the markers through a semi-transparent or transparent screen to capture the images of the markers.

[0033] FIG. 3 is a diagram showing the components of a high-speed projection mapping system. The high-speed projection mapping system includes a high-speed projector 20, a content server equipped with a CPU and other components, a display 30, and a PoE-powered hub 70. The high-speed projector 20 may also include sensing components. Examples of the sensing components include an industrial camera, a camera lens, a visible light cut filter, an infrared projector, and a retroreflective sheet. The high-speed projector is connected to the content server via an HDMI (registered trademark) 90 (High Definition Multimedia Interface) as a video signal line. The content server is then connected to a PoE (Power over Ethernet)-powered hub 70 via a LAN cable 60. The PoE-powered hub 70 is also connected to each component of the high-speed projector 20 via the LAN cable 60. The industrial camera or the infrared projector may be powered by the PoE-powered hub 70. Images stored in the content server may be projected from the high-speed projector via HDMI (registered trademark) and simultaneously displayed on the display.

[0034] [Projection control issues in high-speed projection mapping] Next, we will explain the issues with projection control in high-speed projection mapping. Fig. 4 is a diagram showing the issues with projection control in a high-speed projection mapping system. As shown in Fig. 4, the projection area, which is the projection area of ​​the projector 20, is included in the imaging area, which is the imaging range of the camera 40. Then, it is assumed that the projection control device 10 wants to project an image onto a marker area located at coordinates (x, y) in the coordinate system (projection coordinate system) in the projection area. Here, the marker area is the area where the marker 50 sensed by the camera 40 is located.

[0035] However, since the projection control device 10 uses the image captured by the camera 40 when sensing the marker 50, the coordinates of the sensed marker 50 are (x', y') in the coordinate system (camera coordinate system) of the image captured by the camera 40. Here, the image captured by the projection control device 10 with the camera 40 to sense the marker 50 is an infrared image. Note that the image captured by the projection control device 10 with the camera 40 to sense the marker 50 may also be a visible light image.

[0036] Therefore, the projection control device 10 needs to convert the coordinates (x', y') of the sensed marker 50 in the coordinate system in the image captured by the camera 40 into coordinates (x, y) in the coordinate system in the projection area.

[0037] Coordinate Conversion Next, the coordinate conversion performed by the projection control device 10 will be described. Fig. 5 is a diagram showing an outline of the projection control method in the first embodiment. First, the first projection control unit 11 of the projection control device 10 causes the projector 20 to project a projection image including multiple projection indices located within the projection range of the projector 20. At this time, the coordinates of the multiple projection indices in the coordinate system in the projection area (projection coordinate system) are known.

[0038] Next, the display control unit 14 of the projection control device 10 displays multiple camera indices, the number of which is the same as the number of projected indices, indicating relative positions of the projected indices in the projected camera image, superimposed on the camera image on the display 30. The camera coordinate detection unit 15 of the projection control device 10 then accepts a user operation to move any of the projected indices and the multiple camera indices included in the camera image displayed on the display 30 so as to adjust the positional relationship between each of the projected indices included in the camera image displayed on the display 30 and each of the multiple camera indices. In this way, the camera coordinate detection unit 15 acquires the coordinates of the projected indices in the coordinate system (camera coordinate system) of the image captured by the camera 40. At this time, the user operation accepted by the camera coordinate detection unit 15 may be, for example, an operation to superimpose each of the projected indices included in the camera image displayed on the display 30 over each of the multiple camera indices.

[0039] Through the above operations, the projection control device 10 acquires the correspondence relationship between a plurality of coordinates in the coordinate system (camera coordinate system) in the image captured by the camera 40 and corresponding coordinates in the coordinate system (projection coordinate system) in the projection area. Then, from the correspondence relationship, the projection control device 10 acquires a conversion formula from the coordinates in the coordinate system (camera coordinate system) in the image captured by the camera 40 to the corresponding coordinates in the coordinate system (projection coordinate system) in the projection area. The conversion method from the coordinates in the coordinate system (camera coordinate system) in the image captured by the camera 40 to the corresponding coordinates in the coordinate system (projection coordinate system) in the projection area may be a method such as linear interpolation, affine transformation, or projective transformation.

[0040] Furthermore, the projection control device 10 may obtain, from the correspondence relationship, a table used to convert coordinates in the coordinate system (camera coordinate system) in the image captured by the camera 40 into corresponding coordinates in the coordinate system (projection coordinate system) in the projection area.

[0041] [Projection control method] Next, a description will be given of a projection control method performed by the projection control device 10. Fig. 6 is a flowchart of the projection control method according to the first embodiment.

[0042] First, the first projection control unit 11 causes the projector 20 to project a projection image including a plurality of projection indices (step S10). Here, the number of projection indices may be three or four. The shape of the projection indices may be a cross, a circle, or a rectangle. The shape of the projection indices may also be a shape other than those mentioned above.

[0043] Next, detection control unit 13 causes camera 40 to capture the projected image so that it fits within the imaging range, thereby generating a camera image (step S11). Here, camera 40 may be an infrared camera or a visible light camera. The camera image may be an infrared image or a visible light image.

[0044] Next, the display control unit 14 causes the display 30 to display a plurality of camera indices superimposed on the camera image (step S12).

[0045] Then, the camera coordinate detection unit 15 accepts an operation by the user to move any of the projected index and the multiple camera indices included in the camera image displayed on the display 30 so as to adjust the positional relationship between the projected index and the multiple camera indices included in the camera image displayed on the display 30 (step S13).

[0046] Here, for example, adjusting the positional relationship between the projected index and the multiple camera indices included in the camera image displayed on the display 30 means overlapping the projected index and the multiple camera indices included in the camera image displayed on the display 30. Note that the projected index and the multiple camera indices included in the camera image displayed on the display 30 do not have to completely overlap.

[0047] Furthermore, for example, adjusting the positional relationship between the projected index and the multiple camera indices included in the camera image displayed on the display 30 may mean bringing the projected index and the multiple camera indices included in the camera image displayed on the display 30 closer to each other than a predetermined distance. Note that adjusting the positional relationship between the projected index and the multiple camera indices included in the camera image displayed on the display 30 may mean moving the projected index in the projected image and / or moving the camera indices displayed on the display 30.

[0048] Furthermore, the camera coordinate detection unit 15 may change the positions of a plurality of projection indices positioned within the projection range, and then accept an operation to move the camera indices.

[0049] Next, the control unit 16 determines the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40, depending on the manner of movement on the display 30 of the projection index and any of the multiple camera indexes included in the camera image (step S14).

[0050] As a result, the control unit 16 calculates a conversion formula from coordinates in the coordinate system (camera coordinate system) in the image captured by the camera 40 to corresponding coordinates in the coordinate system (projection coordinate system) in the projection area, based on the correspondence relationship between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40. The control unit 16 may calculate a table expressing the correspondence relationship instead of the conversion formula. For example, when there are four projection indices and four camera indices, the control unit 16 may associate the two using a projective transformation method.

[0051] Next, the second projection control unit 12 uses the projection coordinates according to the determined correspondence to project an image on the projector 20 (step S15). For example, the second projection control unit 12 uses the conversion equation calculated in step S14 to calculate projection coordinates located within the projection range of the corresponding projector 20 from the camera coordinates located within the imaging range of the camera 40, and projects the image on the calculated projection coordinates. The second projection control unit 12 may use the conversion equation calculated in step S14 to convert the coordinates of the coordinate system in the image captured by the camera 40 of the image to be projected (camera coordinate system) into coordinates in the coordinate system in the projection area (projection coordinate system), and project the image of the projection target.

[0052] [Calibration and Content Playback] Next, the application of the projection control method according to the first embodiment to high-speed projection mapping will be described. Fig. 7 is a diagram showing an overview of image projection using the projection control method according to the first embodiment. The projection control method according to the first embodiment described in Fig. 6 is applied to high-speed projection mapping as follows.

[0053] First, we will explain the calibration performed in high-speed projection mapping. In (1) shown in Fig. 7, the projection control device 10 outputs a projection image for calibration. Here, calibration refers to the alignment of a projected image so that the projection control device 10 can project the image from the projector 20. Furthermore, the multiple units in the projection control device 10 that perform a series of calibration processes are referred to as calibration functions.

[0054] Next, the projection control device 10 acquires the camera image captured by the camera 40 at (2) shown in (a) of Fig. 7. Then, the projection control device 10 performs a calibration operation at (3) shown in (a) of Fig. 7. The calibration operation at (3) shown in (a) of Fig. 7 mainly corresponds to the processing performed in steps S12 to S14 described in Fig. 6.

[0055] 7(a) (4), the projection control device 10 generates a setting file relating to the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40, calculated as a result of the calibration operation. The setting file generated by the calibration function is output to the position detection function, which will be described next.

[0056] The position detection function refers to multiple parts of the high-speed projection mapping system that perform a series of processes to detect the position of marker 50 when playing back content in high-speed projection mapping. The position detection function may or may not be included in the projection control device 10.

[0057] In (1) shown in FIG. 7(b), the position detection function reads the setting file generated by the calibration function. Next, in (2) shown in FIG. 7(b), the position detection function acquires a camera image captured by the camera 40, which is an image including the marker 50. Next, in (3) shown in FIG. 7(b), the position detection function calculates the coordinates of the marker 50 from the acquired camera image based on the setting file, and further converts the marker coordinates into projection coordinates and provides them to the image generation unit. The image generation unit generates an image that tracks the marker 50 and is used for high-speed projection mapping. Then, in (4) shown in FIG. 7(b), the image generation unit generates a tracking image that tracks the marker 50. Next, in (5) shown in FIG. 7(b), the image generation unit outputs the tracking image generated.

[0058] [Projection control method implementation] Next, an example of implementing the projection control method according to the first embodiment will be described. First, the projection control device 10 determines the area to be sensed by the camera 40. Here, sensing means detecting the marker 50 using an image captured by the camera 40. FIG. 8A is a diagram showing the sensing area for image projection using the projection control method according to the first embodiment. For example, the largest rectangular area shown in FIG. 8A is set as the sensing area. The projection control device 10 determines the size of the area to be sensed from within the range that can be captured by the camera 40. The area may be, for example, an area of ​​768 x 360 pixels.

[0059] The projection control device 10 also determines a coordinate system within the region. For example, the projection control device 10 may determine that the top left corner of the region is (x, y)=(0, 0).

[0060] The projection control device 10 then adjusts the size and position of the sensing area so that the multiple projected indices included in the projected image fit within the sensing area. Here, the projected image is a rectangle including four cross marks and is an image divided into four areas. For example, as shown in FIG. 8A, the projection control device 10 adjusts the size and position of the sensing area so that the multiple projected indices fit within the sensing area. At this time, the size and position of the projected image may also be adjusted. For example, the projected indices may be cross marks as shown in FIG. 8A. Furthermore, each of the four projected indices is positioned so that it fits within one of the four divided rectangles.

[0061] Next, the projection control device 10 associates the camera coordinates with the projection coordinates. Fig. 8B is a diagram showing calibration for image projection using the projection control method according to the first embodiment.

[0062] In this embodiment, before the user's operation to move the camera index is accepted, each of the four default camera indexes is positioned so that it fits into one of the four areas obtained by dividing the camera image into two horizontally and two vertically.

[0063] The projection control device 10 accepts an operation by the user to move the position of each of the multiple camera indices so that it matches the position of each of the multiple corresponding projected indices. Here, the camera indices are square cursors with a cross mark inside. For example, the camera indices are moved up, down, left, right, and up by pressing the arrow keys on a computer keyboard. Note that the camera indices may also be moved up, down, left, and right by clicking and dragging a mouse. Furthermore, multiple projected indices corresponding to multiple camera indices refer to those in which the four divided projected images in which the indices are located are in the same position as the camera image.

[0064] The selection of each of the multiple camera indices may be switched by pressing the Tab key. Note that the selection of each of the multiple camera indices may be switched by clicking the mouse.

[0065] At this time, the projection control device 10 displays the camera image on the display 30 in a coordinate system (camera coordinate system) for the image captured by the camera 40. Here, the camera image is a visible light image.

[0066] Next, the projection control device 10 sets (calibrates) parameters for sensing the marker 50. Fig. 8C is a diagram showing parameter settings for image projection using the projection control method according to the first embodiment.

[0067] The projection control device 10 captures an infrared image with a visible light cut filter attached in front of the camera lens of the camera 40. This infrared image may be called a marker image. At this time, the marker 50 is positioned so that it appears within the sensing area described in FIG. 8A.

[0068] Next, the projection control device 10 adjusts the brightness of the infrared image captured by the camera 40 by adjusting the camera gain.

[0069] Next, the projection control device 10 adjusts the image obtained by binarizing the infrared image so that only the portion where the marker 50 appears is displayed bright. For example, the projection control device 10 displays the image in a range above a threshold brightly (e.g., red) and the image in a range below the threshold darkly (e.g., in a color other than red). Adjusting the image obtained by binarizing the infrared image so that only the portion where the marker 50 appears is displayed brightly may mean that the projection control device 10 adjusts the image so that only the portion where the marker 50 appears is displayed red. At this time, the projection control device 10 adjusts the threshold used in the binarization process performed on the infrared image so that only the portion where the marker 50 appears is displayed brightly (cutting out the marker area). At this time, the projection control device 10 may display the images in the order of red, yellow, green, and blue, from areas with the most light reflection to areas with the least light reflection.

[0070] (Embodiment 2) [Projection control device with multiple cameras] 9 is a diagram showing the configuration of a projection control device 10 according to the second embodiment. The projection control device 10 may be connected to multiple cameras 40a, 40b, and 40c via a LAN cable 60. The cameras 40a, 40b, and 40c are connected to a PoE power supply compatible hub 70 via the LAN cable 60. The PoE power supply compatible hub 70 is then connected to the projection control device 10 via the LAN cable 60. The number of cameras is not limited to three, cameras 40a, 40b, and 40c, and any number of cameras may be used as long as they are plural.

[0071] First, the cameras 40a, 40b, and 40c irradiate the marker 50 with infrared light. Then, the cameras 40a, 40b, and 40c capture images of the marker 50 that reflect the infrared light. Here, the marker 50 is a retroreflective marker.

[0072] Next, the camera coordinate detection unit 15 detects the position of the marker 50 photographed by the cameras 40a, 40b, and 40c. The camera coordinate detection unit 15 may use dedicated software to detect the position of the marker 50 photographed by the cameras 40a to 40n.

[0073] Then, the control unit 16 determines the correspondence between the projection coordinates located within the projection range of the projector and the camera coordinates located within the imaging range of the camera, according to the relative positional relationship between the projection indices and the multiple camera indices. In other words, the control unit 16 performs coordinate conversion of the camera coordinates. Then, based on the converted camera coordinates, the projector 20 generates content to be projected. The projector 20 projects the content transmitted from the control unit 16.

[0074] Fig. 10 is a diagram showing another configuration of the projection control device 10 according to the second embodiment. The projection control device 10 may be connected to multiple cameras 40a, 40b, and 40c via a LAN cable 60. In the configuration shown in Fig. 10, the marker 50 is an infrared LED marker that emits light itself. The projection control device 10 is also connected to a synchronization signal transmitter 80 via the LAN cable 60. The other configurations are the same as those shown in Fig. 9.

[0075] First, the marker 50 emits light. The user may place the marker 50 in any location and cause the marker 50 to emit light. Then, the cameras 40a, 40b, and 40c capture the illuminated marker 50.

[0076] Next, the camera coordinate detection unit 15 detects the position of the marker 50 photographed by the cameras 40a, 40b, and 40c. At this time, the camera coordinate detection unit 15 synchronizes with the marker 50 via the synchronization signal transmitter 80. Specifically, the marker 50 receives a synchronization signal transmitted by the synchronization signal transmitter 80 and emits light in accordance with the synchronization signal, thereby synchronizing the marker 50 with the camera coordinate detection unit 15. The camera coordinate detection unit 15 may detect the position of the marker 50 photographed by the cameras 40a, 40b, and 40c using dedicated software.

[0077] By using multiple LEDs and making each LED emit pulses in a different pattern using a synchronization signal, it is possible to assign a unique ID to the marker. If there is no need to assign an ID, there is no need to send or receive a synchronization signal.

[0078] Then, the control unit 16 determines the correspondence between the projection coordinates located within the projection range of the projector and the camera coordinates located within the imaging range of the camera, according to the relative positional relationship between the projection indices and the multiple camera indices. In other words, the control unit 16 performs coordinate conversion of the camera coordinates. Then, based on the converted camera coordinates, the projector 20 generates content to be projected. The projector 20 projects the content transmitted from the control unit 16 under the control of the second projection control unit 12.

[0079] 11 is a diagram showing projection of projection indices by the projection control device 10 in embodiment 2. The projection control device 10 acquires the coordinates of four points on the same surface and two points on a surface different from the four points on the same surface. Specifically, the display control unit 14 of the projection control device 10 causes the projector 20 to project the four projection indices on the same surface, such as a screen.

[0080] Then, the cameras 40a, 40b, and 40c capture images of the markers 50 that the user has placed on the projected indices. Subsequently, the camera coordinate detection unit 15 detects the position coordinates of the markers 50 captured by the cameras 40a, 40b, and 40c.

[0081] The projection control device 10 performs similar processing for four points on the same surface and two points on different surfaces. Specifically, the display control unit 14 of the projection control device 10 projects two projected indices into the space on the same surface where the four points were projected and on a different surface. For example, the display control unit 14 may cause the projector 20 to project the projected indices onto a board placed by the user.

[0082] Then, cameras 40a, 40b, and 40c capture images of markers 50 attached to the board placed by the user. Subsequently, camera coordinate detection unit 15 detects the position coordinates of markers 50 captured by cameras 40a, 40b, and 40c. These position coordinates are three-dimensional coordinates in the space in which the cameras are placed. By capturing images with multiple cameras, three-dimensional coordinates can be obtained using techniques such as stereo measurement.

[0083] For example, the control unit 16 uses the coordinates of the detected projected indices projected at the positions of the four points A, B, C, and D to calculate a formula for correcting discrepancies in the positions and rotation degrees of the reference points and discrepancies in the sizes of the projected indices. This formula is, for example, an affine transformation. Then, the control unit 16 uses the coordinates of the detected projected indices projected at the positions of the four points A, B, C, and D to calculate a formula for removing distortion in the projection of the projector 20. This formula is, for example, a projective transformation.

[0084] Then, for example, the control unit 16 calculates the installation position of the projector 20 and the setting state of the position or angle of view using the coordinates of the detected projection indices projected at the positions of the four points A, B, E, and F. In addition, the control unit 16 calculates a calculation formula that represents the projection direction and lens shift, taking into account the shapes of the four points A, B, C, and D.

[0085] Fig. 12 is a diagram showing an outline of a projection control method in embodiment 2. As shown in (a) of Fig. 12, a user places a marker 50 provided on a board 51 at a position instructed by a display control unit 14.

[0086] As shown in (b) of Fig. 12, the display control unit 14 instructs the projector 20 to project a projection index 52 at a predetermined position, thereby instructing the user on the position where the marker 50 should be placed. As shown in (b) of Fig. 12, the display control unit 14 may project multiple projection indexes. The number of projection indexes is, for example, four.

[0087] The display control unit 14 may project four projected indices onto the same plane such as a screen, and may project two more projected indices onto a plane different from the plane onto which the four projected indices are projected. The display control unit 14 may project multiple projected indices simultaneously, or may project them one by one in sequence.

[0088] The user places a marker 50 provided on the board 51 at the position where the projected index is projected. Here, the marker 50 is a marker that can be detected by an infrared camera. Then, the camera coordinate detection unit 15 detects the position of the marker 50 photographed by the cameras 40a to 40n. The camera coordinate detection unit 15 performs the above-mentioned marker 50 detection process at four points on the same plane. Then, the control unit 16 performs projection control on the projector 20 using the relationship between the coordinates of the marker 50 detected by the camera coordinate detection unit 15 and the coordinates of the projected index on the projection image projected by the display control unit 14.

[0089] 12(c), for two of the four projected indices projected on the same plane, the user places markers 50 on a plane different from the plane on which the four projected indices are projected. Specifically, the user places markers 50 provided on board 51 on a line connecting projector 20 and the projected indices on the plane on which the four projected indices are projected, but on a plane different from the plane on which the four projected indices are projected.

[0090] Then, the camera coordinate detection unit 15 detects the coordinates of the markers 50 placed at the two points. Subsequently, the second projection control unit 12 performs projection control on the projector 20 by detecting the position of the projector 20 using the relationship between the coordinates of the markers 50 placed at the two points and the coordinates of the projection indices on the projection image projected by the display control unit 14 to the projector 20.

[0091] That is, the second projection control unit 12 uses the detected position of the projector 20 to adjust the projection image to be projected by the projector 20, and controls the image projection operation of the projector 20. Here, the image projection operation may be controlling the position or size at which the content is projected.

[0092] [Operation of a projection control device with multiple cameras] Next, a description will be given of the operation of the projection control device 10 in embodiment 2. Fig. 13 is a flowchart showing the projection control method in embodiment 2.

[0093] First, the display control unit 14 projects a projection image including a plurality of projection indices using the projector 20 (step S10). The display control unit 14 may project the projection indices sequentially at four locations on the same surface. Note that the positions at which the projection indices are projected do not need to be fixed. The positions at which the projection indices are projected may be changed arbitrarily.

[0094] The following steps S12 and S13 are repeated for each of four locations on the same surface (step S11).

[0095] Next, the user places a marker 50 that can be detected by the camera 40 at the position of the projected index (step S12). The user places a board 51 on which the marker 50 is provided on the plane on which the projected index is projected. Here, the plane on which the projected index is projected may be, for example, a screen.

[0096] Next, the camera coordinate detection unit 15 acquires the coordinates of the marker 50 by the detection means (step S13). The camera coordinate detection unit 15 detects the position of the marker 50 by, for example, an infrared camera.

[0097] Here, the process returns to step S11 (step S14).

[0098] The plane onto which the projected indices are projected may be an empty space. In this case, if marks are placed on the floor, for example, so that the surface on which the board 51 is to be placed is clear, the user can place the board 51 using the marks as a guide, and the four points where the user has placed the markers 50 will be distributed approximately on the same plane.

[0099] Next, the control unit 16 acquires the correspondence relationship between the coordinates of the marker 50 from the detection result and the coordinates of the projected index on the projected image (step S15). The control unit 16 acquires the correspondence relationship between the coordinates of the marker 50 placed by the user, detected by the camera coordinate detection unit 15, and the coordinates of the projected index on the projected image projected by the display control unit 14 with the projector 20. In other words, the control unit 16 acquires the relationship between the coordinates of the content projected by the control unit 16 with the coordinates in real space.

[0100] Next, the second projection control unit 12 projects an image using the projector 20 using the acquired correspondence relationship (step S16).

[0101] If the camera 40 functions as a visible light camera, the user does not need to manually position the markers 50. For four projected indices projected on the same plane, the control unit 16 projects them onto an existing plane, such as a screen, and detects the positions of the projected indices from an image captured by the camera 40 functioning as a visible light camera, thereby detecting the correspondence between the coordinates of the markers 50 and the coordinates of the projected indices on the projected image. Furthermore, for two of the four projected on a different plane from the four projected on the same plane, the control unit 16 can detect the positions of the projected indices from an image captured by the camera 40 functioning as a visible light camera by placing a board 51. In this case, the markers 50 do not need to be provided on the board 51.

[0102] In this case, the camera 40 may be configured to be switchable between a function as an infrared camera and a function as a visible light camera.

[0103] Furthermore, for four projected indices projected on the same plane, the control unit 16 may cause the projector 20 to project the projected indices while moving the projected indices so that the four points match the positions of markers 50 that have been placed in advance on an existing plane such as a screen onto which the four points are projected. Furthermore, for two of the four points that are projected on a plane different from the four points projected on the same plane, the control unit 16 may obtain the correspondence between the coordinates of the markers 50 and the coordinates of the projected indices on the projected image by detecting the positions at which the two points are projected using the two of the four points.

[0104] Next, a description will be given of a projection method of the projector 20 controlled by the second projection control section 12. Fig. 14 is a diagram showing the projection method of the projected image.

[0105] 14(a), parallel projection is a projection method in which an image is projected onto the foot of a perpendicular line drawn from the marker 50 to the screen 200. In parallel projection, even if the marker 50 is located at a distance from the screen 200, the positions of the marker 50 and the projected image 100 coincide in front of the audience.

[0106] 14(b), projection projection is a projection method in which an image is projected onto a screen 200 on a straight line connecting the projector 20 and the marker 50. In projection projection, a viewer positioned close to the projector 20 can view the projected image with the marker 50 and the projected image 100 relatively overlapping in position.

[0107] 14(c), virtual viewpoint projection is a projection method in which an image is projected onto a screen 200 on a straight line connecting a specific virtual viewpoint 300 set at a predetermined position and a marker 50. When a camera is installed at the position set as the virtual viewpoint 300, the marker 50 and the projected image 100 always appear to be aligned from the camera.

[0108] As shown in FIG. 14(d), object mapping is a projection method in which an image is projected onto an object on which a marker 50 is provided. Here, the object is assumed to be a plate-like object, a human body, or the like. The object may also be a building or a moving object. In object mapping, a projected image 100 is projected on a line connecting the projector 20 and the marker 50, but the size of the projected image 100 is changed depending on the distance between the projector 20 and the marker 50. In object mapping, the projected image 100 appears to the audience as if it is following the object on which the marker 50 is provided.

[0109] 14(a) to 14(d), the screen 200 may be placed in front of the marker 50 as viewed from the projector 20. In this case, the screen 200 may be made of a translucent material. For example, the translucent material may be a mesh material or a transparent plate containing diffusing particles.

[0110] In the projection method of FIG. 14(d), if the screen 200 is placed in front of the marker 50 as viewed from the projector 20, the image can be projected onto the object without creating a sense of incongruity even after passing through the screen, provided that the translucent material has a sufficiently high transmittance. The transmittance of the translucent material should be 90% or higher, preferably about 95%. Even with a screen with such high transmittance and low reflectance, the projected image is still visible. However, because the image projected onto the object placed behind the translucent screen is brighter than the image reflected by the screen, the image projected onto the object is primarily visible.

[0111] The control unit 16 may be able to switch between the following processes depending on which of the four projection methods described in (a) to (d) of FIG. 14 is used. The control unit 16 detects the positions of only four projected indices projected on the same plane, thereby detecting the correspondence between the coordinates of the markers 50 and the coordinates of the projected indices on the projected image 100. Alternatively, the control unit 16 may detect the positions of two of the four projected indices that are projected on a plane different from the four projected on the same plane, in addition to the four projected indices, thereby detecting the correspondence between the coordinates of the markers 50 and the coordinates of the projected indices on the projected image 100. The control unit 16 may be able to determine which of the above two processes to perform.

[0112] 14(a) to 14(d) is used, the display control unit 14 may switch the projection mode of the projected image 100 among four modes, from the first mode to the fourth mode, based on the correspondence between the coordinates of the marker 50 calculated by the control unit 16 and the coordinates of the projection index on the projected image 100. When switching the projection mode, the size, angle, distance to the projection destination, etc. of the projected image 100 may be switched for the same source content.

[0113] The number of projection modes is not limited to four, but may be any number as long as there are a plurality of projection modes, for example, two or three.

[0114] [Effects, etc.] The projection control method according to an embodiment of the present disclosure is a projection control method in which a processor controls the projection of an image by the projector 20, and includes: a first projection control step of causing the projector 20 to project a projection image including a plurality of projection indices located within the projection range of the projector 20; a detection control step of causing the camera 40 to capture the projection image so that it fits within an imaging range; a display control step of causing a display to display instructions for a user to place a plurality of camera indices in the imaging range, the camera indices indicating relative positions to the projection indices in a camera coordinate system obtained by the imaging; a camera coordinate detection step of detecting the coordinates of the placed camera indices; and a control step of determining the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40, depending on the relative positional relationship between the projection indices and the plurality of camera indices.

[0115] As a result, the projection control method according to the embodiment of the present disclosure can associate the positions of the projector 20 and the camera 40 with a simple configuration.

[0116] Furthermore, for example, the projection control method according to the embodiment of the present disclosure may further include a second projection control step of causing the projector 20 to project an image using projection coordinates according to the determined correspondence relationship.

[0117] As a result, the projection control method according to the embodiment of the present disclosure can project an image in which the positions of the projector 20 and the camera 40 are appropriately associated with each other.

[0118] In the projection control method according to an embodiment of the present disclosure, in the camera coordinate detection step, a user operation is accepted to move one of the projection index and the multiple camera indices included in the camera image displayed on the display on the display so as to adjust the positional relationship between the projection index and the multiple camera indices included in the camera image displayed on the display.

[0119] As a result, the projection control method according to the embodiment of the present disclosure can associate the positions of the projector 20 and the camera 40 with a simple configuration.

[0120] Also, for example, in the projection control method according to an embodiment of the present disclosure, the operation may be an operation in which the user moves a projection index included in a camera image displayed on the display 30 so that the positional relationship between the projection index included in the camera image and multiple camera indices is adjusted.

[0121] As a result, the projection control method according to the embodiment of the present disclosure can associate the projection indices included in the camera image with multiple camera indices.

[0122] Also, for example, in the projection control method according to an embodiment of the present disclosure, the operation may be an operation in which the user moves multiple camera indices on the display 30 so that the positional relationship between the projection indices included in the camera image displayed on the display 30 and the multiple camera indices is adjusted.

[0123] As a result, the projection control method according to the embodiment of the present disclosure can associate a plurality of camera indices with the projection indices included in the camera image.

[0124] Furthermore, for example, in the projection control method according to the embodiment of the present disclosure, the user adjusting the positional relationship between the projection index included in the camera image displayed on the display 30 and the multiple camera indices may mean that the user adjusts the positional relationship between the projection index included in the camera image displayed on the display 30 and the multiple camera indices so that they overlap.

[0125] As a result, the projection control method according to the embodiment of the present disclosure can accurately associate the coordinates of an image projected from a projector with the coordinates of an image captured by a camera.

[0126] Also, for example, in the projection control method according to the embodiment of the present disclosure, the number of the multiple projection indices and the multiple camera indices is four each.

[0127] As a result, the projection control method according to the embodiment of the present disclosure can use projective transformation to establish a positional relationship between the projected indices included in the camera image displayed on the display 30 and the multiple camera indices.

[0128] Also, for example, in the projection control method according to the embodiment of the present disclosure, the control step uses four projection indices and four camera indices to determine the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40.

[0129] As a result, the projection control method according to the embodiment of the present disclosure can calibrate the projection image of the projector 20 using the four projection indices.

[0130] Also, for example, in the projection control method according to the embodiment of the present disclosure, the number of the plurality of projection indices and the number of the plurality of camera indices are six each.

[0131] As a result, by using two specific points in addition to the four points, the position of the projector 20 can be detected.

[0132] Also, for example, in the projection control method according to the embodiment of the present disclosure, the control step uses six projection indices and six camera indices to determine the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40.

[0133] As a result, the projection control method according to the embodiment of the present disclosure can calibrate the projection image of the projector 20 using six projection indices.

[0134] Furthermore, for example, the projection control method in an embodiment of the present disclosure may further include a marker imaging step of causing camera 40 to capture an image of marker 50 present in the projection range; a marker calculation step of calculating the camera coordinates of the marker image of marker 50 captured by camera 40; a conversion step of converting the camera coordinates of marker 50 into projection coordinates using the determined correspondence; and a third projection control step of causing projector 20 to project an image at the position of marker 50 captured by camera 40 using the converted projection coordinates of marker 50.

[0135] As a result, the control method according to the embodiment of the present disclosure can detect the position of the marker 50 and project an image according to the position of the marker 50.

[0136] Furthermore, for example, in the projection control method according to the embodiment of the present disclosure, the camera image may be a visible light image, and the marker image may be an infrared image.

[0137] As a result, the projection control method according to the embodiment of the present disclosure can associate the positions of the projector 20 and the camera 40 using both the visible light image and the infrared image.

[0138] Furthermore, for example, in the projection control method according to the embodiment of the present disclosure, the marker 50 may include an infrared reflecting member.

[0139] As a result, the projection control method according to the embodiment of the present disclosure can detect the marker 50 that reflects infrared rays and associate the positions of the projector and camera.

[0140] Furthermore, for example, in the projection control method according to the embodiment of the present disclosure, the marker 50 may include an LED and emit light by itself.

[0141] As a result, the projection control method according to the embodiment of the present disclosure can detect the light emitted from the self-emitting marker 50 and associate the positions of the projector 20 and the camera 40.

[0142] Furthermore, for example, the projection control method according to the embodiment of the present disclosure may further include a change step of moving the camera index after changing the positions of the multiple projection indexes located within the projection range.

[0143] As a result, the projection control method according to the embodiment of the present disclosure can move the camera index after adjusting the position of the projection index to match the position of the screen, and therefore the projection control method according to the embodiment of the present disclosure can appropriately associate the positions of the projector 20 and the camera 40.

[0144] The projection control device 10 in the embodiment of the present disclosure is a projection control device 10 that controls the projection of an image by the projector 20, and includes: a first projection control unit 11 that causes the projector 20 to project a projection image including multiple projection indices located within the projection range of the projector 20; a detection control unit 13 that causes the camera 40 to capture the projection image so that it fits within the imaging range; a display control unit 14 that causes a display to display instructions for a user to place multiple camera indices in the imaging range, which indicate relative positions to the projection indices in the camera coordinate system obtained by the imaging; a camera coordinate detection unit 15 that detects the coordinates of the placed multiple camera indices; and a control unit 16 that determines the correspondence between the projection coordinates located within the projection range of the projector 20 and the camera coordinates located within the imaging range of the camera 40, depending on the relative positional relationship between the projection indices and the multiple camera indices.

[0145] As a result, the projection control device according to the embodiment of the present disclosure can achieve the same effects as the projection control method described above.

[0146] (others) Furthermore, in the above embodiment, each of the components, particularly the first projection control unit 11, the second projection control unit 12, the detection control unit 13, the display control unit 14, the camera coordinate detection unit 15, and the control unit 16, may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory on a PC or smartphone.

[0147] Furthermore, each component may be realized by hardware. Each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0148] Furthermore, the general or specific aspects of the present invention may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0149] For example, the present invention may be realized as the terminal of the above-described embodiment, or as a system corresponding to the terminal. Furthermore, the present invention may be realized as a projection control method, as a program for causing a computer to execute the projection control method, or as a computer-readable non-transitory recording medium on which such a program is recorded. The program includes an application program for operating a general-purpose mobile terminal as the mobile terminal of the above-described embodiment.

[0150] Furthermore, in the above embodiment, the projection control device 10 is realized by a single device, but it may also be realized by multiple devices. When the projection control device 10 is realized by multiple devices, the components of the projection control device 10 described in the above embodiment may be distributed among the multiple devices in any manner.

[0151] Furthermore, in the above embodiment, the projector 20 and the camera 40 are configured as separate bodies, but this is not limiting, and the projector 20 may have a built-in camera 40.

[0152] In addition, the present invention also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the spirit of the present invention. [Industrial Applicability]

[0153] The present invention can be used in commercial performances involving video projection. [Explanation of symbols]

[0154] 10 Projection control device 11 First projection control unit 12 Second projection control unit 13 Detection control section 14 Display control unit 15 Camera coordinate detection unit 16 Control Unit 20 Projector 30 Display 40, 40a, 40b, 40c cameras 50 markers 51 Board 52 Projection Index 60 LAN cable 70 PoE-powered hub 80 Synchronization signal transmitter 81 Infrared Light 90 HDMI 100 projected images 200 screens 300 Virtual Viewpoint

Claims

1. A projection control method in which a processor controls projection of an image by a projector, comprising: a first projection control step of causing the projector to project a projection image including a plurality of projection indices positioned within a projection range of the projector; a detection control step of causing a camera to capture the projected image so that the image fits within an imaging range; a display control step of displaying on a display an instruction for a user to arrange a plurality of camera indices, each indicating a relative position with respect to the projected indices in a camera coordinate system obtained by the imaging, in the imaging range; a camera coordinate detection step of detecting coordinates of the plurality of arranged camera indices; a control step of determining a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera in accordance with a relative positional relationship between the projection indices and the plurality of camera indices, Projection control method.

2. a second projection control step of causing the projector to project the image using the projection coordinates according to the determined correspondence relationship; The projection control method according to claim 1 .

3. In the camera coordinate detection step, a user operation is accepted to move, on the display, any of the projected index and the plurality of camera indexes included in the camera image displayed on the display so as to adjust a positional relationship between the projected index and the plurality of camera indexes included in the camera image displayed on the display.

3. The projection control method according to claim 1.

4. the operation is an operation in which the user moves the projected index included in the camera image displayed on the display so as to adjust a positional relationship between the projected index included in the camera image displayed on the display and the plurality of camera indices; The projection control method according to claim 3 .

5. the operation is an operation in which the user moves the plurality of camera indices on the display so as to adjust a positional relationship between the projection indices included in the camera image displayed on the display and the plurality of camera indices; 5. The projection control method according to claim 3 or 4.

6. The user adjusting the positional relationship between the projection index included in the camera image displayed on the display and the plurality of camera indices means that the user adjusts the positional relationship between the projection index included in the camera image displayed on the display and the plurality of camera indices so that the projection index included in the camera image displayed on the display overlaps with the plurality of camera indices. The projection control method according to any one of claims 3 to 5.

7. the number of the plurality of projection indices and the number of the plurality of camera indices are each four; The projection control method according to any one of claims 1 to 6.

8. In the control step, a correspondence relationship between a projection coordinate located within the projection range of the projector and a camera coordinate located within the imaging range of the camera is determined using four projection indices and four camera indices. The projection control method according to claim 7.

9. the number of the plurality of projection indices and the number of the plurality of camera indices are each six; The projection control method according to any one of claims 1 to 6.

10. In the control step, a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera is determined using six projection indices and six camera indices. The projection control method according to claim 9.

11. moreover, a marker imaging step of causing the camera to capture an image of a marker present in the projection range; a marker calculation step of calculating the camera coordinates of the marker captured by the camera in the camera coordinate system; a transformation step of transforming the camera coordinates of the markers into the projected coordinates using the determined correspondence; a third projection control step of causing the projector to project the image at the position of the marker captured by the camera using the converted projection coordinates of the marker; The projection control method according to any one of claims 1 to 10.

12. the camera image is a visible light image and the marker image is an infrared image; The projection control method according to claim 11.

13. The marker includes an infrared reflecting member.

13. The projection control method according to claim 11 or 12.

14. The marker is equipped with an LED (Light Emitting Diode) and emits light by itself. The projection control method according to any one of claims 11 to 13.

15. The method further includes a changing step of moving a camera target after changing the positions of the plurality of projection targets located within the projection range. The projection control method according to any one of claims 1 to 14.

16. A projection control device that controls the projection of an image by a projector, a first projection control unit that causes the projector to project a projection image including a plurality of projection indices positioned within a projection range of the projector; a detection control unit that causes a camera to capture the projected image so that the image fits within an imaging range; a display control unit that displays on a display an instruction for a user to arrange a plurality of camera indices, each indicating a relative position with respect to the projected indices in a camera coordinate system obtained by the imaging, in the imaging range; and a camera coordinate detection unit that detects the coordinates of the plurality of camera indices that are arranged; a control unit that determines a correspondence relationship between projection coordinates located within the projection range of the projector and camera coordinates located within the imaging range of the camera, in accordance with a relative positional relationship between the projection indices and the plurality of camera indices; Projection control device.

Images