Display method, program, and information processing device

Abstract

The system allows users to easily determine which of several patterns is preferable for detecting the edges of the projection surface. [Solution] The projection device 10 includes a projector 110 that projects an image onto a projection surface SC, and a camera 120 that images the projection surface SC. The information processing device 20, which communicates with the projection device 10, displays a first superimposed image by superimposing a first instruction image, which shows a plurality of first feature portions corresponding to edges detected from the first image, onto a first image captured by the camera 120 of the projection surface SC onto which a first pattern is projected from the projector 110. The information processing device 20 also displays a second superimposed image by superimposing a second instruction image, which shows a plurality of second feature portions corresponding to edges detected from the second image, onto a second image captured by the camera 120 of the projection surface SC onto which a second pattern is projected from the projector 110.

Description

Technical Field

[0004] , , , ,

[0005] , , , , ,

[0001] The present disclosure relates to a display method, a program, and an information processing apparatus.

Background Art

[0002] In the technology disclosed in Patent Document 1, in a first period, a first imaging image including a first projection image and a projection plane is acquired by an imaging device. Next, four first projection points that are the four corners of the first projection image and four first feature points as feature points of the projection plane are extracted from the first imaging image. In a second period after the first period, a second imaging image including a second projection image and a projection plane is acquired by the imaging device. Then, four second projection points that are the four corners of the second projection image and four second feature points that correspond one-to-one with the four first feature points are extracted from the second imaging image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The accuracy of extracting the four first feature points and the four second feature points varies depending on the type of pattern projected onto the projection plane. Conventionally, users have not been able to grasp which pattern can improve the accuracy of extracting feature points.

Means for Solving the Problems

[0005] A first aspect of the display method of this disclosure is characterized by displaying a first superimposed image on a display device, which is obtained by superimposing a first indicator image showing a first feature portion as a marker on the projection surface, extracted based on the first image, onto a first captured image obtained by imaging the projection surface and a first pattern projected from a projector onto a projection surface with a camera; and displaying a second superimposed image on a display device, which is obtained by superimposing a second indicator image showing a second feature portion as a marker on the projection surface, extracted based on the second captured image, onto a second captured image obtained by imaging the projection surface and a second pattern different from the first pattern projected from the projector onto the projection surface with the camera.

[0006] Furthermore, the display method of the second embodiment of this embodiment is characterized by displaying a first superimposed image on a display device, which is obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first image obtained by imaging a pattern projected from a projector onto a projection surface and the projection surface itself; and displaying a second superimposed image on a display device, which is obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second image onto a second image obtained by imaging the projection surface illuminated by ambient light from a light source different from the projector.

[0007] Furthermore, a program according to one aspect of the present disclosure causes a computer controlling the display device to display a first superimposed image, which is obtained by superimposing a first instruction image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first captured image obtained by capturing a first pattern projected from a projector onto the projection surface and the projection surface itself, and a second superimposed image, which is obtained by superimposing a second instruction image showing a plurality of second feature portions as markers on the projection surface extracted based on the second captured image onto a second captured image obtained by capturing a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface itself.

[0008] Furthermore, a program according to another aspect of the present disclosure causes a computer controlling the display device to display, on the display device, a first superimposed image obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first captured image obtained by capturing a first pattern projected from a projector onto the projection surface and the projection surface itself, and a second superimposed image obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second captured image onto a second captured image obtained by capturing a second image of the projection surface illuminated by ambient light from a light source different from the projector.

[0009] Furthermore, an information processing device in one aspect of the present disclosure includes a processing device which displays, on a display device, a first superimposed image obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface, extracted based on the first image, onto a first captured image obtained by imaging the projection surface with a camera of a first pattern projected from a projector onto a projection surface, and a second superimposed image obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface, extracted based on the second image, onto a second captured image obtained by imaging the projection surface with a camera of a second pattern different from the first pattern projected from the projector onto the projection surface, and a second indicator image showing a plurality of second feature portions as markers on the projection surface, extracted based on the second image.

[0010] Furthermore, an information processing device according to another aspect of the present disclosure includes a processing device which displays, on a display device, a first superimposed image obtained by superimposing a first image obtained by imaging a first pattern projected from a projector onto a projection surface and the projection surface itself, onto a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image, and a second superimposed image obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second image, onto a second image obtained by imaging the projection surface illuminated by ambient light from a light source different from the projector. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example configuration of a system 1 including an information processing device 20 according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of the installation of the projection surface SC in this embodiment. [Figure 3] This is a diagram illustrating the second pattern PT1 in this embodiment. [Figure 4] This figure shows an example of the brightness gradient in the second part A1 of the second pattern PT1. [Figure 5] This diagram illustrates the influence of reflected light from other surfaces near the projection surface SC. [Figure 6] This figure shows examples of the display of the first captured image G1, the second captured image G2, and the third captured image G3. [Figure 7] This is a diagram showing an example of superimposed images. [Figure 8] This figure shows an example of a notification image displayed when feature point detection fails. [Figure 9] This flowchart shows the processing flow in the reference data setting method executed by the processing unit 250 according to program PR2. [Figure 10] This is a diagram illustrating the second pattern in modified example (1). [Figure 11]This figure shows an example of the brightness gradient in the second part A2 of the second pattern in modified example (1). [Figure 12] This figure shows an example of the brightness gradient in the second part A2 of the second pattern in modified example (2). [Figure 13] This figure shows an example of the second pattern in modified example (3). [Figure 14] This figure shows an example of an enlarged image in modified example (6). [Modes for carrying out the invention]

[0012] The embodiments described below are subject to various technically preferred limitations. However, the embodiments of this disclosure are not limited to those described below. A. Embodiment Figure 1 shows an example configuration of a system 1 including an information processing device 20 and a projection device 10 according to one embodiment of the present disclosure. The projection device 10 is a device that projects an image onto a projection surface SC. The projection device 10 communicates with the information processing device 20 via a communication network NW. In this embodiment, the communication network NW is a wireless communication network, but it may also be a wired communication network. The projection surface SC is installed on the wall W1 of the room where the projection device 10 is located, as shown in Figure 2. As shown in Figure 2, the wall W1 is connected to the ceiling TS, floor BS, wall W2 and wall W3 that partition the room, and the projection surface SC is installed near the ceiling TS. The projection device 10 includes a projector 110, a camera 120, a communication device 130, a storage device 140, and a processing device 150, as shown in Figure 1. Each of the projector 110, camera 120, communication device 130, and storage device 140 is connected to the processing device 150 via a bus (not shown in Figure 1).

[0013] The processing device 150 includes one or more processors. The processor includes, for example, a CPU (Central Processing Unit). The processing device 150 is an example of a computer in the present disclosure. Although details will be described later, the processing device 150 functions as a control center of the projection device 10 by operating according to the program PR1 stored in advance in the storage device 140.

[0014] The projector 110 includes a light source (internal light source), a generation unit that generates image light according to the image data given from the processing device 150, and an optical system that guides the image light generated by the generation unit to the projection surface SC. In FIG. 1, illustration of the generation unit and the optical system is omitted. Specific examples of the generation unit include a display panel including a light modulation element such as an LCD (Liquid crystal display), an LCOS (Liquid crystal on silicon), or a DMD (Digital Mirror Device).

[0015] The camera 120 includes a CMOS or CCD image sensor. In the present embodiment, the imaging area of the camera 120 is set in advance so as to cover the entire projection area where an image is projected by the projector 110. The camera 120 images the imaging area under the control of the processing device 150. The camera 120 outputs image data representing the captured image to the processing device 150.

[0016] The communication device 130 is wirelessly connected to the communication network NW. The communication device 130 includes an antenna and a wireless communication circuit for wirelessly communicating with the information processing device 20 via the communication network NW under the control of the processing device 150. The communication device 130 delivers the data received from the information processing device 20 to the processing device 150, while transmitting the data given from the processing device 150 to the information processing device 20.

[0017] The storage device 140 includes non-volatile memory such as flash ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory). The non-volatile memory of the storage device 140 stores program PR1, which causes the processing unit 150 to function as the control center for the projection device 10. The non-volatile memory of the storage device 140 is used by the processing unit 150 as a work area when executing program PR1. When the power to the projection device 10 (not shown in Figure 1) is turned on, the processing unit 150 reads program PR1 from the non-volatile memory to the volatile memory and starts executing program PR1 read into the volatile memory.

[0018] In accordance with program PR1, the processing unit 150 projects an image using the projector 110 and captures an image using the camera 120, according to the instruction data received via the communication device 130. Furthermore, the processing unit 150, operating according to program PR1, transmits image data representing the image captured by the camera 120 to the information processing unit 20 via the communication device 130.

[0019] The information processing device 20 is a device that controls the projection of images by the projector 110 and the capture of images by the camera 120 in response to user operations. In this embodiment, the information processing device 20 is a smartphone, but it may also be a tablet terminal. As shown in Figure 1, the information processing device 20 includes a display device 210, an input device 220, a communication device 230, a storage device 240, and a processing device 250. Each of the display device 210, the input device 220, the communication device 230, and the storage device 240 is connected to the processing device 250 via a bus (not shown in Figure 1).

[0020] The processing unit 250, like the processing unit 150, includes one or more processors. As will be described in detail later, the processing unit 250 functions as the control center of the information processing unit 20 by operating according to the program PR2 pre-stored in the storage device 240.

[0021] The display device 210 is, for example, a liquid crystal display. The display device 210 displays an image under the control of the processing device 250. The input device 220 is, for example, a touch sensor provided to cover the display area of ​​the display device 210. When a user touches the input device 220, the input device 220 outputs data indicating the user's touch position to the processing device 250. The display device 210 may also be a projection-type display device similar to the projector 110. Alternatively, the display device 210 may also function as the projector 110.

[0022] The communication device 230 is wirelessly connected to the communication network NW. The communication device 230 includes an antenna and a wireless communication circuit, similar to the communication device 130. Under the control of the processing unit 250, the communication device 230 passes data received from the projection device 10 to the processing unit 250, while transmitting data provided by the processing unit 250 to the projection device 10.

[0023] The storage device 240, like the storage device 140, includes non-volatile memory and volatile memory. The non-volatile memory of the storage device 240 pre-stores program PR2 and coordinate transformation data D for causing the processing unit 250 to function as the control center of the information processing unit 20. The coordinate transformation data D is data representing a transformation matrix for mutually converting between a first coordinate in a first coordinate system that defines the position on the display panel of the projector 110 and a second coordinate in a second coordinate system that defines the position on the image captured by the camera 120, for each of a plurality of adjustment points for adjusting the shape of the image projected from the projector 110. The coordinate transformation data D is appropriately generated by a well-known calibration using, for example, image data representing an image of a projection surface SC onto which structured light, such as a sine wave pattern, is projected from the projector 110, and image data representing the pattern. The coordinate transformation data D is an example of a correspondence in this disclosure. The coordinate transformation data D may be stored in the storage device 140. In this case, the coordinate transformation data D may be transmitted from the storage device 140 to the storage device 240 by the functions of the communication device 130 and the communication device 230.

[0024] The non-volatile memory of the storage device 240 is used by the processing unit 250 as a work area when executing program PR2. When an operation to instruct the input device 220 to execute program PR2 is performed, for example, by pressing the icon corresponding to program PR2, the processing unit 250 reads program PR2 from the non-volatile memory to the volatile memory and starts executing program PR2 read into the volatile memory. While operating according to program PR2, the processing unit 250 functions as an acquisition unit 250a, a detection unit 250b, a display control unit 250c, and a setting unit 250d.

[0025] The acquisition unit 250a acquires first image data representing the first captured image G1, which is acquired by imaging the first pattern projected onto the projection surface SC from the projector 110 of the projection device 10 and the projection surface SC with the camera 120. The first captured image G1 is an example of the first captured image in this disclosure. In this embodiment, the first pattern is an all-white image with uniform and constant brightness throughout the entire image. The acquisition unit 250a causes the projection device 10 to project the first pattern by transmitting instruction data to the projection device 10 using the communication device 230, which instructs the projection of white light with uniform and constant brightness. The acquisition unit 250a then transmits instruction data to the projection device 10, which is in the state of projecting the first pattern onto the projection surface SC, instructing the camera 120 to take images and transmit the captured images, and acquires the image data returned from the projection device 10 as the first image data.

[0026] Furthermore, the acquisition unit 250a acquires a second image data representing a second image G2, which is acquired by using the camera 120 to image the projection surface SC and a second pattern, which is different from the first pattern projected onto the projection surface SC from the projector 110 of the projection device 10. The second image G2 is an example of a second image in this disclosure. Figure 3 is a diagram for explaining the second pattern PT1 in this embodiment. As shown in Figure 3, the second pattern PT1 in this embodiment is an image that includes a first part A0 and a band-shaped second part A1 provided above the first part A0. The first part A0 in the second pattern PT1 is entirely white, and the brightness in the first part A0 is uniform and constant.

[0027] In the second part A1, the brightness continuously decreases in the first direction, which is an example of a direction moving outward from the center of the projection surface SC. In other words, the second part A1 appears blurred. In Figure 3, this is indicated by arrow Y1, which shows an example of the first direction. In this embodiment, in the second part A1, the brightness decreases from K1 to K2 along the first direction, and in Figure 3, this decrease is shown by drawing arrow Y1 as a dotted line. Figure 4 is a graph showing the gradient of the decrease in brightness from position P1 to position P2 in Figure 3. In this embodiment, in the first direction, the number of pixels in the second part A1 accounts for about 5-10% of the total number of pixels in the second pattern PT1. In the detection of corners of the projection surface SC based on the first captured image G1, when the corner of the projection surface SC is close to another surface such as the ceiling TS, the detection accuracy of corners close to the other surface may decrease due to the influence of reflected light L of the projected light from the other surface, as shown in Figure 5. The second pattern PT1 is used to enable the detection of corners of the projection surface SC without being affected by reflected light from other surfaces, such as the ceiling T, even when the corners of the projection surface SC are in close proximity to other surfaces. In this embodiment, as shown in Figure 2, it is provided on the ceiling TS side of wall W1, and there is concern about the influence of reflected light from ceiling TS when detecting corners. For this reason, in this embodiment, the second part A1 is provided above the first part A0, i.e., on the ceiling TS side. If there is concern about the influence of reflected light from surfaces other than ceiling TS (walls W2, wall W3, and floor surface BS), an image in which an annular second part A1 is provided around the first part A0 may be used as the second pattern PT1.

[0028] Furthermore, the acquisition unit 250a acquires a third image data representing a third image G3, which is acquired by imaging the projection surface SC illuminated by ambient light from a light source different from the projector 110 (an external light source). An example of ambient light from a light source different from the projector 110 is, for example, ambient light in the room where the projection device 10 is installed. The acquisition unit 250a transmits instruction data to the projection device 10 using the communication device 230 to instruct it to stop projecting image light, and the projection device 10 stops projecting image light. Then, the acquisition unit 250a transmits instruction data to the projection device 10, which is in a state where it has stopped projecting image light, in other words, the projection device 10 is projecting a completely black image onto the projection surface SC, instructing the camera 120 to take an image and transmit the captured image, and acquires the image data returned from the projection device 10 as the third image data. Another example of creating an environment for imaging a projection surface SC illuminated by ambient light from an external light source is to reduce the output of the projector 110's light source to zero or near zero, or to close the mechanical shutter of the projector 110.

[0029] The detection unit 250b detects a plurality of first feature points that indicate the corners (edges forming the contour of the projection surface SC) of the projection surface SC in the first captured image G1, based on the first captured image data. The corners of the projection surface SC in the first captured image G1 are an example of the first corners in this disclosure. Specifically, the detection unit 250b detects a plurality of first feature points that indicate the corners of the projection surface SC by applying a feature point detection algorithm to the first captured image G1. A feature point is an example of a feature portion. A feature portion may be a point or a line. The detection unit 250b may also detect a plurality of first feature points by applying an edge detection algorithm to the first captured image G1. As described above, the first captured image G1 is obtained by imaging the projection surface SC, which is projecting an all-white image, with the camera 120. Detecting the corners of the projection surface SC based on the first captured image G1 has the advantage that, if the corners of the projection surface SC and their surroundings have a sufficiently large contrast ratio, the corners can be detected in most environments. On the other hand, detecting the edges of the projection surface SC based on the first captured image G1 has the disadvantage that, when a full-white image is projected, the area near the corners of the projection surface SC may appear to be the same color (when the contrast between the corners and the surroundings is low), the corners may not be detected. The detection unit 250b may also detect a plurality of first feature points indicating the corners of the projection surface SC in the first captured image G1 based on the first captured image data and the third captured image data. For example, the detection unit 250b calculates a difference image representing the difference between the two images by subtracting the pixel value of the pixel corresponding to that pixel in the image represented by the third captured image data for each pixel included in the image represented by the first captured image data.

[0030] Furthermore, the detection unit 250b detects a plurality of second feature points indicating corners of the projection surface SC in the second image G2 based on the second image data and the third image data. The corner of the projection surface SC in the second image G2 corresponds to the corner of the projection surface SC in the first image. For example, if the corner of the projection surface SC in the first image G1 is the upper left corner, then the corner of the projection surface SC in the second image G2 is the upper left corner or its vicinity, corresponding to the upper left corner of the projection surface SC in the first image G1. The corner of the projection surface SC in the second image G2 is an example of a second corner in this disclosure. In this embodiment, the detection unit 250b calculates a difference image representing the difference between the two images by subtracting the pixel value of the pixel corresponding to that pixel in the image represented by the third image data for each pixel included in the image represented by the second image data. This process removes noise around the corners of the projection surface SC in the second image G2, so that only the effects caused by the second projection pattern appear in the difference image.

[0031] Next, the detection unit 250b obtains information about the corners of the projection surface SC by extracting dark areas from the difference image. One example of the process for extracting dark areas is local minimum detection. In this embodiment, the detection unit 250b applies a first-order differential filter and a second-order differential filter to the difference image and detects a set of pixels where "the result of the first derivative is below a certain threshold" and "the result of the second derivative is above a certain threshold" as pixels corresponding to the edges of the projection surface SC. Detecting the corners of the projection surface SC based on the second captured image G2 has the advantage that it can detect the corners even when the contrast between the corners of the projection surface SC and the surrounding area is low. However, detecting the edges of the projection surface SC based on the second captured image G2 has the disadvantage that it can be used when there is a surface that generates reflected light around the projection surface SC (e.g., a ceiling), but cannot be used when there is no surface that generates reflected light.

[0032] Furthermore, the detection unit 250b detects multiple third feature points indicating the edges of the projection surface SC in the third image G3 based on the third image data. Specifically, the detection unit 250b detects multiple third feature points indicating the edges of the projection surface SC by applying a well-known edge detection method to the third image G3. Detecting the corners of the projection surface SC based on the third image G3 has the advantage that if there is contrast between the corner of the projection surface SC and its surroundings, the corner can be detected. In addition, the third feature points can be used as markers even outside the screen surface. These markers serve as a reference when correcting the position and shape of the image from the projector 110, and are, for example, the corners or edges of the projection surface SC as described above, or other parts of the projection surface SC that define its characteristics. Therefore, the position and shape of the image from the projector 110 are changed according to the markers. Furthermore, detecting the edges of the projection surface SC based on the third image G3 has disadvantages, including the possibility of false detection of corners due to noise in the image captured by the camera 120 or landmarks unrelated to the corners of the projection surface SC (for example, patterns on the wall on which the projection surface SC is installed), and the inability to use it in a darkroom where ambient light is absent. However, even when the projection surface SC is installed in a darkroom, it may be possible to detect the edges of the projection surface SC based on the third image G3 by projecting a rectangular all-white image onto the center of the projection surface SC and adjusting the exposure to the corners.

[0033] Prior to the detection of feature points by the detection unit 250b, the display control unit 250c displays the first image G1, second image G2, and third image G3 on the display device 210 in the order in which they were captured, each time one of them is captured. Figure 6 shows an example of the display of the first image G1, second image G2, and third image G3 in this embodiment. As shown in Figure 6, the display control unit 250c displays the first image G1, second image G2, and third image G3 in order on the display device 210. In Figure 6, the outline of the projection surface SC captured in each of the first image G1, second image G2, and third image G3 is drawn with a dotted line.

[0034] In this embodiment, when the display control unit 250c displays the first captured image G1, it displays the first setting image G4, which sets the first range (camera zoom) for performing calculations to extract first feature points in the first captured image G1 and the exposure of the camera 120, on the display device 210, side by side above the first captured image G1. In the first setting image G4, seek bar B1 is a virtual operator for adjusting the zoom of the camera 120, and seek bar B2 is a virtual operator for adjusting the exposure of the camera 120. A user who views the first captured image G1 and the first setting image G4 displayed on the display device 210 can adjust at least one of the zoom and exposure of the camera 120 by operating on the first setting image G4 and instruct the system to recapture the first captured image G1.

[0035] When the display control unit 250c displays the second captured image G2, it displays a second setting image G5 above the second captured image G2 on the display device 210 for setting a second range for performing the calculation to extract second feature points in the second captured image G2. The second setting image G5 is also provided with seek bars B1 and B2. Similarly, when the display control unit 250c displays the third captured image G3, it displays a fourth setting image G6 above the third captured image G3 on the display device 210 for setting a third range for performing the calculation to extract third feature points in the third captured image G3. The fourth setting image G6 is also provided with seek bars B1 and B2. Alternatively, instead of displaying the second setting image G5 and the fourth setting image G6, the display control unit 250c may display a third setting image on the display device 210 for setting whether or not to set the second range for performing the calculation to extract second feature points in the second captured image G2 to the same range as the first range. Although not shown in the diagram, the third setting image is, for example, an image showing a checkbox or a toggle switch. That is, the third setting image is a UI (User Interface) image that switches the function of setting the second range to the same range as the first range from enabled and disabled to enabled and disabled. In this case, when the processing unit 250 receives an instruction from the user via the third setting image to set the second range to the same range as the first range, it should set the second range to the same range as the first range. Note that one of the seek bars B1 and B2 may be a function to adjust the threshold for detecting multiple first feature points. In this case, the user can adjust the detection sensitivity of multiple first feature points by operating one of the seek bars B1 and B2, and set it to an appropriate detection sensitivity for the user.

[0036] Furthermore, after processing by the detection unit 250b, the display control unit 250c superimposes a first instruction image G7 showing the first feature point onto the first captured image G1 and displays it on the display device 210, as shown in Figure 7, thereby displaying a first superimposed image G8, which is the first captured image G1 with the first instruction image G7 superimposed on it, on the display device 210. The first instruction image G7 is a first position image that shows the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the first feature point, i.e., the position of the corner of the projection surface SC in the first captured image G1, as shown in Figure 7. That is, the multiple first instruction images G7 include a first position image that shows the position of the corner of the projection surface SC in the first captured image G1. One of the multiple first instruction images G7, for example, the first instruction image G7 closest to the corner, may also be the first position image.

[0037] Furthermore, as shown in Figure 7, the display control unit 250c displays a second superimposed image G10 on the display device 210 by superimposing a second instruction image G9 showing the second feature point onto the second captured image G2. The second instruction image G9, like the first instruction image G7, is a second position image that shows the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the second feature point, i.e., the position of the corner of the projection surface SC in the second captured image G2. That is, the multiple second instruction images G9 include a second position image that shows the position of the corner of the projection surface SC in the second captured image G2. One of the multiple second instruction images G9, for example, the second instruction image G9 closest to the corner, may also be the second position image.

[0038] Furthermore, as shown in Figure 7, the display control unit 250c displays a third superimposed image G12 on the display device 210 by superimposing a third instruction image G11, which shows the third feature point, onto the third captured image G3, thereby displaying the third superimposed image G12 on the display device 210. The third instruction image G11, like the first instruction image G7 and the second instruction image G9, is a third position image that shows the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the third feature point, i.e., the position of the corner of the projection surface SC in the third captured image G3. That is, the multiple third instruction images G11 include a third position image that shows the position of the corner of the projection surface SC in the third captured image G3. One of the multiple third instruction images G11, for example, the third instruction image G11 closest to the corner, may also be the third position image.

[0039] If the display control unit 250c fails to detect any of the first, second, or third feature points, it may display a notification image on the display device 210 to inform the user that the feature point detection failed. For example, if the detection of the first feature point fails, the display control unit 250c displays a first notification image G13 containing a message indicating that the detection of the first feature point failed on the first captured image G1 on the display device 210, as shown in Figure 8. If the detection of the first feature point fails, the display control unit 250c may also prompt the user to adjust the parameters of the camera 120 by displaying the first setting image G4 mentioned above. Furthermore, if the area in the first captured image G1 that is either completely white or completely blacked out accounts for 80% or more of the total area, the display control unit 250c may display an image on the display device 210 with a message prompting the user to reduce or decrease the exposure.

[0040] The setting unit 250d receives user inputs made to the input device 220 to select one of the first superimposed image G8, the second superimposed image G10, and the third superimposed image G12. The user selects the image from the first superimposed image G8, the second superimposed image G10, and the third superimposed image G12 that they feel best detects the corners of the projection surface SC. The setting unit 250d sets the coordinates of the corners of the projection surface SC in the first coordinate system based on the above operation in order to define the projection position of the image projected from the projector 110 on the projection surface SC. For example, if the operation to select the first superimposed image G8 is performed, the setting unit 250d sets the coordinates of the corners of the projection surface SC in the first coordinate system by converting the coordinates of the first feature point in the second coordinate system to the coordinates in the first coordinate system using coordinate transformation data D. The setting unit 250d then stores data representing the coordinates of the corners of the projection surface SC in the first coordinate system in the storage device 140 of the projection device 10 as reference data to maintain (adjust) the position of the projected image on the projection surface SC to the position before the change in relative position, even if the relative position of the projection device 10 on the projection surface SC changes. The setting unit 250d only needs to accept an operation to select one of at least two images from the first superimposed image G8, the second superimposed image G10, and the third superimposed image G12. Therefore, for example, the setting unit 250d may accept an operation to select one of the first superimposed image G8 and the second superimposed image G10.

[0041] Furthermore, the processing unit 250, operating according to program PR2, executes a reference data setting method that includes a display method that prominently demonstrates the features of this disclosure. Figure 9 is a flowchart showing the processing flow in this reference data setting method. As shown in Figure 9, this reference data setting method includes the processes from step SA110 to step SA190. The processes from step SA110 to step SA170 correspond to the display method in this disclosure.

[0042] In step SA110, the processing unit 250 functions as an acquisition unit 250a. In step SA110, the processing unit 250 projects the first pattern onto the projection device 10. In step SA120, which follows step SA110, the processing unit 250 functions as an acquisition unit 250a and a display control unit 250c. In step SA120, the processing unit 250 acquires the first captured image data by instructing the projection device 10, which is projecting the first pattern onto the projection surface SC, to take an image of the projection surface SC with the camera 120. Then, in step SA120, the processing unit 250 displays the first captured image G1 and the first setting image G4 on the display device 210. If the camera 120 parameters are instructed to be adjusted by an operation on the first setting image G4, the processing unit 250 executes the processes of step SA110 and step SA120 again using the adjusted parameters.

[0043] In step SA130, which follows step SA120, the processing unit 250 functions as an acquisition unit 250a. In step SA130, the processing unit 250 projects the second pattern onto the projection device 10. In step SA140, which follows step SA130, the processing unit 250 functions as an acquisition unit 250a and a display control unit 250c. In step SA140, the processing unit 250 acquires the second image data by instructing the projection device 10, which is projecting the second pattern onto the projection surface SC, to take an image of the projection surface SC with the camera 120. Then, in step SA140, the processing unit 250 displays the second image G2 and the second setting image G5 on the display device 210. If the camera 120 parameters are instructed to be adjusted by an operation on the second setting image G5, the processing unit 250 executes the processes of steps SA130 and SA140 again using the adjusted parameters.

[0044] In step SA150, which follows step SA140, the processing unit 250 functions as an acquisition unit 250a. In step SA150, the processing unit 250 instructs the projection device 10 to stop projecting image light. In step SA160, which follows step SA150, the processing unit 250 functions as an acquisition unit 250a and a display control unit 250c. In step SA160, the processing unit 250 acquires a third image image by instructing the projection device 10, which has stopped projecting image light, to take an image of the projection surface SC with the camera 120. Then, in step SA160, the processing unit 250 displays the third image image G3 and the fourth setting image G6 on the display device 210. If the processing unit 250 is instructed to adjust the parameters of the camera 120 by an operation on the fourth setting image G6, the processing unit 250 executes steps SA150 and SA160 again using the adjusted parameters.

[0045] In step SA170, which is executed following step SA160, the processing unit 250 functions as a detection unit 250b. In step SA170, the processing unit 250 detects a plurality of first feature points based on the first captured image data. Also in step SA170, the processing unit 250 detects a plurality of second feature points based on the second captured image data. Also in step SA170, the processing unit 250 detects a plurality of third feature points based on the third captured image data. If the detection of any of the first, second, or third feature points fails in step SA170, the processing unit 250 may function as a display control unit 250c and display a notification image on the display device 210 to inform the user that the feature point detection failed.

[0046] In step SA180, which follows step SA170, the processing unit 250 functions as a setting unit 250d. In step SA180, the processing unit 250 receives user input to the input device 220 to select one of the first superimposed image G8, the second superimposed image G10, and the third superimposed image G12. Then, in order to define the projection position of the image projected from the projector 110 on the projection surface SC, the processing unit 250 sets the coordinates of the corners of the projection surface SC in the first coordinate system based on the above operation.

[0047] According to this embodiment, the user can select the most appropriate detection result for the corners of the projection surface SC based on the first image G1, the second image G2, and the third image G3. Each of the corner detection methods for the projection surface SC based on the first image G1, the second image G2, and the third image G3 has its own advantages and disadvantages. Since patterns that one algorithm struggles with can be complemented by other algorithms, the accuracy of corner detection is improved, the robustness of maintaining the position of the projected image on the projection surface SC is enhanced, and the variety of real-world environments that can be maintained increases. In addition, the projection device 10 may store the parameters of the camera 120 (zoom amount and exposure) used when capturing each pattern with the camera 120. This allows the user to select a corner detection method suitable for the real-world environment once, and then omit fine adjustments for subsequent maintenance.

[0048] B. Transformation The above embodiment can be modified as follows. (1) As a second pattern, a second pattern PT2 may be used, as shown in Figure 10, which has a first part A0 and an annular second part A2 provided around the first part A0. Figure 11 is a graph showing how the brightness in the second part A2 increases from K2 to K1 along the first direction from position P1 to position P2. As shown in Figure 11, in the second part A2, the brightness increases continuously in the first direction, which is the direction from the center of the projection surface SC outward. In Figure 10, the continuous increase in brightness in the second part A2 in the first direction from the center of the projection surface SC outward is shown by the dashed-dotted arrow Y2. By projecting the second pattern PT2 onto the projection device 10 so that the projection surface SC fits within the first part A0 of the second pattern of this embodiment, the influence of reflected light from other surfaces located near the corners of the projection surface SC can be mitigated, similar to the above embodiment.

[0049] (2) In the second part A1 of the second pattern in the above embodiment, as shown in Figure 4, the brightness decreased continuously along the first direction from position P1 to position P2. However, as shown in Figure 12, an image having a second part in which the brightness decreases in steps from K1 to K2 along the first direction from position P1 to position P2 may be used as the second pattern PT1. By using a second pattern in which the brightness decreases in steps from the center of the projection surface outwards, the influence of secondary reflection from surfaces other than the projection surface is suppressed from extending to the corners of the projection surface SC and its vicinity. Similarly, the second part A2 of the modified example (1) may also have a brightness change portion in which the brightness increases in steps along the first direction from position P1 to position P2.

[0050] (3) As shown in Figure 13, a second pattern PT3 comprising a second part A1 and a second part A2 may be used, and the second pattern PT3 may be projected so as to sandwich the corner of the projection surface SC into the region A3 between the second part A1 and the second part A2. In this embodiment as well, the influence of secondary reflection from surfaces other than the projection surface (e.g., the ceiling) on ​​the corner of the projection surface and its vicinity is suppressed. The size of region A3 may be adjusted depending on the positional relationship between the projection surface SC and surfaces other than the projection surface SC.

[0051] (4) In the above-described configuration, the second pattern PT1 was a fully white image with a second portion A2 around it, but it may also be a monochrome image of R, G, or B with a second portion A2 around it. By using a second pattern PT1 with a color corresponding to the color of the wall surface of the room where the projection surface is installed (for example, a complementary color), it is expected that the influence of secondary reflection from surfaces other than the projection surface will be further suppressed from reaching the corners of the projection surface SC and its vicinity. The same applies to the second pattern in modified example (1). When using a monochrome image of R, G, or B with a second portion A2 around it as the second pattern PT1, the detection unit 250b may use different differential filters on the image depending on the color.

[0052] (5) The distribution of the second part A1 in the second pattern PT1 may be changed. Changing the distribution of the second part A1 means changing the range (i.e., width) of the second part A1, or changing the gradient of the brightness change in the second part A1. The wider the second part A2 and the gentler the gradient of the brightness change, the wider the margin that can maintain the position of the projected image on the projection surface SC. Conversely, if the gradient of the brightness change is steep, it becomes easier to find the corners of the projection surface SC. Also, the projection surface SC onto which the second pattern PT1 is projected is imaged using the camera 120, and the detection unit 250b may change the differential filter according to at least one of the range and gradient of the distribution of the second part A1 in the captured image. Similarly, the distribution of the second part A2 may also be changed for the second pattern in the modified example (1).

[0053] (6) Corner detection based on the first captured image G1 and corner detection based on the second captured image G2 may be omitted, and corner detection based on either the first captured image G1 or the second captured image G2 and corner detection based on the third captured image G3 may be performed. If corner detection based on the first captured image G1 is omitted, the second captured image G2 corresponds to the first captured image in this disclosure, and the second pattern in the above embodiment corresponds to the "pattern projected from the projector onto the projection surface" in this disclosure. Also, if corner detection based on the second captured image G2 is omitted, the first captured image G1 corresponds to the first captured image in this disclosure, and the first pattern in the above embodiment corresponds to the "pattern projected from the projector onto the projection surface" in this disclosure. Also, corner detection based on the third captured image G3 may be omitted, and corner detection based on the first captured image G1 and corner detection based on the second captured image G2 may be performed. According to this embodiment, the user can select the most appropriate detection result from the corner detection results of the projection surface SC based on the first captured image G1, the second captured image G2, and the third captured image G3, respectively. If at least two of the corner detections based on the first captured image G1, the second captured image G2, and the third captured image G3 are performed, the accuracy of corner detection is increased, the robustness of maintaining the position of the projected image on the projection surface SC is improved, and the variety of real-world environments that can be maintained is increased.

[0054] (7) As shown in Figure 14, if a point near the first corner in the first superimposed image G8 is selected by a user selection operation on the input device 220, the display control unit 250c may display on the display device 210 a first magnified angle image G20, which is an enlarged version of a predetermined range including the selected point in the first captured image G1, and a first magnified position image G21, which shows the position of the first corner in the first magnified angle image G20. The image including the first magnified angle image G20 and the first magnified position image G21 may be referred to as the first magnified superimposed image. The first magnified superimposed image is another example of the first superimposed image in this disclosure, or another first superimposed image that is distinct from the first superimposed image in this disclosure. The first magnified position image G21 is another example of the first position image. The user selection operation of the indicated image is, for example, an operation to select the area near the first corner in the first superimposed image G8 on the input device 220. In Figure 14, the point selected by the user is indicated by a star. The first enlarged position image G21 is an image showing the contour line of a corner detected based on the first captured image G1, and in Figure 14, the contour line is shown as a dashed-dot line and a dashed-dot line. If the first corner is not included in a predetermined range including the selected point, the display control unit 250c may display a notification image to notify the user of the error on the display device 210. In addition, although not shown in the figure, the display control unit 250c may display only the point image showing the point corresponding to the star mark, which is the point selected by the user, as the first enlarged position image G21 on the display device 210, or it may display only the image showing only the contour line as the first enlarged position image G21 on the display device 210, or it may display both the point image and the image showing the contour line as the first enlarged position image G21 on the display device 210. In this case, the display control unit 250c may display only the point image showing the point corresponding to the star mark, which is the point selected by the user. This prevents images other than the point image from being displayed, thus preventing the user from having difficulty determining which point was selected by the user. Furthermore, if the first enlarged superimposed image is a different first superimposed image that is distinct from the first superimposed image in this disclosure, the first enlarged superimposed image may be displayed simultaneously with the first superimposed image.

[0055] (8) The first enlarged position image G21, that is, at least one of the point image and the image showing the contour lines, may not only be displayed on the display device 210 but may also be projected from the projector 110 onto the projection surface SC. For example, the processing device 150 converts the position of a point selected by the user on the display device 210 to a position on the display panel of the projector 110 based on the coordinate transformation data D, and controls the projection of an image representing the converted position, that is, at least one of the point image and the image showing the contour lines, onto the corner of the projection surface SC or its vicinity. As a result, at least one of the point image and the image showing the contour lines is projected onto the actual projection surface SC, making it easier for the user to understand whether the corner of the actual projection surface SC is clearly identified.

[0056] (9) Instead of the first superimposed image G8, the second superimposed image G10, and the third superimposed image G12 being displayed on the display device 210, at least one of the point image and the image showing the contour lines may be projected from the projector 110 onto the corner of the projection surface SC.

[0057] (10) The first superimposed image G8, the second superimposed image G10, and the third superimposed image G12 may not be displayed on the display device 210 simultaneously, but rather one by one sequentially.

[0058] (11) At least one of the first instruction image G7, the second instruction image G9, or the third instruction image G11 may be an image showing the edges of the projection surface SC, or a character or graphic icon showing the position of the corners or edges of the projection surface SC. In other words, as long as the user can recognize the characteristic part (such as the first characteristic part) as a landmark on the projection surface SC, the first instruction image G7, the second instruction image G9, or the third instruction image G11 is not limited to directly showing the corners of the projection surface SC, but may also be in a manner that indirectly shows the corners of the projection surface SC, or may be a manner that shows the edges of the projection surface SC, and the design is not particularly limited. For example, at least one of the first instruction image G7, the second instruction image G9, or the third instruction image G11 may be the intersection of two straight lines corresponding to two intersecting edges of the projection surface SC, that is, two straight lines along the two edges. In this case, the intersection may be a manner that shows a corner. Furthermore, the two edges may be of different colors. The edges of projection plane SC may also be the boundaries between projection plane SC and regions other than projection plane SC.

[0059] (12) In the above embodiment, the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d were software modules. However, at least one of the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d may be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if at least one of the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d is a hardware module, the same effects as in the above embodiment will be achieved.

[0060] (13) Another example of a computer in this disclosure may be a processing unit 250. Alternatively, some steps of the flowchart shown in Figure 9 may be performed by the processing unit 150 and other steps by the processing unit 250.

[0061] (14) Program PR2 may be manufactured as a standalone product and may be provided for a fee or free of charge. Specific ways of providing Program PR2 include providing it by writing it to a computer-readable recording medium such as flash ROM, or providing it by downloading it via a telecommunications line such as the Internet.

[0062] C. Summary of this disclosure This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms. The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate. A summary of this disclosure is provided below.

[0063] (Note 1) A display method according to a first aspect of this disclosure is characterized by displaying a first superimposed image on a display device, which is obtained by superimposing a first indicator image, which shows a first feature portion as a marker on the projection surface extracted based on the first image, onto a first captured image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface with a camera; and displaying a second superimposed image on a display device, which is obtained by superimposing a second indicator image, which shows a second feature portion as a marker on the projection surface extracted based on the second captured image, onto a second captured image obtained by capturing a second pattern, which is different from the first pattern projected from the projector onto the projection surface and the projection surface with the camera. According to this aspect, a user can easily determine which pattern is preferable for detecting the edge of the projection surface by referring to the first superimposed image and the second superimposed image.

[0064] (Note 2) The display method of the second embodiment of this embodiment is characterized by displaying a first superimposed image on a display device, which is obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first captured image obtained by capturing a pattern projected from a projector onto a projection surface and the projection surface with a camera; and displaying a second superimposed image on a display device, which is obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second captured image onto a second captured image obtained by capturing a projection surface illuminated by ambient light from a light source different from the projector with the camera. With this embodiment as well, the user can easily determine which pattern is preferable for detecting the edge of the projection surface by referring to the first superimposed image and the second superimposed image.

[0065] (Note 3) A more preferred embodiment of the display method is the display method described in (Note 1) or (Note 2), wherein the first indicator image is a first position image showing the position of the first corner of the projection surface in the first captured image, which is the first feature portion. According to this embodiment, by checking the first position image, the user can easily understand that the first corner of the projection surface has been detected by the first pattern, and can understand that the corner of the projection surface can be detected by at least the first pattern.

[0066] (Note 4) A further preferred embodiment of the display method is the display method described in (Appendix 3), which includes setting the coordinates of the first corner in the first coordinate system based on a correspondence relationship that associates a first coordinate in a first coordinate system that defines the position on the projector's drawing panel for each of the multiple adjustment points for adjusting the shape of the image projected from the projector and projected from the projector onto the projection surface with a second coordinate in a second coordinate system that defines the position on the image captured by the camera. According to this embodiment, since the second coordinate of each of the multiple adjustment points can be set as the coordinate of the first corner in the first captured image, the step of the user manually selecting the first corner from the first superimposed image can be omitted, improving user convenience.

[0067] (Note 5) Another preferred embodiment of the display method is the display method described in (Appendix 3), in which the first superimposed image is an image obtained by superimposing a first enlarged angle image obtained by enlarging the first corner portion of the first captured image and a first enlarged position image which is the first position image and indicates the position of the corner in the first enlarged angle image. According to this embodiment, the visibility of the first corner portion is improved.

[0068] (Note 6) Another preferred embodiment of the display method is the display method described in (Note 1) or (Note 2), wherein the first indicator image is a first position image showing the position of the first corner of the projection surface in the first captured image as the first feature portion, the second indicator image is a second position image showing the position of the second corner of the projection surface in the second captured image as the second feature portion, the second corner corresponds to the first corner, and the display method includes displaying the first superimposed image including the first position image side by side with the second superimposed image including the second position image using the display device. According to this embodiment, by displaying the first superimposed image and the second superimposed image side by side, it becomes easier to compare the extraction result of the first corner using the first pattern with the extraction result of the second corner using the second pattern.

[0069] (Note 7) A further preferred embodiment of the display method is the display method described in any of (Note 1), (Note 2), (Note 3), (Note 4), (Note 5), or (Note 6), which includes displaying a first setting image on the display device for setting a first range for performing an operation to extract the first feature portion in the first captured image, and displaying a second setting image on the display device for setting a second range for performing an operation to extract the second feature portion in the second captured image. According to this embodiment, the first range and the second range can be arbitrarily set by the user, and feature portions can be extracted within the range desired by the user.

[0070] (Note 8) A further preferred embodiment of the display method is the display method described in any of (Note 1), (Note 2), (Note 3), (Note 4), (Note 5), or (Note 6), in which the user can set the second range for which the calculation of extracting the first feature portion in the first captured image is performed, by displaying a first setting image on the display device, and the user can set the second range for which the calculation of extracting the second feature portion in the second captured image is set to the same range as the first range, and when the user gives an instruction to set the second range to the same range as the first range via the third setting image, the second range is set to the same range as the first range. According to this embodiment, the user can set the second range by setting only the first range, thus improving usability for the user.

[0071] (Note 9) A further preferred embodiment of the display method is the display method described in any of (Note 1), (Note 2), (Note 3), (Note 4), (Note 5), (Note 6), (Note 7), or (Note 8), which includes displaying a first notification image on the display device indicating that the extraction of the first feature portion based on the first captured image has failed. According to this embodiment, the user can easily understand that the detection of the first feature portion has failed.

[0072] (Note 10) A further preferred embodiment of the display method is the display method described in any of (Note 1), (Note 2), (Note 3), (Note 4), (Note 5), (Note 6), (Note 7), (Note 8), or (Note 9), which includes: the first indicator image being a first position image showing the position of the first corner of the projection surface in the first captured image as the first feature portion; the second indicator image being a second position image showing the position of the second corner of the projection surface in the second captured image as the second feature portion, the second corner corresponding to the first corner; and the display method receiving an operation from the user to select one of the first superimposed image including the first position image and the second superimposed image including the second position image; and setting the corner of the projection surface that defines the projection position of the image projected from the projector on the projection surface based on the operation. According to this embodiment, the user can set a corner that is preferable to them.

[0073] (Note 11) A program in one aspect of the present disclosure causes a computer controlling a display device to display a first superimposed image, which is obtained by superimposing a first instruction image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first captured image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface with a camera, and a second superimposed image, which is obtained by superimposing a second instruction image showing a plurality of second feature portions as markers on the projection surface extracted based on the second captured image onto a second captured image obtained by capturing a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface with the camera. In this aspect as well, the user can easily determine which pattern is preferable for detecting the edges of the projection surface by referring to the first superimposed image and the second superimposed image.

[0074] (Note 12) A program according to another aspect of the present disclosure causes a computer controlling the display device to display a first superimposed image, which is obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first captured image obtained by imaging the projection surface and a pattern projected from a projector onto the projection surface with a camera, and a second superimposed image, which is obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second captured image onto a second captured image obtained by imaging the projection surface with a camera when ambient light from a light source different from the projector is irradiated onto the projection surface. In this aspect as well, the user can easily determine which pattern is preferable for detecting the edges of the projection surface by referring to the first superimposed image and the second superimposed image.

[0075] (Note 13) An information processing device in one embodiment of the present disclosure includes a processing device which displays, on a display device, a first superimposed image obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface, extracted based on the first image, onto a first captured image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface with a camera, and a second superimposed image obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface, extracted based on the second image, onto a second captured image obtained by capturing a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface with the camera. In this embodiment as well, the user can easily determine which pattern is preferable for detecting the edges of the projection surface by referring to the first superimposed image and the second superimposed image.

[0076] (Note 14) An information processing device according to another aspect of the present disclosure includes a processing device which displays on a display device a first superimposed image obtained by superimposing a first indicator image showing a plurality of first feature portions as markers on the projection surface extracted based on the first image onto a first image obtained by imaging the projection surface and a pattern projected from a projector onto a projection surface with a camera, and a second superimposed image obtained by superimposing a second indicator image showing a plurality of second feature portions as markers on the projection surface extracted based on the second image onto a second image obtained by imaging the projection surface with a camera when ambient light from a light source different from the projector is irradiated onto the projection surface. In this aspect as well, the user can easily determine which pattern is preferable for detecting the edges of the projection surface by referring to the first superimposed image and the second superimposed image. [Explanation of Symbols]

[0077] 1...System, 10...Projection device, 20...Information processing device, 110...Projector, 120...Camera, 210...Display device, 210...Input device, 130, 230...Communication device, 140, 240...Storage device, 150, 250...Processing device, 250a...Acquisition unit, 250b...Detection unit, 250c...Display control unit, 250d...Setting unit, PR1, PR2...Program.

Claims

1. A first superimposed image is displayed on a display device by superimposing a first image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface itself with a camera, onto a first image obtained by superimposing a first indicator image that shows a first feature portion as a marker on the projection surface, which is extracted based on the first image. The display device displays a second superimposed image obtained by superimposing a second indicator image, which shows a second feature portion as a marker on the projection surface extracted based on the second image, onto a second image obtained by capturing the projection surface and a second pattern different from the first pattern projected from the projector onto the projection surface with the camera, A method of display characterized by including

2. A first superimposed image is displayed on a display device by superimposing a first image obtained by capturing a pattern projected from a projector onto a projection surface and the projection surface itself with a camera, onto a first indicator image which shows a plurality of first feature portions as markers on the projection surface extracted based on the first image. The display device displays a second superimposed image obtained by superimposing a second indicator image, which shows a plurality of second feature portions as markers on the projection surface extracted based on the second image, onto a second image obtained by capturing the projection surface with the camera when the projection surface is illuminated with ambient light from a light source different from the projector, A method of display characterized by including

3. The display method according to claim 1 or claim 2, wherein the first indicator image is a first position image showing the position of the first corner of the projection surface in the first captured image, which is the first feature portion.

4. The display method according to claim 3, further comprising setting the coordinates of the first corner in the first coordinate system based on a correspondence relationship that associates a first coordinate in a first coordinate system defining the position on the drawing panel of the projector for each of the plurality of adjustment points for adjusting the shape of the image projected from the projector and projected from the projector onto the projection surface with a second coordinate in a second coordinate system defining the position on the image captured by the camera.

5. The first superimposed image is A first magnified angle image obtained by magnifying the first corner portion of the first captured image, The first position image is a first enlarged position image that shows the position of the corner in the first enlarged angle image, The display method according to claim 3, which is an image in which the two are superimposed.

6. The first indicator image is a first position image that shows the position of the first corner of the projection surface in the first captured image, which is the first feature portion. The second indicator image is a second position image that shows the position of the second corner of the projection surface in the second captured image, which is the second feature portion. The second corner corresponds to the first corner, The aforementioned display method is, The display method according to claim 1 or claim 2, comprising displaying the first superimposed image, which includes the first position image, alongside the second superimposed image, which includes the second position image, using the display device.

7. A first setting image for setting a first range for performing calculations to extract the first feature portion in the first captured image is displayed by the display device, The display device displays a second setting image for setting a second range for performing calculations to extract the second feature portion in the second captured image, A display method according to claim 1 or claim 2, including the method described in claim 1 or 2.

8. A first setting image for setting a first range for performing calculations to extract the first feature portion in the first captured image is displayed by the display device, The process includes displaying a third setting image on the display device for determining whether or not to set the second range for performing the calculation to extract the second feature portion in the second captured image to the same range as the first range, When the system receives an instruction from the user via the third setting image to set the second range to the same range as the first range, the system sets the second range to the same range as the first range. The display method according to claim 1 or claim 2.

9. If the extraction of the first feature portion based on the first captured image fails, the display device will display a first notification image indicating that the extraction of the first feature portion has failed. The display method according to claim 1 or claim 2.

10. The first indicator image is a first position image that shows the position of the first corner of the projection surface in the first captured image, which is the first feature portion. The second indicator image is a second position image that shows the position of the second corner of the projection surface in the second captured image, which is the second feature portion. The second corner corresponds to the first corner, The aforementioned display method is, The system accepts an operation from the user to select one of the first superimposed image including the first position image and the second superimposed image including the second position image, Based on the operation, the corners of the projection surface that define the projection position of the image projected from the projector are set. A display method according to claim 1 or claim 2, including the following:

11. To the computer that controls the display device, A first superimposed image is obtained by superimposing a first image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface with a camera, onto a first indicator image which shows a plurality of first feature portions as markers on the projection surface extracted based on the first image, A second superimposed image is obtained by superimposing a second image, which is obtained by imaging the projection surface and a second pattern different from the first pattern projected from the projector onto the projection surface with the camera, onto a second image, which is obtained by superimposing a second indicator image that shows a plurality of second feature portions as markers on the projection surface, extracted based on the second image, on the second image, A program that causes the display device to display the above.

12. To the computer that controls the display device, A first superimposed image is obtained by superimposing a first image obtained by capturing a pattern projected from a projector onto a projection surface and the projection surface itself with a camera, onto a first indicator image which shows a plurality of first feature portions as markers on the projection surface extracted based on the first image, A second superimposed image is obtained by superimposing a second image, which is obtained by imaging the projection surface illuminated by ambient light from a light source different from the projector using the camera, onto a second image, and a second indicator image is obtained by superimposing a second indicator image that shows a plurality of second feature portions as markers on the projection surface, which are extracted based on the second image. A program that causes the display device to display the above.

13. Equipped with a processing device, The aforementioned processing apparatus is A first superimposed image is obtained by superimposing a first image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface with a camera, onto a first indicator image which shows a plurality of first feature portions as markers on the projection surface extracted based on the first image, A second superimposed image is obtained by superimposing a second image, which is obtained by imaging the projection surface and a second pattern different from the first pattern projected from the projector onto the projection surface with the camera, onto a second image, which is obtained by superimposing a second indicator image that shows a plurality of second feature portions as markers on the projection surface, extracted based on the second image, on the second image, An information processing device that displays information using a display device.

14. Equipped with a processing device, The aforementioned processing apparatus is A first superimposed image is obtained by superimposing a first image obtained by capturing a pattern projected from a projector onto a projection surface and the projection surface itself with a camera, onto a first indicator image which shows a plurality of first feature portions as markers on the projection surface extracted based on the first image, An information processing device that displays, on a display device, a second superimposed image obtained by superimposing a second image obtained by imaging the projection surface, which is illuminated by ambient light from a light source different from the projector, with the camera, onto a second image obtained by superimposing a second indicator image that shows a plurality of second feature portions as markers on the projection surface, extracted based on the second image.

Images