Display method, program product, and information processing device

Abstract

A display method, a program product, and an information processing apparatus. A projection device (10) includes a projector (110) that projects an image onto a projection surface (SC), and a camera (120) that captures an image of the projection surface (SC). An information processing device (20) communicating with a projection device (10) displays a first superimposed image in which a first image, which is an image of a first captured image obtained by capturing a projection surface (SC) on which a first pattern is projected from a projector (110) by a camera (120), and a first instruction image indicating a first feature portion corresponding to an edge detected from the first image are superimposed. The information processing device (20) displays a second superimposed image in which a second image, which is an image of a second captured image obtained by capturing an image of a projection surface (SC) on which a second pattern is projected from the projector (110) by the camera (120), and a second instruction image indicating a second feature portion corresponding to an edge detected from the second image are superimposed.

Description

Technical Field

[0001] This disclosure relates to display methods, program products, and information processing apparatus. Background Technology

[0002] In the technology disclosed in Patent Document 1, a first captured image, including a first projected image and a projection surface, is acquired by an imaging device during a first period. Then, four first projection points at the four corners of the first projected image and four first feature points, representing feature points of the projection surface, are extracted from the first captured image. During a second period following the first period, a second captured image, including a second projected image and a projection surface, is acquired by an imaging device. Then, four second projection points at the four corners of the second projected image and four second feature points corresponding one-to-one with the four first feature points are extracted from the second captured image.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-092169

[0004] The accuracy of extracting the four first feature points and four second feature points varies depending on the type of pattern projected onto the projection surface. Previously, users could not determine which pattern would improve the accuracy of feature point extraction. Summary of the Invention

[0005] The first aspect of the display method disclosed herein includes: displaying a first overlapping image via a display device, the first overlapping image being obtained by overlaying an image based on a first captured image (i.e., a first image) obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface using a camera, the first indicating image representing a first feature portion extracted from the first image as a mark on the projection surface; and displaying a second overlapping image via the display device, the second overlapping image being obtained by overlaying an image based on a second captured image (i.e., a second image) obtained by capturing a second pattern projected from the projector onto the projection surface (different from the first pattern) and the projection surface using the camera, the second indicating image representing a second feature portion extracted from the second image as a mark on the projection surface.

[0006] Furthermore, the second aspect of the display method disclosed herein includes: displaying a first overlapping image via a display device, the first overlapping image being obtained by overlaying a first captured image (i.e., a first image) obtained by capturing a pattern projected from a projector onto a projection surface and the projection surface using a camera, the first indicating image representing a first feature portion extracted from the first image as a mark on the projection surface; and displaying a second overlapping image via the display device, the second overlapping image being obtained by overlaying a second captured image (i.e., a second image) obtained by capturing the projection surface illuminated by external light from a light source different from the projector using the camera, the second indicating image representing a second feature portion extracted from the second image as a mark on the projection surface.

[0007] Furthermore, one aspect of the program product of this disclosure enables a computer controlling a display device to execute the display of a first overlapping image and a second overlapping image via the display device. The first overlapping image is obtained by overlaying an image based on a first captured image (i.e., a first image) obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface using a camera, wherein the first indicating image represents a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying an image based on a second captured image (i.e., a second image) obtained by capturing a second pattern projected from the projector onto the projection surface (different from the first pattern) and the projection surface using the camera, wherein the second indicating image represents a second feature portion extracted from the second image as a mark on the projection surface.

[0008] Furthermore, another aspect of the program product of this disclosure enables a computer controlling a display device to execute the display of a first overlapping image and a second overlapping image via the display device. The first overlapping image is obtained by overlaying a first image (i.e., a first image) and a first indicator image obtained by taking a picture of a pattern projected from a projector onto a projection surface using a camera, wherein the first indicator image represents a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second image (i.e., a second image) and a second indicator image obtained by taking a picture of the projection surface illuminated by external light from a light source different from the projector using the camera, wherein the second indicator image represents a second feature portion extracted from the second image as a mark on the projection surface.

[0009] Furthermore, one aspect of the information processing apparatus of this disclosure includes a processing unit that displays a first overlapping image and a second overlapping image via the display device. The first overlapping image is obtained by overlaying a first captured image (i.e., a first image obtained by capturing a first pattern projected from a projector onto a projection surface and the projection surface using a camera) and a first indicator image, wherein the first indicator image represents a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second captured image (i.e., a second image obtained by capturing a second pattern projected from the projector onto the projection surface, which is different from the first pattern) and the projection surface using the camera, and a second indicator image, wherein the second indicator image represents a second feature portion extracted from the second image as a mark on the projection surface.

[0010] Furthermore, another aspect of the information processing apparatus disclosed herein includes a processing unit that displays a first overlapping image and a second overlapping image via a display device. The first overlapping image is obtained by overlaying a first captured image (i.e., a first image) obtained by capturing a pattern projected from a projector onto a projection surface using a camera, and the projection surface itself. The first indicating image represents a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second captured image (i.e., a second image) obtained by capturing the projection surface illuminated by external light from a light source different from the projector using the camera, and a second indicating image. The second indicating image represents a second feature portion extracted from the second image as a mark on the projection surface. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating a structural example of a system 1 containing an information processing apparatus 20 according to an embodiment of the present disclosure.

[0012] Figure 2 This is a diagram showing an example of the arrangement of the projection surface SC in this embodiment.

[0013] Figure 3 This is a diagram used to illustrate the second pattern PT1 in this embodiment.

[0014] Figure 4 This is a diagram illustrating an example of the brightness gradient in the second part A1 of the second pattern PT1.

[0015] Figure 5 This diagram illustrates the effect of reflected light from other surfaces near the projection surface SC.

[0016] Figure 6 This is a diagram showing an example of the first captured image G1, the second captured image G2, and the third captured image G3.

[0017] Figure 7 This is a diagram showing an example of an overlapping image.

[0018] Figure 8 This is a diagram showing an example of a notification image indicating a failure in feature point detection.

[0019] Figure 9 This is a flowchart illustrating the processing flow of the reference data setting method executed by the processing device 250 according to the program PR2.

[0020] Figure 10 This is a diagram used to illustrate the second pattern in variation example (1).

[0021] Figure 11 This is a diagram illustrating an example of the brightness gradient of the second part A2 of the second pattern in variant example (1).

[0022] Figure 12 This is a diagram illustrating an example of the brightness gradient of the second part A2 of the second pattern in variant example (2).

[0023] Figure 13 This is a diagram showing an example of the second pattern in variation (3).

[0024] Figure 14 This is a diagram showing an example of an enlarged image in variant example (6).

[0025] Label Explanation

[0026] 1 System; 10 Projection device; 20 Information processing device; 110 Projector; 120 Camera; 210 Display device; 220 Input device; 130, 230 Communication device; 140, 240 Storage device; 150, 250 Processing device; 250a Acquisition unit; 250b Detection unit; 250c Display control unit; 250d Setting unit; PR1, PR2 Program. Detailed Implementation

[0027] Various technically preferred limitations are attached to the embodiments described below. However, the embodiments of this disclosure are not limited to the methods described below.

[0028] A. Implementation Method

[0029] Figure 1This diagram illustrates a configuration example of a system 1 including an information processing apparatus 20 and a projection apparatus 10, representing one embodiment of the present disclosure. The projection apparatus 10 is a device for projecting images onto a projection surface SC. The projection apparatus 10 communicates with the information processing apparatus 20 via a communication network NW. In this embodiment, the communication network NW is a wireless communication network, but it can also be a wired communication network. Figure 2 As shown, the projection surface SC is set on the wall W1 of the room where the projection device 10 is located. Figure 2 As shown, the ceiling TS, floor BS, wall W2, and wall W3 of the aforementioned room are connected to wall W1, and the projection surface SC is located near the ceiling TS. Figure 1 As shown, the projection device 10 includes a projector 110, a camera 120, a communication device 130, a storage device 140, and a processing device 150. The projector 110, camera 120, communication device 130, and storage device 140 are respectively connected via a bus (in... Figure 1 (Illustration omitted) is connected to the processing device 150.

[0030] The processing device 150 includes one or more processors. Processors include, for example, a CPU (Central Processing Unit). The processing device 150 is an example of a computer in this disclosure. Details will be described later. The processing device 150 operates according to a program PR1 pre-stored in the storage device 140, thereby functioning as the control center of the projection device 10.

[0031] The projector 110 includes: a light source (internal light source); a generating unit that generates image light corresponding to image data provided from the processing device 150; and an optical system that guides the image light generated by the generating unit to the projection surface SC. Figure 1 The diagrams of the generation unit and optical system are omitted. Specific examples of the generation unit include display panels containing light modulation elements such as LCD (Liquid crystal display), LCOS (Liquid crystal on silicon), or DMD (Digital Mirror Device).

[0032] The camera 120 includes a CMOS or CCD image sensor. In this embodiment, the shooting area of ​​the camera 120 is preset to cover the entire projection area of ​​the image projected by the projector 110. The camera 120 captures images of the shooting area under the control of the processing device 150. The camera 120 outputs image data representing the captured image to the processing device 150.

[0033] The communication device 130 is wirelessly connected to the communication network NW. The communication device 130 includes an antenna and a wireless communication circuit for wirelessly communicating with the information processing device 20 via the communication network NW under the control of the processing device 150. The communication device 130 transmits data received from the information processing device 20 to the processing device 150, and conversely, sends data provided from the processing device 150 to the information processing device 20.

[0034] Storage device 140 includes non-volatile memory such as flash ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory). The non-volatile memory of storage device 140 stores a program PR1 for enabling processing device 150 to function as the control center of projection device 10. The non-volatile memory of storage device 140 is used by processing device 150 as a working area when executing program PR1. When the power supply to projection device 10 (in...) Figure 1 (Illustration omitted) When the device is turned on, the processing unit 150 reads the program PR1 from the non-volatile memory into the volatile memory and begins to execute the program PR1 read into the volatile memory.

[0035] According to procedure PR1, the processing unit 150, operating according to the instruction data received using the communication device 130, projects an image from the projector 110 and captures an image using the camera 120. Additionally, the processing unit 150, operating according to procedure PR1, uses the communication device 130 to send image data representing the image captured by the camera 120 to the information processing unit 20.

[0036] The information processing device 20 is a device that controls the image projection of the projector 110 and the image capture of the camera 120 according to the user's operation. In this embodiment, the information processing device 20 is a smartphone, but it can also be a tablet terminal. Figure 1 As shown, the information processing device 20 includes a display device 210, an input device 220, a communication device 230, a storage device 240, and a processing device 250. The display device 210, input device 220, communication device 230, and storage device 240 are respectively connected via a bus (in... Figure 1 (Illustration omitted) is connected to the processing device 250.

[0037] Processing device 250, like processing device 150, includes one or more processors. Processing device 250 operates according to a program PR2 pre-stored in storage device 240, thereby functioning as the control center of information processing device 20, as detailed later.

[0038] The display device 210 is, for example, a liquid crystal display (LCD). The display device 210 displays images under the control of the processing device 250. The input device 220 is, for example, a touch sensor provided to cover the display area of ​​the display device 210. When a user touches the input device 220, the input device 220 outputs data indicating the user's touch position to the processing device 250. Alternatively, the display device 210 may also be a projection-type display device like the projector 110. Furthermore, the projector 110 may also function as the display device 210.

[0039] Communication device 230 is wirelessly connected to the communication network NW. Like communication device 130, communication device 230 includes an antenna and wireless communication circuitry. Under the control of processing device 250, communication device 230 transmits data received from projection device 10 to processing device 250, and conversely, sends data provided by processing device 250 to projection device 10.

[0040] Storage device 240, like storage device 140, includes both non-volatile memory and volatile memory. The non-volatile memory of storage device 240 pre-stores a program PR2 for enabling processing device 250 to function as the control center of information processing device 20, and coordinate transformation data D. Coordinate transformation data D is data representing a transformation matrix that, for each of a plurality of adjustment points used to adjust the shape of an image projected from projector 110, transforms between a first coordinate in a first coordinate system defining the position on the display panel of projector 110 and a second coordinate in a second coordinate system defining the position on the image captured by camera 120. Coordinate transformation data D is appropriately generated, for example, using captured image data representing an image of a structured light projection surface SC from projector 110 projecting a sinusoidal pattern or the like, and image data representing that pattern, through known calibration. Coordinate transformation data D is an example of a correspondence in this disclosure. Coordinate transformation data D may also be stored in storage device 140. In this case, the coordinate transformation data D can be sent from the storage device 140 to the storage device 240 through the functions of the communication devices 130 and 230.

[0041] The non-volatile memory of storage device 240 is used by processing device 250 as the working area when executing program PR2. When an operation is performed to instruct the input device 220 to execute program PR2, such as pressing the icon corresponding to program PR2, processing device 250 reads program PR2 from non-volatile memory into volatile memory and begins executing program PR2 read from volatile memory. Processing device 250, which operates according to program PR2, functions as acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d.

[0042] The acquisition unit 250a acquires first captured image data representing the first captured image G1, which is obtained by the camera 120 capturing a first pattern projected from the projector 110 of the projection device 10 onto the projection surface SC and the projection surface SC. The first captured image G1 is an example of the first image in this disclosure. In this embodiment, the first pattern is a pure white image with uniform and constant brightness throughout the image. The acquisition unit 250a sends instruction data indicating the projection of uniform and constant white light to the projection device 10 using the communication device 230, thereby causing the projection device 10 to project the first pattern. Then, the acquisition unit 250a sends instruction data indicating the camera 120's capture and the transmission of the captured image to the projection device 10, which is in a state of projecting the first pattern onto the projection surface SC, and acquires the image data returned from the projection device 10 as the first captured image data.

[0043] Furthermore, the acquisition unit 250a acquires second image data representing the second captured image G2, which is obtained by using the camera 120 to capture a second pattern different from the first pattern and the projection surface SC projected from the projector 110 of the projection device 10 onto the projection surface SC. The second captured image G2 is an example of a second image in this disclosure. Figure 3 This is a diagram used to illustrate the second pattern PT1 in this embodiment. For example... Figure 3 As shown, the second pattern PT1 in this embodiment is an image that includes a first portion A0 and a strip-shaped second portion A1 disposed above the first portion A0. The first portion A0 in the second pattern PT1 is completely white, and the brightness of the first portion A0 is uniform and constant.

[0044] In part 2A1, in one example, the brightness continuously decreases in the direction from the center of the projection surface SC outwards, i.e., the first direction. In other words, part 2A1 appears blurry. Figure 3 In this embodiment, the first direction is represented by an arrow Y1. In part A1, the brightness decreases from K1 to K2 along the first direction. Figure 3 In the diagram, this reduction is represented by the arrow Y1 depicted with a dashed line. Figure 4 It shows from Figure 3 A graph showing the gradient of brightness reduction from position P1 to position P2. In this embodiment, in the first direction, the proportion of pixels in the second part A1 of the total number of pixels in the second pattern PT1 is approximately 5% to 10%. In the detection of the corner of the projection surface SC based on the first captured image G1, if the corner of the projection surface SC is close to other surfaces such as the ceiling TS, such as... Figure 5As shown, due to the influence of reflected light L from the projected light of other surfaces, the detection accuracy of corners near those other surfaces sometimes decreases. The second pattern PT1 is used to enable corner detection even when the corner of the projection surface SC is close to other surfaces such as the ceiling T, without being affected by reflected light from those other surfaces. In this embodiment, as... Figure 2 As shown, the ceiling TS side of the wall W1 is provided, and there is concern about the influence of reflected light from the ceiling TS during corner detection. Therefore, in this embodiment, the second part A1 is provided above the first part A0, i.e., on the ceiling TS side. In addition, in case of concerns about the influence of reflected light from surfaces other than the ceiling TS (wall W2, wall W3, and floor BS), an image of the second part A1 with a ring shape around the first part A0 can be used as the second pattern PT1.

[0045] Furthermore, the acquisition unit 250a acquires third captured image data representing the third captured image G3, which is obtained by capturing a projection surface SC illuminated by external light from a light source different from the projector 110 (external light source). An example of external light from a light source different from the projector 110 is ambient light in a room where the projection device 10 is installed. The acquisition unit 250a uses the communication device 230 to send instruction data instructing the projection device 10 to stop the projection of image light, thus stopping the projection device 10 from projecting image light. Then, the acquisition unit 250a sends instruction data instructing the camera 120 to capture images and transmit captured images to the projection device 10, which is in a state where the projection of image light is stopped—in other words, the projection device 10 is projecting a completely black image onto the projection surface SC—and acquires the image data returned from the projection device 10 as the third captured image data. As another example of creating an environment for photographing a projection surface SC illuminated by external light from an external light source, the output of the light source of the projector 110 can be reduced to zero or near zero, or the mechanical shutter of the projector 110 can be turned off.

[0046] The detection unit 250b detects multiple first feature points in the first captured image G1 that represent the corners (edges forming the outline of the projection surface SC) of the projection surface SC, based on the first captured image data. The corners of the projection surface SC in the first captured image G1 are an example of the first corners in this disclosure. Specifically, the detection unit 250b detects multiple first feature points representing the corners of the projection surface SC by applying a feature point detection algorithm to the first captured image G1. A feature point is an example of a feature portion. A feature portion can be a point or a line. The detection unit 250b can also detect multiple first feature points by applying an edge detection algorithm to the first captured image G1. The detection unit 250b is not limited to detecting multiple first feature points, i.e., multiple feature portions; it can also detect only one feature portion. For example, the detection unit 250b can also detect only one line or only one point as a feature portion corresponding to the edge by applying an edge detection algorithm to the first captured image G1. As described above, the first captured image G1 is obtained by capturing the projection surface SC, which is being projected with a pure white image, using the camera 120. In detecting the corners of the projection surface SC based on the first captured image G1, if the corners of the projection surface SC have a sufficiently large contrast ratio with their surroundings, it has the advantage of being able to detect the corners in most environments. On the other hand, in detecting the edges of the projection surface SC based on the first captured image G1, there is a disadvantage: when the area around the corners of the projection surface SC appears to be the same color during the projection of a pure white image (when the contrast between the corners and their surroundings is low), the corners sometimes cannot be detected. Furthermore, the detection unit 250b can also detect multiple first feature points representing the corners of the projection surface SC in the first captured image G1 based on the first captured image data and the third captured image data. For example, the detection unit 250b subtracts the pixel value of the corresponding pixel from the image represented by the third captured image data for each pixel contained in the image represented by the first captured image data, thereby calculating a difference image representing the difference between the two images.

[0047] Furthermore, the detection unit 250b detects multiple second feature points representing the corner of the projection surface SC in the second captured image G2 based on the second captured image data and the third captured image data. This corner of the projection surface SC in the second captured image G2 corresponds to the corner of the projection surface SC in the first captured image. For example, if the corner of the projection surface SC in the first captured image G1 is the upper left corner, then the corner of the projection surface SC in the second captured image G2 is the upper left corner of the upper left corner of the projection surface SC in the first captured image G1 or its vicinity. The corner of the projection surface SC in the second captured image G2 is an example of the second corner in this disclosure. In this embodiment, the detection unit 250b subtracts the pixel value of the pixel corresponding to each pixel in the image represented by the second captured image data from the image represented by the third captured image data, thereby calculating a difference image representing the difference between the two images. Through this processing, the noise around the corner of the projection surface SC in the second captured image G2 is removed, and only the effects caused by the projection of the second pattern appear in the differential image.

[0048] Next, the detection unit 250b extracts dark areas from the difference image to obtain information related to the corners of the projection surface SC. As an example of dark area extraction, a minimum value detection process can be cited. In this embodiment, the detection unit 250b applies a first-order differential filter and a second-order differential filter to the difference image, detecting the set of pixels whose first-order differential result is below a certain threshold and whose second-order differential result is above a certain threshold as pixels corresponding to the edges of the projection surface SC. In detecting the corners of the projection surface SC based on the second captured image G2, the following advantage is that the corners can be detected even when the contrast between the corners of the projection surface SC and their surroundings is low. However, in detecting the edges of the projection surface SC based on the second captured image G2, the following disadvantage exists: it can be used when there are surfaces around the projection surface SC that generate reflected light (e.g., a ceiling), but it cannot be used when there are no such surfaces.

[0049] Furthermore, the detection unit 250b detects multiple third feature points representing the edges of the projection surface SC in the third captured image G3 based on the third captured image data. Specifically, the detection unit 250b detects multiple third feature points representing the edges of the projection surface SC by performing known edge detection on the third captured image G3. In detecting the corners of the projection surface SC based on the third captured image G3, it has the advantage that if there is contrast between the corner and its surroundings, the corner can be detected. Additionally, the third feature points can also be used as markers outside the screen surface. The markers serve as a reference for correcting the position or shape of the image from the projector 110, such as the corners or edges of the projection surface SC, which define the characteristics of the projection surface SC. Therefore, the position and shape of the image from the projector 110 change according to the markers. Furthermore, the edge detection of the projection surface SC based on the third captured image G3 has the following drawbacks: corners may be falsely detected due to noise appearing in the image captured by the camera 120 or the influence of markings unrelated to the corners of the projection surface SC (e.g., patterns on the walls of the projection surface SC); and it cannot be used in a dark room where there is no ambient light. However, even when the projection surface SC is set in a dark room, if a rectangular, completely white image is projected onto the center of the projection surface SC and the exposure is consistent with that of the corners, it may be possible to perform edge detection of the projection surface SC based on the third captured image G3.

[0050] Before feature point detection by detection unit 250b, display control unit 250c arranges and displays the first captured image G1, the second captured image G2, and the third captured image G3 in their respective capturing order on display device 210 whenever the first captured image G1, the second captured image G2, and the third captured image G3 are captured. Figure 6 This diagram illustrates an example of the first captured image G1, the second captured image G2, and the third captured image G3 in this embodiment. (See diagram for example.) Figure 6 As shown, the display control unit 250c displays the first captured image G1, the second captured image G2, and the third captured image G3 side by side on the display device 210. Figure 6 In the image, the outline of the projection surface SC, which appears in the first image G1, the second image G2, and the third image G3, is depicted with dashed lines.

[0051] In this embodiment, when the first captured image G1 is displayed, the display control unit 250c displays a first setting image G4 on the display device 210, side by side, above the first captured image G1. This setting image G4 is used to set the first range (zoom of the camera) for the calculation of extracting the first feature point in the first captured image G1, and the exposure of the camera 120. The drag bar B1 in the first setting image G4 is a virtual operator for adjusting the zoom of the camera 120, and the drag bar B2 is a virtual operator for adjusting the exposure of the camera 120. The user, seeing the first captured image G1 and the first setting image G4 displayed on the display device 210, can adjust at least one of the zoom and exposure of the camera 120 by operating the first setting image G4, and instruct the first captured image G1 to be re-captured.

[0052] Furthermore, when displaying the second captured image G2, the display control unit 250c displays a second setting image G5, used to set the second range for the calculation of extracting the second feature point in the second captured image G2, side-by-side above the second captured image G2 in the display device 210. Drag bars B1 and B2 are also provided in the second setting image G5. Similarly, when displaying the third captured image G3, the display control unit 250c displays a fourth setting image G6, used to set the third range for the calculation of extracting the third feature point in the third captured image G3, side-by-side above the third captured image G3 in the display device 210. Drag bars B1 and B2 are also provided in the fourth setting image G6. Alternatively, the display control unit 250c may display a third setting image instead of the second setting image G5 and the fourth setting image G6 in the display device 210. This third setting image is used to set whether the second range for the calculation of extracting the second feature point in the second captured image G2 is set to the same range as the first range. Although the illustration is omitted, the third setting image is, for example, an image representing a checkbox or toggle switch. That is, the third setting image is a UI (User Interface) image that switches the function of setting the second range to the same range as the first range from one of being active or inactive to the other. In this case, when the processing device 250 receives an instruction from the user via the third setting image to set the second range to the same range as the first range, it simply sets the second range to the same range as the first range. Furthermore, one of the drag bars B1 and B2 can also be a function for adjusting the threshold used to detect multiple first feature points. In this case, the user can adjust the detection sensitivity of multiple first feature points by operating one of the drag bars B1 and B2, and can set a detection sensitivity appropriate for the user.

[0053] Furthermore, after the processing of the testing unit 250b, such as Figure 7As shown, the display control unit 250c displays a first indicator image G7 representing a first feature point superimposed on a first captured image G1 on the display device 210, thereby displaying a first overlaid image G8, obtained by overlaying the first indicator image G7 onto the first captured image G1, on the display device 210. Figure 7 As shown, the first indicator image G7 is a first position image representing the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the first feature point, that is, the position of the corner of the projection surface SC in the first captured image G1. In other words, the plurality of first indicator images G7 includes a first position image representing the position of the corner of the projection surface SC in the first captured image G1. One of the plurality of first indicator images G7, for example, the first indicator image G7 closest to the corner, can also be a first position image.

[0054] In addition, such as Figure 7 As shown, the display control unit 250c displays a second overlaid image G10 on the display device 210 by overlaying a second indicator image G9 representing the second feature point onto the second captured image G2. The second indicator image G9, like the first indicator image G7, is a second position image representing the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the second feature point, i.e., the position of the corner of the projection surface SC in the second captured image G2. That is, the plurality of second indicator images G9 includes a second position image representing the position of the corner of the projection surface SC in the second captured image G2. One of the plurality of second indicator images G9, for example, the second indicator image G9 closest to the corner, can also be a second position image.

[0055] In addition, such as Figure 7 As shown, the display control unit 250c displays a third overlaid image G12 on the display device 210 by overlaying the third indicator image G11, which represents the third feature point, onto the display device 210. The third indicator image G11, obtained by overlaying the third indicator image G11 onto the third captured image G3, is similar to the first indicator image G7 and the second indicator image G9. The third indicator image G11, like the first indicator image G7 and the second indicator image G9, is a third position image representing the position of each of the multiple points belonging to the point group detected by the detection unit 250b as the third feature point, i.e., the position of the corner of the projection surface SC in the third captured image G3. That is, the plurality of third indicator images G11 includes a third position image representing the position of the corner of the projection surface SC in the third captured image G3. One of the plurality of third indicator images G11, for example, the third indicator image G11 closest to the corner, can also be a third position image.

[0056] The display control unit 250c can also display a notification image on the display device 210 indicating that the detection of any one of the first, second, and third feature points has failed. For example, in the case of a failure to detect the first feature point, such as... Figure 8 As shown, the display control unit 250c displays a first notification image G13, containing a message indicating that the detection of the first feature point failed, superimposed on the first captured image G1 on the display device 210. Furthermore, in the event of a failure to detect the first feature point, the display control unit 250c can also display the aforementioned first setting image G4 to prompt the user to adjust the parameters of the camera 120. Additionally, if the area in the first captured image G1 that appears as a white spot or black spot occupies 80% or more of the overall area, the display unit 250c can display an image prompting the user to reduce or soften the exposure.

[0057] The setting unit 250d receives user operations performed on the input device 220 to select any one of the first overlapping image G8, the second overlapping image G10, and the third overlapping image G12. The user selects the image from the first overlapping image G8, the second overlapping image G10, and the third overlapping image G12 that is deemed most suitable for detecting the corner of the projection surface SC. To define the projection position of the image projected from the projector 110 on the projection surface SC, the setting unit 250d sets the coordinates of the corner of the projection surface SC in the first coordinate system based on the aforementioned operation. For example, when the operation of selecting the first overlapping image G8 is performed, the setting unit 250d uses coordinate transformation data D to transform the coordinates of the first feature point in the second coordinate system to the coordinates in the first coordinate system, thereby setting the coordinates of the corner of the projection surface SC in the first coordinate system. Then, the setting unit 250d stores data representing the coordinates of the corner of the projection surface SC in the first coordinate system as reference data in the storage device 140 of the projection device 10. This reference data is used to maintain (adjust) the position of the projected image relative to the projection surface SC at the position before the relative position change, even if the relative position of the projection device 10 relative to the projection surface SC changes. Furthermore, the setting unit 250d can handle operations for selecting at least two of the first overlapping image G8, the second overlapping image G10, and the third overlapping image G12. Therefore, for example, the setting unit 250d can also handle operations for selecting one of the first overlapping image G8 and the second overlapping image G10.

[0058] In addition, the processing device 250, which operates according to procedure PR2, executes a reference data setting method that includes a display method that prominently displays the features of this disclosure. Figure 9 This is a flowchart illustrating the processing flow in the baseline data setting method. For example... Figure 9As shown, the reference data setting method includes steps SA110 to SA190. Steps SA110 to SA170 correspond to the display method in this disclosure.

[0059] In step SA110, the processing device 250 functions as an acquisition unit 250a. In step SA110, the processing device 250 causes the projection device 10 to project the first pattern. In step SA120, following step SA110, the processing device 250 functions as both the acquisition unit 250a and the display control unit 250c. In step SA120, the processing device 250 instructs the camera 120 to capture images of the projection surface SC by instructing the projection device 10, which projects the first pattern onto the projection surface SC, thereby acquiring first captured image data. Then, in step SA120, the processing device 250 displays the first captured image G1 and the first set image G4 on the display device 210. Furthermore, if the parameters of the camera 120 are adjusted by instructing the operation on the first set image G4, the processing device 250 executes the processing of steps SA110 and SA120 again using the adjusted parameters.

[0060] In step SA130, following step SA120, processing device 250 functions as acquisition unit 250a. In step SA130, processing device 250 causes projection device 10 to project the second pattern. In step SA140, following step SA130, processing device 250 functions as both acquisition unit 250a and display control unit 250c. In step SA140, processing device 250 instructs camera 120 to capture the projection surface SC, which is projecting onto the projection surface SC of the second pattern, thereby acquiring second captured image data. Then, in step SA140, processing device 250 displays the second captured image G2 and the second setting image G5 on display device 210. Furthermore, if the parameters of camera 120 are adjusted by operating on the second setting image G5, processing device 250 executes the processing of steps SA130 and SA140 again using the adjusted parameters.

[0061] In step SA150, following step SA140, processing device 250 functions as acquisition unit 250a. In step SA150, processing device 250 instructs projection device 10 to stop projecting image light. In step SA160, following step SA150, processing device 250 functions as both acquisition unit 250a and display control unit 250c. In step SA160, processing device 250 instructs camera 120 to capture image data of projection surface SC by instructing projection device 10, which has stopped projecting image light, to do so. Then, in step SA160, processing device 250 displays the third captured image G3 and the fourth setting image G6 on display device 210. Furthermore, if the parameters of camera 120 are adjusted by operation on the fourth setting image G6, processing device 250 executes steps SA150 and SA160 again using the adjusted parameters.

[0062] In step SA170, executed after step SA160, the processing device 250 functions as a detection unit 250b. In step SA170, the processing device 250 detects multiple first feature points based on the first captured image data. Additionally, in step SA170, the processing device 250 detects multiple second feature points based on the second captured image data. Furthermore, in step SA170, the processing device 250 detects multiple third feature points based on the third captured image data. Moreover, in step SA170, if the detection of any one of the first, second, or third feature points fails, the processing device 250 may also function as a display control unit 250c, displaying a notification image to the user indicating that the feature point detection failed on the display device 210.

[0063] In step SA180, following step SA170, the processing unit 250 functions as a setting unit 250d. In step SA180, the processing unit 250 receives user operations performed on the input device 220 to select any one of the first overlapping image G8, the second overlapping image G10, and the third overlapping image G12. Then, to define the projection position of the image projected from the projector 110 on the projection surface SC, the processing unit 250 sets the coordinates of the corner of the projection surface SC in the first coordinate system based on the aforementioned operations.

[0064] According to this embodiment, the user can select the most appropriate detection result based on the detection results of the corners of the projection surface SC based on the first captured image G1, the second captured image G2, and the third captured image G3, respectively. Each of the corner detection methods based on the first captured image G1, the second captured image G2, and the third captured image G3 has its own advantages and disadvantages. For patterns that are not well-suited to one algorithm, other algorithms can be used to supplement them. Therefore, the corner detection accuracy is improved, the robustness related to maintaining the position of the projected image relative to the projection surface SC is improved, and the ability to withstand changes in the actual environment is increased. Furthermore, the parameters (zoom amount and exposure) of the camera 120 when capturing each pattern can be stored in the projection device 10. Thus, if the user selects a corner detection method suitable for the actual environment once, fine adjustments can be omitted in subsequent maintenance.

[0065] B. Deformation

[0066] The above embodiments can be modified as follows.

[0067] (1) As the second pattern, such as Figure 10 As shown, a second pattern PT2 can also be used, which has a first part A0 and a second part A2 arranged in an annular shape around the first part A0. Figure 11 This is a graph showing the increase in brightness from position P1 along the first direction to position P2 in Part 2, A2, from K2 back to K1. Figure 11 As shown, in part A2, the brightness increases continuously in the first direction, from the center of the projection surface SC outwards. Figure 10 In section A2, the continuous increase in brightness in a first direction from the center of the projection surface SC outward is shown by the arrow Y2 drawn with a dotted line. By projecting the second pattern PT2 with the projection surface SC located in section A0 of the second pattern in this embodiment, the influence of reflected light from other surfaces located near the corners of the projection surface SC can be mitigated in the same way as in the above embodiment.

[0068] (2) In the second part A1 of the second pattern in the above embodiment, as shown Figure 4 As shown, the brightness decreases continuously along the first direction from position P1 to position P2. However, as... Figure 12As shown, an image with a second portion whose brightness decreases progressively from position P1 to position P2 along the first direction from K1 to K2 can also be used as the second pattern PT1. By using the second pattern whose brightness decreases progressively from the center of the projection surface outwards, it is also possible to suppress the effects of secondary reflections from surfaces other than the projection surface on the corners and vicinity of the projection surface SC. Furthermore, the second portion A2 of the modified example (1) can also have a brightness variation portion with a progressively increasing brightness along the first direction from position P1 to position P2.

[0069] (3) Figure 13 As shown, a second pattern PT3 having second parts A1 and second parts A2 can also be used, with the second pattern PT3 projected in a region A3 between second parts A1 and second parts A2, sandwiching the corner of the projection surface SC. According to this method, the effects of secondary reflections from surfaces other than the projection surface (e.g., the ceiling) on ​​the corner of the projection surface and its vicinity can also be suppressed. Furthermore, the width of region A3 can be adjusted according to the positional relationship between the projection surface SC and surfaces other than the projection surface SC.

[0070] (4) In the above embodiment, the second pattern PT1 is a pure white image with a second portion A2 around it, but it can also be any monochrome image among R, G, and B with a second portion A2 around it. By using the second pattern PT1, which corresponds to the color of the wall of the room where the projection surface is set (e.g., a color in a complementary relationship), it is expected that the influence of secondary reflections from surfaces other than the projection surface on the corners and vicinity of the projection surface SC can be further suppressed. The second pattern in the modified example (1) is the same. In addition, when using any monochrome image among R, G, and B with a second portion A2 around it as the second pattern PT1, the detection unit 250b can also use different differential filters on the image according to the color.

[0071] (5) The distribution of the second part A1 in the second pattern PT1 can also be changed. Changing the distribution of the second part A1 means changing the range (i.e., width) of the second part A1 or changing the gradient of the brightness change in the second part A1. The wider the second part A2 and the gentler the gradient of the brightness change, the greater the margin for maintaining the position of the projected image relative to the projection surface SC. Conversely, when the gradient of the brightness change is steep, it is easy to detect the corners of the projection surface SC. In addition, when the camera 120 is used to photograph the projection surface SC on which the second pattern PT1 is projected, the detection unit 250b can change the differential filter according to at least one of the range and gradient of the distribution of the second part A1 in the photographed image. The distribution of the second part A2 can also be changed in the same way for the second pattern in the modified example (1).

[0072] (6) Alternatively, either the corner detection based on the first captured image G1 or the corner detection based on the second captured image G2 can be omitted, and corner detection based on either the first captured image G1 or the second captured image G2 and corner detection based on the third captured image G3 can be performed. When the corner detection based on the first captured image G1 is omitted, the second captured image G2 corresponds to the first captured image in this disclosure, and the second pattern in the above embodiment corresponds to the "pattern projected from the projector onto the projection surface" in this disclosure. Furthermore, when the corner detection based on the second captured image G2 is omitted, the first captured image G1 corresponds to the first captured image in this disclosure, and the first pattern in the above embodiment corresponds to the "pattern projected from the projector onto the projection surface" in this disclosure. Additionally, corner detection based on the third captured image G3 can also be omitted, and corner detection based on the first captured image G1 and the corner detection based on the second captured image G2 can be performed. According to this embodiment, the user can select the most appropriate detection result based on the detection results of the corners of the projection surface SC based on the first captured image G1, the second captured image G2, and the third captured image G3, respectively. If at least two of the corner detections based on the first captured image G1, the second captured image G2, and the third captured image G3 are performed, the corner detection accuracy is improved, the robustness related to maintaining the position of the projected image relative to the projection surface SC is improved, and the ability to withstand actual environmental changes is increased.

[0073] (7) Figure 14 As shown, when a user selects a point near the first corner in the first overlapping image G8 through a selection operation on the input device 220, the display control unit 250c can also display a first magnified angle image G20 containing a predetermined range of the selected point in the first captured image G1, and a first magnified position image G21 indicating the position of the first corner in the first magnified angle image G20, on the display device 210. The image containing the first magnified angle image G20 and the first magnified position image G21 can also be referred to as a first magnified overlapping image. This first magnified overlapping image is another example of the first overlapping image in this disclosure, or another first overlapping image distinct from the first overlapping image in this disclosure. The first magnified position image G21 is another example of a first position image. The user's selection operation indicating the image is, for example, an operation of selecting a point near the first corner in the first overlapping image G8 through the input device 220. Figure 14 In the image, points selected by the user are indicated by asterisks. The first magnified position image G21 is an image representing the contour line of the corner detected based on the first captured image G1. Figure 14The outline is represented by single-dash and double-dash lines. If the first corner is not included within the specified range containing the selected point, the display control unit 250c may also display an error notification image on the display device 210. Furthermore, although the illustration is omitted, the display control unit 250c may display only the dot image representing the point selected by the user (corresponding to the asterisk) as the first magnified position image G21 on the display device 210, or only the image representing the outline as the first magnified position image G21, or both the dot image and the image representing the outline as the first magnified position image G21 on the display device 210. In this case, the display control unit 250c may also only display the dot image representing the point selected by the user (corresponding to the asterisk). Therefore, since images other than the dot image are not displayed, it is possible to prevent the user from having difficulty determining which point was selected by the user. In addition, if the first magnified overlapping image is another first overlapping image that is distinct from the first overlapping image in this disclosure, the first magnified overlapping image may also be displayed simultaneously with the first overlapping image.

[0074] (8) Alternatively, at least one of the first magnified position image G21, i.e., the dot image and the image representing the outline, may not only be displayed on the display device 210, but may also be projected from the projector 110 onto the projection surface SC. For example, the processing device 150 may control the process as follows: based on the coordinate transformation data D, the position of a point selected by the user on the display device 210 may be transformed to the position on the display panel of the projector 110, and an image representing the transformed position may be projected, i.e., at least one of the dot image and the image representing the outline may be projected at or near the corner of the projection surface SC. Thus, at least one of the dot image and the image representing the outline may be projected onto the projection surface SC of the object, making it easy for the user to determine whether the corner of the projection surface SC of the object has been reliably determined.

[0075] (9) Alternatively, instead of displaying the first overlapping image G8, the second overlapping image G10 and the third overlapping image G12 in the display device 210, at least one of the dot image and the image representing the outline is projected from the projector 110 onto the corner of the projection surface SC.

[0076] (10) Alternatively, instead of displaying the first overlapping image G8, the second overlapping image G10 and the third overlapping image G12 on the display device 210 at the same time, they can be displayed one by one on the display device 210.

[0077] (11) At least one of the first indicator image G7, the second indicator image G9, or the third indicator image G11 may be an image representing an edge of the projection surface SC, or text or graphic icons representing the position of a corner or edge of the projection surface SC. That is, as long as the user can identify the marked feature portion (such as the first feature portion) in the projection surface SC, the first indicator image G7, the second indicator image G9, or the third indicator image G11 is not limited to directly representing a corner of the projection surface SC, but may also represent a corner of the projection surface SC indirectly, or may represent an edge of the projection surface SC, and its appearance is not particularly limited. For example, at least one of the first indicator image G7, the second indicator image G9, or the third indicator image G11 may be the intersection of two straight lines corresponding to two intersecting edges of the projection surface SC, that is, the intersection of two straight lines along the two edges. In this case, the intersection point may also represent a corner. In addition, the colors of the two edges may be different from each other. The edge of the projection surface SC may also be the boundary between the projection surface SC and an area outside the projection surface SC.

[0078] (12) In the above embodiments, the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d are software modules. However, at least one of the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d may also be a hardware module such as an ASIC (Application Specific Integrated Circuit). Even if at least one of the acquisition unit 250a, detection unit 250b, display control unit 250c, and setting unit 250d is a hardware module, it achieves the same effect as the above embodiments.

[0079] (13) Another example of the computer in this disclosure may be the processing device 250. Alternatively, it may be executed by the processing device 150. Figure 9 The flowchart shown represents one part of the process, while other processes are performed by the processing unit 250.

[0080] (14) Program PR2 may be made separately or provided for a fee or free of charge. Specific methods of providing program PR2 include writing and providing program PR2 in a computer-readable recording medium such as flash ROM, or providing program PR2 by downloading it via telecommunications lines such as the Internet.

[0081] C. Summary of this disclosure

[0082] This disclosure is not limited to the embodiments and variations described above, and can be implemented in various ways without departing from its spirit. For example, this disclosure can also be implemented in the following ways. In order to solve part or all of the problems of this disclosure, or to achieve part or all of the effects of this disclosure, the technical features in the above embodiments corresponding to the technical features in the various methods described below can be appropriately replaced or combined. In addition, if a technical feature is not described as a necessary feature in this specification, it can be appropriately deleted.

[0083] The following is a summary published in this note.

[0084] (Postscript 1)

[0085] The first method of displaying this invention includes: displaying a first overlapping image via a display device, the first overlapping image being obtained by overlaying an image of a first captured image (i.e., a first image obtained by capturing images of a first pattern projected from a projector onto a projection surface and the projection surface using a camera) with a first indicator image, the first indicator image representing a first feature portion extracted from the first image as a mark on the projection surface; and displaying a second overlapping image via the display device, the second overlapping image being obtained by overlaying an image of a second captured image (i.e., a second image obtained by capturing images of a second pattern projected from the projector onto the projection surface, which is different from the first pattern) with the projection surface using the camera with a second indicator image, the second indicator image representing a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first overlapping image and the second overlapping image.

[0086] (Postscript 2)

[0087] The second method of display disclosed herein includes: displaying a first overlapping image via a display device, the first overlapping image being obtained by overlaying a first image (i.e., a first image obtained by capturing a pattern projected from a projector onto a projection surface using a camera) with a first indicator image, the first indicator image representing a first feature portion extracted from the first image as a mark on the projection surface; and displaying a second overlapping image via the display device, the second overlapping image being obtained by overlaying a second image (i.e., a second image obtained by capturing a projection surface illuminated by external light from a light source different from the projector) with a second indicator image, the second indicator image representing a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first overlapping image and the second overlapping image.

[0088] (Note 3)

[0089] A more preferred method is the display method according to Appendix 1 or 2, wherein the first indication image is a first position image representing the position of the first corner of the projection surface in the first image, which is the first feature portion. According to this method, by confirming the first position image, the user can easily grasp the situation where the first corner of the projection surface is detected according to the first pattern, and can at least grasp that the corner of the projection surface can be detected according to the first pattern.

[0090] (Postscript 4)

[0091] A more preferred method is the display method described in Appendix 3, which includes: setting the coordinates of the first corner in the first coordinate system according to a correspondence between the first coordinate in the first coordinate system and the second coordinate in the second coordinate system. The first coordinate system defines the positions of each of a plurality of adjustment points used to adjust the shape of the image projected from the projector and projected onto the projection surface on the projector's drawing panel, and the second coordinate system defines the positions of each adjustment point on the image captured by the camera. According to this method, the second coordinates of each of the plurality of adjustment points can be set as the coordinates of the first corner in the first captured image, thus eliminating the need for the user to manually select the first corner from the first overlapping image, improving user convenience.

[0092] (Note 5)

[0093] A more preferred embodiment, according to the display method described in Appendix 3, is that the first overlapping image is an image obtained by magnifying the first corner of the first image and overlapping a first magnified position image, which is the first position image and indicates the position of the corner in the first magnified angle image. According to this embodiment, the visibility of the first corner is improved.

[0094] (Note 6)

[0095] A more preferred embodiment, according to the display method described in Appendix 1 or 2, is a first position image representing the position of a first corner of the projection surface in the first image, which is a first feature portion; and a second position image representing the position of a second corner of the projection surface in the second image, which is a second feature portion, corresponding to the first corner. The display method includes displaying, via the display device, a first overlapping image containing the first position image and a second overlapping image containing the second position image side-by-side. According to this embodiment, by displaying the first overlapping image and the second overlapping image side-by-side, it is easy to compare the extraction results of the first corner based on the first pattern and the extraction results of the second corner based on the second pattern.

[0096] (Note 7)

[0097] A more preferred embodiment is the display method according to any one of Appendix 1, Appendix 2, Appendix 3, Appendix 4, Appendix 5, and Appendix 6, wherein the display method comprises: displaying, via the display device, a first setting image for setting a first range in the first image for performing the operation of extracting the first feature portion; and displaying, via the display device, a second setting image for setting a second range in the second image for performing the operation of extracting the second feature portion. According to this embodiment, the first and second ranges can be arbitrarily set by the user, and feature portions can be extracted within the range desired by the user.

[0098] (Note 8)

[0099] A more preferred embodiment is the display method according to any one of Appendix 1, Appendix 2, Appendix 3, Appendix 4, Appendix 5, and Appendix 6, wherein the display method comprises: displaying, via the display device, a first setting image for setting a first range in the first image for performing the operation of extracting the first feature portion; and displaying, via the display device, a third setting image for setting whether, in the second image, the second range for performing the operation of extracting the second feature portion is set to the same range as the first range, wherein, upon receiving an instruction from the user via the third setting image to set the second range to the same range as the first range, the second range is set to the same range as the first range. According to this embodiment, the user can set the second range simply by setting the first range, thus improving ease of use for the user.

[0100] (Note 9)

[0101] A more preferred method is the display method according to any one of Appendix 1, Appendix 2, Appendix 3, Appendix 4, Appendix 5, Appendix 6, Appendix 7, and Appendix 8, wherein the display method comprises: displaying a first notification image notifying of the extraction failure of the first feature portion based on the first image via the display device in the event that extraction of the first feature portion fails. According to this method, the user can easily understand the situation where the detection of the first feature portion has failed.

[0102] (Postscript 10)

[0103] A more preferred embodiment is the display method according to any one of Appendix 1, Appendix 2, Appendix 3, Appendix 4, Appendix 5, Appendix 6, Appendix 7, Appendix 8, and Appendix 9, wherein the first indicating image is a first position image representing the position of a first corner of the projection surface in the first image as a first feature portion, and the second indicating image is a second position image representing the position of a second corner of the projection surface in the second image as a second feature portion, the second corner corresponding to the first corner. The display method includes: selecting, from a user acceptance, one of a first overlapping image containing the first position image and a second overlapping image containing the second position image; and, based on the operation, setting the corner of the projection surface that defines the projection position of the image projected from the projector on the projection surface. According to this embodiment, the user can set an angle preferred for themselves.

[0104] (Postscript 11)

[0105] One aspect of the program product disclosed herein enables a computer controlling a display device to execute the display of a first overlapping image and a second overlapping image via the display device. The first overlapping image is obtained by overlaying a first image (i.e., a first captured image obtained by using a camera to capture a first pattern projected from a projector onto a projection surface and the projection surface) with a first indicator image, the first indicator image representing a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second image (i.e., a second captured image obtained by using the camera to capture a second pattern projected from the projector onto the projection surface, different from the first pattern) with a second indicator image, the second indicator image representing a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first and second overlapping images.

[0106] (Postscript 12)

[0107] Another aspect of the program product disclosed herein enables a computer controlling a display device to execute the display of a first overlapping image and a second overlapping image via the display device. The first overlapping image is obtained by overlaying a first image (i.e., a first image captured by a camera to photograph a pattern projected from a projector onto a projection surface and the projection surface) with a first indicator image, the first indicator image representing a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second image (i.e., a second image captured by the camera to photograph the projection surface illuminated by external light from a light source different from the projector) with a second indicator image, the second indicator image representing a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first and second overlapping images.

[0108] (Postscript 13)

[0109] One aspect of the information processing apparatus disclosed herein includes a processing unit that displays a first overlapping image and a second overlapping image via a display device. The first overlapping image is obtained by overlaying a first image (i.e., a first captured image obtained by capturing a first pattern projected from a projector onto a projection surface using a camera) with a first indicator image, where the first indicator image represents a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second image (i.e., a second captured image obtained by capturing a second pattern projected from the projector onto the projection surface, which is different from the first pattern) with the second indicator image, where the second indicator image represents a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first and second overlapping images.

[0110] (Postscript 14)

[0111] Another aspect of the information processing apparatus disclosed herein includes a processing unit that displays a first overlapping image and a second overlapping image via a display device. The first overlapping image is obtained by overlaying a first image (i.e., a first image captured by a camera to photograph a pattern projected from a projector onto a projection surface) with a first indicator image, the first indicator image representing a first feature portion extracted from the first image as a mark on the projection surface. The second overlapping image is obtained by overlaying a second image (i.e., a second image captured by the camera to photograph the projection surface illuminated by external light from a light source different from the projector) with a second indicator image, the second indicator image representing a second feature portion extracted from the second image as a mark on the projection surface. According to this method, a user can easily determine which pattern is preferred for detecting the edge of the projection surface by referring to the first and second overlapping images.

[0112] The first image is not necessarily limited to the first captured image G1. The first image can instead be a diagram or line drawing, which is an image representing the first captured image G1. For example, a diagram or line drawing is a visually simplified representation of the first captured image G1. Diagrams and line drawings can be generated by the processing device 250. Specifically, an image generation tool for generating AI can be used to generate diagrams and line drawings based on the first captured image G1. Diagrams and line drawings can also be pre-prepared images. The first image used to overlap with the indicator image can be one of the first captured image G1, a diagram, or a line drawing, and the indicator image can be extracted based on another of these.

[0113] The second image is not limited to the second captured image G2. The second image can instead be a diagram or line drawing, which is an image representing the second captured image G2. For example, a diagram or line drawing is a visually simplified representation of the second captured image G2. Diagrams and line drawings can be generated by the processing device 250. Specifically, an image generation tool for generating AI can be used to generate diagrams and line drawings based on the second captured image G2. Diagrams and line drawings can also be pre-prepared images. The second image used for overlaying with the indicator image can be one of the second captured image G2, a diagram, or a line drawing, and the indicator image can be extracted based on another of these.

[0114] Note that steps or operations other than those necessary to solve at least one problem in this application may be omitted from each flowchart. For example, step SA160 may be omitted from each flowchart. Figure 9 The flowchart shown is omitted.

[0115] Note that the order of steps or operations within each flowchart can be changed, as long as it solves at least one problem in this application or maintains technical consistency. For example, the order of steps SA130 and SA140 can be relative to... Figure 9 The order of steps SA150 and SA160 in the flowchart shown is changed.

Claims

1. A display method characterized by comprising: The display method includes: displaying, by a display device, a first superimposed image obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a first pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image; and displaying, by the display device, a second superimposed image obtained by superimposing a second indication image on a second captured image obtained by capturing, using the camera, a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

2. A display method characterized by comprising: The display method includes: displaying, by a display device, a first superimposed image obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image; and displaying, by the display device, a second superimposed image obtained by superimposing a second indication image on a second captured image obtained by capturing, using the camera, the projection surface irradiated with external light from a light source different from the projector, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

3. The display method according to claim 1 or 2, wherein the first indication image is a first position image indicating a position of a first corner portion of the projection surface in the first captured image as the first characteristic portion.

4. The display method according to claim 3, wherein The display method includes: coordinates of the first corner portion in a first coordinate system that defines positions of each of a plurality of adjustment points for adjusting a shape of an image projected from the projector and projected onto the projection surface on a drawing panel of the projector, and a second coordinate system that defines positions of the each of the adjustment points on a captured image of the camera, in accordance with a correspondence relationship in which a first coordinate in the first coordinate system and a second coordinate in the second coordinate system correspond to each other.

5. The display method according to claim 3, wherein the first superimposed image is an image obtained by superimposing a first enlarged corner image obtained by enlarging the first corner portion in the first captured image and a first enlarged position image indicating a position of a corner in the first enlarged corner image as the first position image.

6. The display method according to claim 1 or 2, wherein the first indication image is a first position image indicating a position of a first corner portion of the projection surface in the first captured image as the first characteristic portion, the second indication image is a second position image as the second feature portion that indicates a position of a second corner of the projection surface in the second captured image, the second corner corresponds to the first corner, the display method includes: displaying, by the display device, the first superimposed image including the first position image and the second superimposed image including the second position image side by side.

7. The display method according to claim 1 or 2, wherein the display method includes: displaying, by the display device, a first setting image for setting a first range in which an operation of extracting the first feature portion is performed in the first captured image; and displaying, by the display device, a second setting image for setting a second range in which an operation of extracting the second feature portion is performed in the second captured image.

8. The display method according to claim 1 or 2, wherein the display method includes: displaying, by the display device, a first setting image for setting a first range in which an operation of extracting the first feature portion is performed in the first captured image; and displaying, by the display device, a third setting image for setting whether a second range in which an operation of extracting the second feature portion is performed in the second captured image is set to be the same range as the first range, in a case where an instruction to set the second range to be the same range as the first range is received from a user via the third setting image, setting the second range to be the same range as the first range.

9. The display method according to claim 1 or 2, wherein the display method includes: in a case where extraction of the first feature portion based on the first captured image fails, displaying, by the display device, a first notification image that notifies of the failure of extraction of the first feature portion.

10. The display method according to claim 1 or 2, wherein the first indication image is a first position image as the first feature portion that indicates a position of a first corner of the projection surface in the first captured image, the second indication image is a second position image as the second feature portion that indicates a position of a second corner of the projection surface in the second captured image, the second corner corresponds to the first corner, the display method includes: receiving, from a user, an operation of selecting one of the first superimposed image including the first position image and the second superimposed image including the second position image; and in accordance with the operation, setting a corner of the projection surface that delineates a projection position of an image projected from the projector in the projection surface.

11. A program product that causes a computer that controls a display device to perform display, by the display device, of a first superimposed image and a second superimposed image, ​ The first superimposed image is obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a first pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image, The second superimposed image is obtained by superimposing a second indication image on a second captured image obtained by capturing, using the camera, a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

12. A program product causing a computer that controls a display device to perform displaying, by the display device, a first superimposed image and a second superimposed image, The first superimposed image is obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image, The second superimposed image is obtained by superimposing a second indication image on a second captured image obtained by capturing, using the camera, the projection surface irradiated with external light from a light source different from the projector, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

13. An information processing apparatus having a processing apparatus that displays a first superimposed image and a second superimposed image by a display device, The first superimposed image is obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a first pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image, The second superimposed image is obtained by superimposing a second indication image on a second captured image obtained by capturing, using the camera, a second pattern different from the first pattern projected from the projector onto the projection surface and the projection surface, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

14. An information processing apparatus having a processing apparatus that displays a first superimposed image and a second superimposed image by a display device, The first superimposed image is obtained by superimposing a first indication image on a first captured image obtained by capturing, using a camera, a pattern projected from a projector onto a projection surface and the projection surface, the first indication image indicating a first characteristic portion extracted as a marker on the projection surface from the first captured image, The second superimposed image is obtained by superimposing a second indication image on a second captured image obtained by capturing the projection surface irradiated with external light from a light source different from the projector using the camera, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image. The second superimposed image is obtained by superimposing a second indication image on a second captured image obtained by capturing the projection surface irradiated with external light from a light source different from the projector using the camera, the second indication image indicating a second characteristic portion extracted as a marker on the projection surface from the second captured image.

Images