Display method, information processing device, and program

Abstract

The user understands whether the points extracted as corners of the projection surface are the same or nearly the same over two different time periods. [Solution] The display method includes displaying a first image on the display device 130 that shows the position on the projection surface PF in the first image of a feature portion that serves as a marker for the projection surface PF, which is extracted based on a first image obtained by imaging the projection surface PF with the imaging device 320 in a first period, and displaying a second image on the display device 130 that shows the position on the projection surface PF in the second image of a feature portion that serves as a marker for the projection surface PF, which is extracted based on a second image obtained by imaging the projection surface PF with the imaging device 320 in a second period after the first period.

Description

Technical Field

[0001] The present invention relates to a display method, an information processing apparatus, and a program.

Background Art

[0002] Conventionally, techniques have been used to suppress the deviation of the projection position of a projection image with respect to a projection plane from an initial position.

[0003] For example, Patent Document 1 discloses a projector that corrects a projection image to a projection image projected at an initial position in a first period based on a first mapping relationship, a second mapping relationship, and projection conversion parameters. In the first period, the projector extracts, from a first captured image, four first projection points that are the four corners of a first projection image and four first feature points as feature points of the projection plane. Also, in a second period after the first period, the projector extracts, from a second captured image, four second projection points that are the four corners of a second projection image and four second feature points as feature points of the projection plane. The first mapping relationship is a relationship between a first projection point and a first corresponding point corresponding to the first projection point on a display panel. The second mapping relationship is a relationship between a second projection point and a second corresponding point corresponding to the second projection point on the display panel. The projection conversion parameters are parameters for performing projection conversion from the first feature points to the second feature points.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The projector described in Patent Document 1 did not provide a means for the user to determine whether the points extracted as corners of the projection surface were the same or nearly the same in two different time periods. [Means for solving the problem]

[0006] A display method according to an aspect of the present invention includes displaying a first image on a display device that shows the position on the projection surface of a feature portion that serves as a marker for the projection surface, which is extracted based on a first image obtained by imaging the projection surface with an imaging device in a first period, and displaying a second image on a display device that shows the position on the projection surface of a feature portion that serves as a marker for the projection surface, which is extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period, in a second image.

[0007] An information processing device according to an aspect of the present invention includes a first display control unit that displays a first image on a display device showing the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a first image obtained by imaging the projection surface with an imaging device in a first period; and a second display control unit that displays a second image on a display device showing the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a second image obtained by imaging the projection surface with the camera in a second period following the first period.

[0008] A program according to an aspect of the present invention causes a processor to perform the following actions: display a first image on a display device, which indicates the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a first image obtained by imaging the projection surface with an imaging device in a first period; and display a second image on a display device, which indicates the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a second image obtained by imaging the projection surface with the camera in a second period following the first period. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the overall configuration of projection system 1. [Figure 2] A block diagram showing an example configuration of the projection device 30[1]. [Figure 3] A block diagram showing an example configuration of the information processing device 10. [Figure 4] An explanatory diagram showing the operation of the processing unit 110 and the storage device 120 during the construction of projection system 1. [Figure 5] An explanatory diagram showing the operation of the processing unit 110 and the storage device 120 during the restoration of the projection system 1. [Figure 6] Functional block diagram of the display control unit 118. [Figure 7] A diagram showing an example of the display screen DI1. [Figure 8] A diagram showing an example of the display screen DI2. [Figure 9] A diagram showing an example of the display screen DI3. [Figure 10] A diagram showing an example of the display screen DI3. [Figure 11] A diagram showing an example of the display screen DI4. [Figure 12] A diagram showing an example of the display screen DI5. [Figure 13] A diagram showing an example of the display screen DI5. [Figure 14] A diagram showing an example of the display screen DI6. [Figure 15]A diagram showing an example of the display screen DI6. [Figure 16] A diagram showing an example of the display screen DI6.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, a display method, an information processing apparatus, and a program according to an embodiment will be described with reference to the drawings. In each figure, the dimensions and scales of each part are appropriately different from the actual ones. Further, the embodiments described below are preferred specific examples, and thus various technically preferable limitations are imposed. However, the scope of the present disclosure is not limited to these forms unless there is a description specifically limiting the present disclosure in the following description.

[0011] 1: First Embodiment 1-1: Overall Configuration FIG. 1 is a block diagram showing the overall configuration of a projection system 1 according to the first embodiment. The projection system 1 includes one information processing apparatus 10 and one or more projection apparatuses 30. For convenience of explanation, the projection system 1 shown in FIG. 1 includes two projection apparatuses 30[1] and 30[2]. However, the number of projection apparatuses 30 is not limited to two, and may be one or three or more.

[0012] The projection system 1 is a system for executing multi-projection. Specifically, the projection apparatus 30[1] projects a projection image PI1 onto a projection surface PF. Similarly, the projection apparatus 30[2] projects a projection image PI2 onto the projection surface PF. On the projection surface PF, a partial image KI1 included in the projection image PI1 and a partial image KI2 included in the projection image PI2 are superimposed, so that one overall image AI is displayed on the projection surface PF.

[0013] The information processing apparatus 10 controls the projection apparatus 30[1] and the projection apparatus 30[2] in order to execute multi-projection. The information processing device 10 may be a PC (Personal Computer), a tablet terminal, or a smartphone.

[0014] In FIG. 1, the information processing device 10, the projection device 30[1], and the projection device 30[2] are connected to be communicable with each other via a communication network NET.

[0015] Also, the projection device 30[1] projects an anchor image AC1 onto the projection surface PF. The anchor image AC1 is an image indicating the projection area of the projection image PI1 from the projection device 30[1] on the projection surface PF. The anchor image AC1 is a frame line or a part of the frame line of the projection image PI1 on the projection surface PF. In the example shown in FIG. 1, the anchor image AC1 is an image including the four vertices of the projection image PI1. In this embodiment, the shape of the anchor image AC1 is L-shaped.

[0016] When constructing the projection system 1, the user of the projection system 1 operates the information processing device 10 or the projection device 30[1] to adjust the position of the anchor image AC1 on the projection surface PF. Further, the user sets the position and size of the area occupied by the projection image PI1 on the projection surface PF by adjusting the position of the anchor image AC1 on the projection surface PF.

[0017] Also, the projection device 30[2] projects an anchor image AC2 onto the projection surface PF. The anchor image AC2 is an image indicating the projection area from the projection device 30[2] on the projection surface PF. The anchor image AC2 is a frame line or a part of the frame line of the projection image PI2 on the projection surface PF. In the example shown in FIG. 1, the anchor image AC2 is an image including two vertices on the side opposite to the projection image PI1 among the four vertices of the projection image PI2.

[0018] When constructing projection system 1, the user of projection system 1 adjusts the position of anchor image AC2 on the projection surface PF by operating the information processing device 10 or projection device 30[2]. Furthermore, by adjusting the position of anchor image AC2 on the projection surface PF, the user sets the position and size of the area occupied by projection image PI2 on the projection surface PF.

[0019] When the projection system 1 is restored after a predetermined amount of time has elapsed since its construction, anchor images AC1 and AC2 can be automatically displayed on the projection surface PF without any user intervention. "Restoration of the projection system 1" refers, for example, to the case where the power to the projection system 1 is turned off after its construction and then turned back on. During the restoration of the projection system 1, the positions of the projection devices 30[1] and 30[2] may have changed due to the passage of time since the construction of the projection system 1.

[0020] The above-mentioned "construction period" is included in the "first period" as an example. On the other hand, the above-mentioned "restoration period" is included in the "second period" which follows the "first period" as an example. The "first period" also includes the "maintenance period," in addition to the above-mentioned "construction period," in which the position and orientation of the projection devices 30[1] and 30[2] constructed during the "construction period" are maintained.

[0021] As described later, when constructing the projection system 1, the information processing device 10 stores the positions of feature portions on the projection surface PF and the positions of anchor image AC1 and anchor image AC2, and when restoring the projection system 1, it calculates the positions for displaying anchor image AC1 and anchor image AC2 again based on the positions of feature portions on the projection surface PF.

[0022] Anchor image AC1 and anchor image AC2 may also be displayed on the display device 130, which is provided in the information processing device 10 and will be described later.

[0023] 1-2: Projection device configuration Figure 2 is a block diagram showing an example configuration of a projection device 30[1]. The projection device 30[1] comprises a projector 310, an imaging device 320, a processing device 330, a storage device 340, and a communication device 350. Each element of the projection device 30[1] is interconnected by one or more buses for communicating information. Furthermore, each element of the projection device 30[1] consists of one or more devices, and some elements of the projection device 30[1] may be omitted. Note that the projection device 30[2] is configured in the same way as the projection device 30[1], so its illustration and description are omitted.

[0024] The projector 310 is a device that projects images acquired by the acquisition unit 332 (described later) onto the projection surface PF. The projector 310 projects various images under the control of the processing unit 330. The projector 310 includes, for example, a light source, a liquid crystal panel, and a projection lens. It modulates light from the light source using the liquid crystal panel and projects the modulated light onto a screen, wall, etc., via the projection lens. The liquid crystal panel is an example of a light modulation element that generates an image according to the input image data. The light modulation element may be a digital mirror device (DMD) instead of a liquid crystal panel.

[0025] The imaging device 320 is a device that images the outside world of the projection device 30[1]. In this embodiment, the imaging device 320 images the projection surface PF. As a result of imaging the projection surface PF, the imaging device 320 generates an image TI of the projection surface PF. The imaging device 320 is, for example, a camera built into the projection device 30[1]. However, as another example, the imaging device 320 may be an external camera such as a webcam.

[0026] The processing unit 330 is a processor that controls the entire projection device 30[1], and is composed of, for example, one or more chips. The processing unit 330 is composed of, for example, a central processing unit (CPU) that includes interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 330 may be implemented by hardware such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The processing unit 330 executes various processes in parallel or sequentially.

[0027] The storage device 340 is a recording medium readable by the processing device 330 and stores multiple programs, including the control program PR3 executed by the processing device 330. The storage device 340 may also store the projected image PI1 projected by the projection device 30[1].

[0028] The storage device 340 may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory). The storage device 340 may also be called a register, cache, main memory, or primary memory.

[0029] The communication device 350 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 350 is also called, for example, a network device, network controller, network card, or communication module. The communication device 350 may be equipped with a connector for wired connection and an interface circuit corresponding to the connector. The communication device 350 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include those compliant with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include those compliant with wireless LAN and Bluetooth®.

[0030] The processing unit 330 functions as a communication control unit 331, an acquisition unit 332, a projection control unit 333, and an imaging control unit 334 by reading and executing the control program PR3 from the storage device 340. The control program PR3 may also be transmitted via the communication network NET from another device, such as a server that manages the information processing unit 10 or the projection device 30[1].

[0031] The communication control unit 331 causes the communication device 350 to send and receive various data, information, and signals with other devices. These "other devices" include the information processing device 10 and the projection device 30[2].

[0032] As an example, the communication control unit 331 causes the communication device 350 to receive various control signals from the information processing device 10. As another example, the communication control unit 331 causes the communication device 350 to receive image data showing the projected image PI1 from the information processing device 10. As yet another example, the communication control unit 331 causes the information processing device 10 to transmit image data showing the captured image TI captured by the imaging device 320 to the communication device 350.

[0033] The acquisition unit 332 acquires the projected image PI1. For example, the acquisition unit 332 may acquire the projected image PI1 stored in the storage device 340. Alternatively, as another example, the acquisition unit 332 may acquire the projected image PI1 from the information processing device 10.

[0034] The projection control unit 333 projects the projection image PI1 acquired by the acquisition unit 332 onto the projection surface PF using the projector 310.

[0035] The imaging control unit 334 causes the imaging device 320 to image the projection surface PF. The image TI of the projection surface PF captured by the imaging device 320 is acquired by the acquisition unit 332 and transmitted to the information processing device 10 by the communication control unit 331.

[0036] 1-3: Configuration of Information Processing Equipment Figure 3 is a block diagram showing an example configuration of the information processing device 10. The information processing device 10 comprises a processing device 110, a storage device 120, a display device 130, an input device 140, and a communication device 150. Each element of the information processing device 10 is interconnected by one or more buses for communicating information.

[0037] The processing unit 110 is a processor that controls the entire information processing unit 10, and is composed of, for example, one or more chips. The processing unit 110 is composed of, for example, a central processing unit (CPU) that includes interfaces with peripheral devices, an arithmetic unit, and registers. Some or all of the functions of the processing unit 110 may be implemented by hardware such as a DSP, ASIC, PLD, and FPGA. The processing unit 110 executes various processes in parallel or sequentially.

[0038] The storage device 120 is a recording medium readable by the processing unit 110, and stores multiple programs, including the control program PR1 executed by the processing unit 110. The storage device 120 also stores various parameters calculated by the calculation unit 117, which will be described later. The storage device 120 may consist of at least one of the following: ROM, EPROM, EEPROM, and RAM. The storage device 120 may also be called a register, cache, main memory, or primary memory.

[0039] The display device 130 is a device that displays images and text information. The display device 130 displays various images under the control of the processing device 110. For example, various display panels such as liquid crystal display panels and organic EL (Electro-Luminescence) display panels are suitably used as the display device 130.

[0040] The input device 140 is a device that accepts input from the user. For example, the input device 140 includes a keyboard, touchpad, touch panel, or pointing device such as a mouse. If the input device 140 includes a touch panel, it may also function as the display device 130.

[0041] The communication device 150 is hardware acting as a transmitting and receiving device for communicating with other devices. The communication device 150 is also called, for example, a network device, network controller, network card, or communication module. The communication device 150 may be equipped with a connector for wired connection and an interface circuit corresponding to the connector. The communication device 150 may also be equipped with a wireless communication interface. Examples of connectors and interface circuits for wired connection include those compliant with wired LAN, IEEE 1394, and USB. Examples of wireless communication interfaces include those compliant with wireless LAN and Bluetooth®.

[0042] The processing unit 110 functions as a communication control unit 111, a reception unit 112, a setting unit 113, an acquisition unit 114, an extraction unit 115, a measurement unit 116, a calculation unit 117, and a display control unit 118 by reading and executing the control program PR1 from the storage device 120. The control program PR1 may also be transmitted via the communication network NET from another device, such as a server that manages the information processing unit 10.

[0043] The communication control unit 111 causes the communication device 150 to send and receive various data, information, and signals with other devices. These "other devices" include the projection device 30[1] and the projection device 30[2].

[0044] As an example, the communication control unit 111 causes the projection devices 30[1] and 30[2] to transmit various control signals to the communication device 150. As another example, the communication control unit 111 causes the projection device 30[1] to transmit image information indicating the projected image PI1 to the communication device 150. As yet another example, the communication control unit 111 causes the projection device 30[2] to transmit image information indicating the projected image PI2 to the communication device 150. As yet another example, the communication control unit 111 causes the communication device 150 to receive image information indicating the captured image TI captured by each of the projection devices 30[1] and 30[2].

[0045] For the sake of simplicity, the following explanation of the functions of the reception unit 112 to the display control unit 118 will mainly describe the processing using the captured image TI captured by the projection device 30[1], and will omit the explanation of the processing using the captured image TI captured by the projection device 30[2].

[0046] The reception unit 112 receives user operations to adjust the position of the anchor image AC1 on the projection surface PF. Specifically, the user adjusts the position of the anchor image AC1 on the projection surface PF by operating the input device 140. The reception unit 112 receives these operations by the user using the input device 140.

[0047] The setting unit 113 sets the position of the anchor image AC1 on the projection surface PF based on the user operation received by the reception unit 112. As a result, the coordinate values ​​of the anchor image AC1 are measured in the panel coordinate system, which is the coordinate system of the liquid crystal panel provided in the projection device 30[1]. The coordinate values ​​of the anchor image AC1 are, for example, the vertex coordinates of an L-shape.

[0048] The acquisition unit 114 acquires an image TI from the projection device 30[1]. The image TI is, for example, an image that includes the entire projection surface PF captured by the projection device 30[1]. More specifically, the acquisition unit 114 acquires an image TI from the projection device 30[1] that captures the entire projection surface PF, both during the construction and restoration of the projection system 1. The image TI1 acquired by the acquisition unit 114 during the construction of the projection system 1 is an example of a "first image". The image TI1 is an image captured during the "first period" described above. The image TI1 includes the anchor image AC1. The image TI2 acquired by the acquisition unit 114 during the restoration of the projection system 1 is an example of a "second image". The image TI2 is an image captured during the "second period" described above.

[0049] When constructing the projection system 1, the extraction unit 115 extracts feature portions of the projection surface PF as markers based on the captured image TI1 acquired by the acquisition unit 114. Here, "markers of the projection surface PF" means "markers for displaying the anchor image AC1 on the projection surface PF." In other words, the markers of the projection surface PF are parts of the projection surface PF, such as corners and edges, that serve as a reference for adjusting the position and shape of the overall image AI. Feature portions are these reference parts of the projection surface PF. The user and the processing unit 110 recognize the parts that characterize the projection surface PF, such as corners and edges, as feature portions, i.e., markers. Specifically, as an example, the extraction unit 115 extracts multiple edge pixels from the captured image TI1 using an edge detection algorithm and detects an edge ED formed by connecting multiple edge pixels. The extraction unit 115 detects multiple edge EDs. The edge detection algorithm is, for example, Canny edge detection. Furthermore, the extraction unit 115 extracts the intersection points of edges ED as feature portions. In other words, the feature portions in this embodiment are the corners of the projection surface PF. The feature portions may also be edges of the projection surface PF instead of corners. If the feature portion is an edge, the extraction unit 115 may only perform the process of detecting the edges ED without extracting the intersection points. If there are multiple such feature portions, the extraction unit 115 may, as an example, extract the feature portion that is closest to the anchor image AC1 in the captured image TI1 as a marker for the projection surface PF. The extraction unit 115 may also use AKAZE (Accelerated-KAZE), which is an example of a feature point extraction algorithm, to extract the feature portions. When the extraction unit 115 uses a feature point extraction algorithm, it may directly extract the corners of the projection surface PF as feature portions without detecting the edges ED.

[0050] Similarly, when the projection system 1 is restored, the extraction unit 115 extracts feature portions of the projection surface PF as markers based on the captured image TI2 acquired by the acquisition unit 114.

[0051] These markers are located, for example, near the corners of the projection surface PF.

[0052] The measurement unit 116 measures the coordinate values ​​of the anchor image AC1 in the panel coordinate system, which is the coordinate system of the liquid crystal panel provided in the projection device 30[1].

[0053] Furthermore, the measurement unit 116 measures the correspondence between the panel coordinate system of the liquid crystal panel provided in the projection device 30[1] and the camera coordinate system, which is the coordinate system of the imaging device 320. The panel coordinate system is a two-dimensional coordinate system for indicating the coordinate values ​​of each pixel of the liquid crystal panel. The camera coordinate system is a two-dimensional coordinate system for indicating the coordinate values ​​of each pixel of the captured image TI.

[0054] As an example, the measurement unit 116 projects a checkerboard-shaped pattern image onto the projection surface PF using the projection device 30[1]. The measurement unit 116 also causes the projection device 30[1] to capture the pattern image displayed on the projection surface PF. Then, the measurement unit 116 measures the correspondence between the coordinate values ​​in the panel coordinate system of the first feature point included in the pattern image on the liquid crystal panel and the coordinate values ​​in the camera coordinate system of the second feature point corresponding to the first feature point included in the pattern image on the captured image TI.

[0055] Furthermore, the measurement unit 116 measures the coordinate values ​​in the camera coordinate system of the feature portion of the projection surface PF that is extracted by the extraction unit 115 during the construction of the projection system 1.

[0056] The calculation unit 117 calculates various parameters, which will be described later.

[0057] Figure 4 is an explanatory diagram showing the operation of the processing unit 110 and the storage device 120 during the construction of the projection system 1. In Figure 4, the processing unit 110 mainly functions as a setting unit 113, an extraction unit 115, a measurement unit 116, and a calculation unit 117. In the explanatory diagram of Figure 4, the black boxes represent the processing itself in each step described later. The white boxes represent the output results from the processing of each step indicated by the black boxes.

[0058] In step S1, the processing unit 110 functions as a setting unit 113. The processing unit 110 sets the position of the anchor image AC1 on the projection surface PF. The processing unit 110 also functions as a measurement unit 116. The processing unit 110 measures the coordinate value OP1 of the anchor image AC1 in the panel coordinate system.

[0059] In step S2, the processing unit 110 functions as a measurement unit 116. The processing unit 110 measures the correspondence between the coordinate values ​​of the first pattern feature points in the panel coordinate system and the coordinate values ​​of the second pattern feature points in the camera coordinate system. The "coordinate values ​​of the first pattern feature points in the panel coordinate system" are feature points included in the pattern image on the liquid crystal panel. The "coordinate values ​​of the second pattern feature points in the camera coordinate system" are feature points included in the pattern image on the captured image TI1 that correspond to the first pattern feature points. The processing unit 110 also functions as a calculation unit 117. Based on this correspondence, the processing unit 110 calculates a transformation matrix OP2 from the panel coordinate system to the camera coordinate system.

[0060] In step S3, the processing unit 110 functions as a calculation unit 117. The processing unit 110 calculates the coordinate values ​​OP3 of the anchor image AC1 in the camera coordinate system by multiplying the coordinate values ​​of the anchor image AC1 in the panel coordinate system calculated in step S1 by the transformation matrix OP2 calculated in step S2.

[0061] In step S4, the processing unit 110 functions as an extraction unit 115. Based on the captured image TI1, the processing unit 110 extracts feature portions of the projection surface PF as markers. The processing unit 110 also functions as a measurement unit 116. The processing unit 110 measures the coordinate values ​​OP4 of the feature portions of the projection surface PF as markers in the camera coordinate system. In step S4, it is preferable that a pattern image that is entirely white, a pattern image that is entirely black, or a pattern image with a gradient is projected from the projection device 30[1] onto the projection surface PF.

[0062] In step S5, the storage device 120 stores the coordinate values ​​OP1 of the anchor image AC1 in the panel coordinate system, which were measured in step S1; the coordinate values ​​OP3 of the anchor image AC1 in the camera coordinate system, which were calculated in step S3; and the coordinate values ​​OP4 of the feature portion of the projection surface PF used as a marker in the camera coordinate system, which were measured in step S4. As a result, the coordinate values ​​OP1 of the anchor image AC1 in the panel coordinate system, the coordinate values ​​OP3 of the anchor image AC1 in the camera coordinate system, and the coordinate values ​​OP4 of the feature portion of the projection surface PF used as a marker in the camera coordinate system are stored in the storage device 120.

[0063] Figure 5 is an explanatory diagram showing the operation of the processing unit 110 and the storage device 120 during the restoration of the projection system 1. In Figure 5, the processing unit 110 mainly functions as an extraction unit 115, a measurement unit 116, and a calculation unit 117. In the explanatory diagram of Figure 5, as in the explanatory diagram of Figure 4, the black boxes represent the processing itself in each step described later. The white boxes represent the output results of the processing in each step indicated by the black boxes.

[0064] In step S11, the processing unit 110 functions as an extraction unit 115. Based on the captured image TI2, the processing unit 110 extracts feature portions of the projection surface PF that serve as markers. More specifically, based on the coordinate values ​​OP4 of the feature portions of the projection surface PF measured in the camera coordinate system during the construction of the projection system 1, the processing unit 110 extracts feature portions of the projection surface PF that serve as markers in the vicinity of those coordinate values ​​in the captured image TI2. The processing unit 110 also functions as a measurement unit 116. The processing unit 110 measures the coordinate values ​​OP11 of the feature portions of the projection surface PF that serve as markers in the camera coordinate system. In step S11, it is preferable that a pattern image that is entirely white, a pattern image that is entirely black, or a pattern image with a gradient is projected from the projection device 30[1] onto the projection surface PF.

[0065] Furthermore, in step S11, the method of calculation by the processing unit 110 to extract feature portions as markers of the projection surface PF based on the captured image TI2 is preferably the same as the method of calculation by the processing unit 110 to extract feature portions as markers of the projection surface PF based on the captured image TI1 in step S4. The calculation method is, for example, the type of algorithm. In this embodiment, the calculation method in steps S4 and S11 is, for example, Canny edge detection. Even if the calculation method in step S11 is the same as the calculation method in step S4, the threshold used in, for example, hysteresis thresholding, i.e., the value of the algorithm-specific parameter, may be different in step S4 and step S11.

[0066] In step S12, the processing unit 110 functions as a measurement unit 116. The processing unit 110 measures the correspondence between the coordinate values ​​of the first pattern feature points in the panel coordinate system and the coordinate values ​​of the second pattern feature points in the camera coordinate system. The "coordinate values ​​of the first pattern feature points in the panel coordinate system" are feature points included in the pattern image on the liquid crystal panel. The "coordinate values ​​of the second pattern feature points in the camera coordinate system" are feature points included in the pattern image on the captured image TI2 that correspond to the first pattern feature points. The processing unit 110 also functions as a calculation unit 117. Based on this correspondence, the processing unit 110 calculates a transformation matrix OP12 from the camera coordinate system to the panel coordinate system.

[0067] In step S13, the processing unit 110 functions as a calculation unit 117. The processing unit 110 calculates the difference between the coordinate values ​​OP4 of the feature portion as a marker in the camera coordinate system, which were saved when the projection system 1 was constructed, and the coordinate values ​​OP11 of the feature portion as a marker in the camera coordinate system, which were measured in step S11. This difference is calculated, for example, as a projection transformation matrix. The processing unit 110 then applies this difference to the coordinate values ​​OP3 of the anchor image AC1 in the camera coordinate system, which were saved when the projection system 1 was constructed, to calculate the coordinate values ​​of the anchor image AC1 in the camera coordinate system when the projection system 1 is restored. Furthermore, the processing unit 110 calculates the coordinate values ​​OP13 of the anchor image AC1 in the panel coordinate system by multiplying the coordinate values ​​of the anchor image AC1 in the camera coordinate system by the transformation matrix calculated in step S12.

[0068] In step S14, the storage device 120 stores the coordinate values ​​OP13 of the anchor image AC1 in the panel coordinate system, which were measured in step S13. As a result, the coordinate values ​​OP13 of the anchor image AC1 in the panel coordinate system are stored in the storage device 120.

[0069] The processing unit 110 can use various parameters stored in the storage device 120 as appropriate in processing after step S14.

[0070] In Figure 3, the display control unit 118 causes various display screens DI to be displayed on the display device 130. The display control unit 118 also causes the display device 130 to display an image on the display device 130 that shows the position of characteristic parts of the projection surface PF that serve as markers on the projection surface PF.

[0071] Figure 6 is a functional block diagram of the display control unit 118. The display control unit 118 comprises a first display control unit 118[1] and a second display control unit 118[2].

[0072] The first display control unit 118[1] causes the display device 130 to display an image on the display device 130 that shows the position of feature portions of the projection surface PF, which are extracted based on the captured image TI1 and serve as markers for the projection surface PF, within the captured image TI1. This control is an example of "first display control". The image is also an example of "first image". More specifically, the first display control unit 118[1] displays the "first image" on the display device 130 based on the coordinate values ​​in the camera coordinate system of the feature portion of the projection surface PF used as a marker during the construction of the projection system 1, which were saved in step S5 of Figure 4.

[0073] The second display control unit 118[2] causes the display device 130 to display an image on the display device 130 that shows the position of feature portions of the projection surface PF, which are extracted based on the captured image TI2 and serve as markers for the projection surface PF, within the captured image TI2. This control is an example of "second display control". The image in question is an example of "second image". More specifically, the second display control unit 118[2] displays the "second image" on the display device 130 based on the coordinate values ​​in the camera coordinate system of the feature portion of the projection surface PF that serves as a marker when the projection system 1 is restored, which were measured in step S11 of Figure 5.

[0074] Figure 7 shows an example of a display screen DI1 that the display control unit 118 displays on the display device 130. The display screen DI1 has regions RM1 to RM5.

[0075] Area RM1 displays two slider bars SB. Slide bars SB are used to set parameter values ​​for extracting feature areas as markers on the projection surface PF. For example, slider bars SB are used to set the threshold used to extract feature areas from the captured image TI, and the maximum number of feature areas to extract.

[0076] Region RM2 displays the entire captured image TI2. The captured image TI2 displayed in region RM2 includes the overall image AP of the projection surface PF. The overall image AP of the projection surface PF includes the overall image AI of the projection image PI, which is generated by superimposing the projection images PI1 and PI2. In addition, multiple candidate markers CM of the projection surface PF are displayed near the corners of the overall image AP.

[0077] Region RM3 displays a portion of the captured image TI2. In the example shown in Figure 7, region RM3 displays the vicinity of the upper left corner of the projection surface PF from the captured image TI2. The captured image TI2 displayed in region RM3 includes a partial image PP of the projection surface PF. In addition, in region RM3, the edges ED1 and ED2 of the projection surface PF, and image MK1 located at the intersection of edges ED1 and ED2, are superimposed on the captured image TI2. Image MK1 is an image that shows the position of the currently selected landmark among the multiple landmark candidate CMs displayed in region RM2. Image MK1 is an example of a "second image".

[0078] Region RM4 displays image FG, which shows the coordinate values ​​corresponding to the position of feature parts that serve as markers on the projection surface PF in the camera coordinate system. In the example shown in Figure 7, image FG shows the coordinate values ​​corresponding to the markers located near the four corners of the projection surface PF that are currently selected. Image FG is an example of the "second image".

[0079] In the example shown in Figure 7, the entire captured image TI2 is displayed in region RM2. However, the entire captured image TI1 may also be displayed in region RM2. In this case, a portion of the captured image TI1 will be displayed in region RM3. Also, in this case, images MK1 and FG are examples of the "first image".

[0080] Figure 8 shows an example of a display screen DI2 that the display control unit 118 causes to be displayed on the display device 130. The display screen DI2 has regions RM5 to RM9.

[0081] Region RM5 displays a portion of the captured image TI1. In the example shown in Figure 8, region RM5 displays the vicinity of the upper left corner of the projection surface PF from the captured image TI1. The captured image TI1 displayed in region RM5 includes a partial image PP of the projection surface PF. In addition, in region RM5, the edges ED3 and ED4 of the projection surface PF, and image MK2 located at the intersection of edges ED3 and ED4, are superimposed on the captured image TI1. Image MK2 is an image that shows the location of a feature portion of the projection surface PF that was detected as a marker during the construction of the projection system 1. Image MK2 is an example of the "first image".

[0082] Region RM6 displays a portion of the captured image TI2. In the example shown in Figure 8, region RM6 displays the vicinity of the upper left corner of the projection surface PF from the captured image TI2. The captured image TI2 displayed in region RM6 includes a partial image PP of the projection surface PF. In addition, in region RM6, the edges ED5 and ED6 of the projection surface PF, and image MK3 located at the intersection of edges ED5 and ED6, are superimposed on the captured image TI2. Image MK3 is an image that shows the position of a feature portion of the projection surface PF that was detected as a marker during the restoration of the projection system 1. Image MK3 is also an image that shows the position of the marker currently selected from among the multiple marker candidate CMs displayed in region RM2 of the display screen DI1. Image MK3 is an example of a "second image".

[0083] Region RM7 displays a portion of the captured image TI1. In the example shown in Figure 8, region RM7 displays the vicinity of the lower left corner of the projection surface PF from the captured image TI1. The captured image TI1 displayed in region RM7 includes a partial image PP of the projection surface PF. In addition, in region RM7, the edges ED7 and ED8 of the projection surface PF, and image MK4 located at the intersection of edges ED7 and ED8, are superimposed on the captured image TI1. Image MK4 is an image that shows the location of a feature portion of the projection surface PF that was detected as a marker during the construction of the projection system 1. Image MK4 is an example of the "first image".

[0084] Region RM8 displays a portion of the captured image TI2. In the example shown in Figure 8, region RM8 displays the vicinity of the lower left corner of the projection surface PF from the captured image TI2. The captured image TI2 displayed in region RM8 includes a partial image PP of the projection surface PF. In addition, in region RM8, the edges ED9 and ED10 of the projection surface PF, and image MK5 located at the intersection of edges ED9 and ED10, are superimposed on the captured image TI2. Image MK5 is an image that shows the position of a feature portion of the projection surface PF that was detected as a marker during the restoration of the projection system 1. Image MK5 is also an image that shows the position of the marker currently selected from among the multiple marker candidate CMs displayed in region RM2 of the display screen DI1. Image MK5 is an example of a "second image".

[0085] For example, nothing is displayed in area RM9.

[0086] In the display screen DI2 illustrated in Figure 8, images MK2 to MK5 are displayed side by side.

[0087] The user of projection system 1 visually compares the position of image MK2 on the projection surface PF displayed in area RM5 with the position of image MK3 on the projection surface PF displayed in area RM6. If the two positions differ significantly, the user operates the input device 140 to display the display screen DI1, as exemplified in Figure 7, on the display device 130. Then, the user operates the slider bar SB displayed in area RM1 and selects a candidate CM for a projection surface PF marker from several candidate CMs displayed in area RM2 that is different from the currently selected marker. As a result, the position of image MK3 in area RM6 of display screen DI2 is updated to the position corresponding to the newly selected marker.

[0088] Similarly, the user of projection system 1 visually compares the position of image MK4 on the projection surface PF displayed in area RM7 with the position of image MK5 on the projection surface PF displayed in area RM8. If the two positions differ significantly, the user operates the input device 140 to display the display screen DI1, as illustrated in Figure 7, on the display device 130. Then, the user operates the slider bar SB displayed in area RM1 and selects a candidate CM for a projection surface PF marker from among several candidate CMs displayed in area RM2 that is different from the currently selected marker. As a result, in area RM8 of display screen DI2, the position of image MK5 is updated to the position corresponding to the newly selected marker.

[0089] If the position and orientation of the projection device 30[1] differ significantly between the construction of projection system 1 and the restoration of projection system 1, the position of image MK2 on the projection surface PF displayed in area RM5 will differ significantly from the position of image MK3 on the projection surface PF displayed in area RM6. Furthermore, the position of image MK4 on the projection surface PF displayed in area RM7 will differ significantly from the position of image MK5 on the projection surface PF displayed in area RM8.

[0090] The fact that the position of the marker on the projection surface PF corresponding to image MK2 in the camera coordinate system is the same as the position of the marker on the projection surface PF corresponding to image MK3 in the camera coordinate system, and that the position of the marker on the projection surface PF corresponding to image MK4 in the camera coordinate system is the same as the position of the marker on the projection surface PF corresponding to image MK5 in the camera coordinate system, is directly related to the accuracy of maintaining the position and orientation of the projection device 30[1]. For this reason, in this embodiment, in order to minimize such errors, the display control unit 118 displays the target extraction result when the projection system 1 is restored, along with the target extraction result when the projection system 1 is constructed, on the display device 130 at the same time.

[0091] Figure 9 shows an example of a display screen DI3 that the display control unit 118 displays on the display device 130. The display screen DI3 has regions RM10 to RM15.

[0092] Region RM10 displays a portion of the captured image TI1. In the example shown in Figure 9, region RM10 displays the vicinity of the upper right corner of the projection surface PF from the captured image TI1. The captured image TI1 displayed in region RM10 includes a partial image PP of the projection surface PF. In addition, in region RM10, the edges ED11 and ED12 of the projection surface PF, and image MK6 located at the intersection of edges ED11 and ED12, are superimposed on the captured image TI1. Image MK6 is an image that shows the location of a feature portion of the projection surface PF that was detected as a marker during the construction of the projection system 1. Image MK6 is an example of the "first image".

[0093] Region RM11 displays a portion of the captured image TI2. In the example shown in Figure 9, region RM11 displays the vicinity of the upper right corner of the projection surface PF from the captured image TI2. The captured image TI2 displayed in region RM11 includes a partial image PP of the projection surface PF. In addition, in region RM11, the edges ED13 and ED14 of the projection surface PF, and image MK7 located at the intersection of edges ED13 and ED14, are superimposed on the captured image TI2. Image MK7 is an image that shows the position of a feature portion of the projection surface PF that was detected as a marker during the restoration of the projection system 1. Image MK7 is also an image that shows the position of the marker currently selected from among the multiple marker candidate CMs displayed in region RM2 of the display screen DI1. Image MK7 is an example of a "second image".

[0094] In the display screen DI2 illustrated in Figure 8, captured images TI1 and TI2 captured by the projection device 30[1] were used, but in the display screen DI3, as an example, captured images TI1 and TI2 captured by the projection device 30[2] are used.

[0095] Region RM12 displays a portion of the captured image TI1. In the example shown in Figure 9, region RM12 displays the vicinity of the lower right corner of the projection surface PF from the captured image TI1. The captured image TI1 displayed in region RM12 includes a partial image PP of the projection surface PF. In addition, in region RM12, the edges ED15 and ED16 of the projection surface PF, and image MK8 located at the intersection of edges ED15 and ED16, are superimposed on the captured image TI1. Image MK8 is an image that shows the location of a feature portion of the projection surface PF that was detected as a marker during the construction of the projection system 1. Image MK8 is an example of the "first image".

[0096] Region RM13 displays a portion of the captured image TI2. In the example shown in Figure 9, region RM13 displays the vicinity of the lower right corner of the projection surface PF from the captured image TI2. The captured image TI2 displayed in region RM13 includes a partial image PP of the projection surface PF. In addition, in region RM13, the edges ED17 and ED18 of the projection surface PF, and image MK9 located at the intersection of edges ED17 and ED18, are superimposed on the captured image TI2. Image MK9 is an image that shows the position of a feature portion of the projection surface PF that was detected as a marker during the restoration of the projection system 1. Image MK9 is also an image that shows the position of the marker currently selected from among the multiple marker candidate CMs displayed in region RM2 of the display screen DI1. Image MK9 is an example of a "second image".

[0097] Note that, as an example, nothing will be displayed in areas RM14 and RM15.

[0098] In the display screen DI3 illustrated in Figure 9, images MK6 to MK9 are displayed side by side.

[0099] The user of projection system 1 visually compares the position of image MK6 on the projection surface PF displayed in area RM10 with the position of image MK7 on the projection surface PF displayed in area RM11. If the two positions differ significantly, the user operates the input device 140 to display the display screen DI1, as exemplified in Figure 7, on the display device 130. Then, the user operates the slider bar SB displayed in area RM1 and selects a candidate CM for a projection surface PF marker from several candidate CMs displayed in area RM2 that is different from the currently selected marker. As a result, the position of image MK7 in area RM11 of display screen DI3 is updated to the position corresponding to the newly selected marker.

[0100] Similarly, the user of projection system 1 visually compares the position of image MK8 on the projection surface PF displayed in area RM12 with the position of image MK9 on the projection surface PF displayed in area RM13. If the two positions differ significantly, the user operates the input device 140 to display the display screen DI1, as illustrated in Figure 7, on the display device 130. Then, the user operates the slider bar SB displayed in area RM1 and selects a candidate CM for a projection surface PF marker from among several candidate CMs displayed in area RM2 that is different from the currently selected marker. As a result, the position of image MK9 in area RM13 of display screen DI3 is updated to the position corresponding to the newly selected marker.

[0101] If the position and orientation of the projection device 30[2] differ significantly between the construction of projection system 1 and the restoration of projection system 1, the position of image MK6 on the projection surface PF displayed in area RM10 will differ significantly from the position of image MK7 on the projection surface PF displayed in area RM11. Similarly, the position of image MK8 on the projection surface PF displayed in area RM12 will differ significantly from the position of image MK9 on the projection surface PF displayed in area RM13.

[0102] The fact that the position of the marker on the projection surface PF corresponding to image MK6 in the camera coordinate system is the same as the position of the marker on the projection surface PF corresponding to image MK7 in the camera coordinate system, and that the position of the marker on the projection surface PF corresponding to image MK8 in the camera coordinate system is the same as the position of the marker on the projection surface PF corresponding to image MK9 in the camera coordinate system, is directly related to the accuracy of maintaining the position and orientation of the projection device 30[2]. For this reason, in this embodiment, in order to minimize such errors, the display control unit 118 displays the marker extraction results when the projection system 1 is restored, along with the marker extraction results when the projection system 1 is constructed, on the display device 130 at the same time.

[0103] In this embodiment, the projection system 1 includes two projection devices 30, a projection device 30[1] and a projection device 30[2]. Therefore, the display control unit 118 individually displays the display screen DI2 corresponding to projection device 30[1] and the display screen DI3 corresponding to projection device 30[2] on the display device 130. When comparing the display screen DI2 illustrated in Figure 8 with the display screen DI3 illustrated in Figure 9, the positions of areas RM5 and RM6 in display screen DI2 correspond to the position of area RM14 in display screen DI3. Also, the positions of areas RM7 and RM8 in display screen DI2 correspond to the position of area RM15 in display screen DI3. Furthermore, the positions of areas RM10 to RM13 in display screen DI3 correspond to the position of area RM9 in display screen DI2.

[0104] In the process of restoring the position and orientation of the projection devices 30[1] and 30[2], the information processing device 10 according to this embodiment simultaneously displays on the display device 130 the results of extracting markers on the projection surface PF during restoration of the projection system 1, along with the results of extracting markers on the projection surface PF during construction of the projection system 1. By comparing the extraction results of both, the user can minimize errors in the location where the markers were extracted. The information processing device 10 also stores the extraction position and extraction method of the markers during construction of the projection system 1. Therefore, when restoring the projection system 1, the information processing device 10 can automatically restore the position of the markers on the projection surface PF without requiring any operation from the user. If the extraction position of the target differs significantly between the construction and restoration of the projection system 1, the user can manually re-select the target position. In other words, the user can verify the validity of the target position extraction function.

[0105] 2: Variant The above configurations can be modified in various ways. Specific examples of modifications are given below. The configurations exemplified below and the configurations shown in the above embodiments can be merged as appropriate, within the bounds of mutual consistency. In the modified configurations exemplified below, for elements whose operation and function are equivalent to those in the embodiments, the reference numerals used in the above description will be reused, and detailed explanations of each will be omitted as appropriate.

[0106] 2-1: Variation 1 In the above embodiment, the display screen DI2 may display the acquisition date and time of the data used to display images MK2 to MK5. Similarly, the display screen DI3 may display the acquisition date and time of the data used to display images MK6 to MK9.

[0107] Figure 10 shows an example of the display screen DI3 in this modified example. The upper part of area RM14 of the display screen DI3 displays image DT1, which shows the acquisition date and time of the data used to display images MK6 and MK8. ​​The lower part of area RM14 of the display screen DI3 displays image DT2, which shows the acquisition date and time of the data used to display images MK7 and MK9.

[0108] Furthermore, if projection system 1 is restored, and images MK7 and MK9 are displayed on display screen DI3, and then projection system 1 is restored again, for example, multiple images including image MK7 displayed in area RM11, multiple images including image MK9 displayed in area RM13, and image DT2 displayed in the lower section of area RM14 will be updated. On the other hand, for example, multiple images including image MK6 displayed in area RM10, multiple images including image MK8 displayed in area RM12, and image DT1 displayed in the upper section of area RM14 will not be updated.

[0109] 2-2: Variation 2 In the above embodiment, the projection system 1 includes two projection devices 30, projection device 30[1] and projection device 30[2]. However, the projection system 1 may include only one projection device 30.

[0110] Figure 11 shows an example of a display screen DI4 that the display control unit 118 displays on the display device 130 in this modified example. The display screen DI4 has regions RM16 to RM23. In the example shown in Figure 11, region RM16 contains an image similar to region RM5 of the display screen DI2 illustrated in Figure 8. Region RM17 contains an image similar to region RM6 of the display screen DI2. Region RM18 contains an image similar to region RM10 of the display screen DI3 illustrated in Figure 10. Region RM19 contains an image similar to region RM11 of the display screen DI3. Region RM20 contains an image similar to region RM12 of the display screen DI3. Region RM21 contains an image similar to region RM13 of the display screen DI3. Region RM22 contains an image similar to region RM7 of the display screen DI2 illustrated in Figure 8. Region RM22 contains an image similar to region RM8 of the display screen DI2.

[0111] 2-3: Variation 3 In the above embodiment, the "first image" and "second image" were displayed on a screen that did not include an overall image AP of the projection surface PF, such as display screen DI2. However, the "first image" and "second image" may also be displayed on a screen that includes an overall image AP of the projection surface PF, such as display screen DI1.

[0112] As an example, in response to an instruction operation on the first corner of the overall image AP of the projection surface PF, the first display control unit 118[1] may display the "first image" at the position closest to the first corner among the multiple corners included in the overall image AP. Alternatively, in response to an instruction operation on the first corner of the overall image AP of the projection surface PF, the second display control unit 118[2] may display the "second image" at the position closest to the first corner among the multiple corners included in the overall image AP. The "first corner" mentioned above is an example of a "first feature point." Similarly, "multiple corners" is an example of "multiple feature points."

[0113] Figures 12 and 13 show an example of a display screen DI5 that the display control unit 118 displays on the display device 130 in this modified example. The display screen DI5 includes areas RM23 and RM24. Area RM23 displays a slider bar SB, similar to area RM1 of the display screen DI1 illustrated in Figure 7. Area RM24 displays an overall image AP of the projection surface PF, similar to area RM2 of the display screen DI1.

[0114] As illustrated in Figure 12, when the user moves the mouse pointer close to the upper left corner CN1 of the overall image AP, images LI1 and LI2 are displayed at the position closest to the upper left corner CN1 among the four corners CN of the overall image AP. Image LI1 is the same image as the one displayed in area RM5 of the display screen DI2 illustrated in Figure 8. Image LI1 includes image MK2. Image LI2 is the same image as the one displayed in area RM6 of the display screen DI2. Image LI2 includes image MK3.

[0115] As illustrated in Figure 13, when the user moves the mouse pointer close to the lower left corner CN2 of the overall image AP, images LI3 and LI4 are displayed at the position closest to the lower left corner CN2 among the four corners CN of the overall image AP. Image LI3 is the same image as the one displayed in area RM7 of the display screen DI2 illustrated in Figure 8. Image LI3 includes image MK4. Also, image LI4 is the same image as the one displayed in area RM8 of the display screen DI2. Image LI4 includes image MK5. The same configuration applies to the upper right corner CN3 and the lower right corner CN4. Furthermore, moving the mouse pointer close to each corner of the overall image AP is an example of an "instruction operation."

[0116] 2-4: Variation 4 In the embodiment described above, the "first image" and the "second image" were displayed simultaneously. However, the "first image" and the "second image" may be displayed sequentially at different times.

[0117] For example, the user operation to display the "first image" on the display device 130 and the user operation to display the "second image" on the display device 130 may be different from each other. In this case, the user operation to display the "first image" is an example of the "first operation," and the user operation to display the "second image" is an example of the "second operation."

[0118] In other words, the "first image" is displayed when the receiving unit 112 receives a "first operation" for displaying the "first image" on the display device 130. The "second image" is displayed when the receiving unit 112 receives a "first operation" for displaying the "second image" on the display device 130.

[0119] If the timing of the execution of "Operation 1" and "Operation 2" are different, "Image 1" and "Image 2" will be displayed sequentially at different times.

[0120] Figures 14 to 16 show an example of a display screen DI6 that the display control unit 118 displays on the display device 130 in this modified example. The display screen DI6 has regions RM25 to RM28. Region RM25 displays a slider bar SB, similar to region RM1 of the display screen DI1 illustrated in Figure 7. Region RM26 displays an overall image AP of the projection surface PF, similar to region RM2 of the display screen DI1. Region RM27 displays either the "first image" or the "second image". Region RM28 displays buttons BT1 to BT4.

[0121] As shown in Figure 14, when button BT1 is operated by the user, the "first image" is displayed in area RM27. In the example shown in Figure 14, area RM27 displays an image similar to area RM5 of the display screen DI2 illustrated in Figure 8. This image includes image MK2. Here, the user's operation on button BT1 is an example of the "first operation". The image of button BT1 is an example of the "first operation image". In the example shown in Figure 14, the image of button BT1 includes information indicating the date and time the data used to display image MK2 was acquired. As another example, the image of button BT1 may also include information indicating the "first period" mentioned above. This information indicating the "first period" is an example of the "first information".

[0122] As shown in Figure 15, when button BT2 is operated by the user, a "second image" is displayed in area RM27. In the example shown in Figure 15, area RM27 displays an image similar to area RM6 of the display screen DI2 illustrated in Figure 8. This image includes image MK3. Here, the user's operation on button BT2 is an example of a "second operation". The image of button BT2 is an example of a "second operation image". In the example shown in Figure 15, the image of button BT2 includes information indicating the date and time the data used to display image MK3 was acquired. As another example, the image of button BT2 may also include information indicating the "second period" mentioned above. This information indicating the "second period" is an example of "second information".

[0123] As shown in Figure 16, when button BT3 is operated by the user, nothing is displayed in area RM27.

[0124] In the examples shown in Figures 14 and 15, area RM27 displayed a partial image PP including the upper left corner of the projection surface PF. To display a partial image PP including other corners of the projection surface PF in area RM27, the user switches the image displayed in area RM27 by operating button BT4.

[0125] 2-5: Variation 5 In the above embodiment, the "first image" and the "second image" were displayed on a display device 130 provided in the information processing device 10. However, the "first image" and the "second image" may also be projected onto the projection surface PF by a projection device 30[1] or projection device 30[2]. In this case, the projection device 30[1] and projection device 30[2] are examples of "display devices".

[0126] Specifically, the processing unit 110 provided in the information processing device 10 captures an image including the "first image" and the "second image" displayed on the display device 130. Then, the processing unit 110 may project the captured image onto the projection surface PF using the projection device 30[1] or projection device 30[2].

[0127] 2-6: Variation 6 In the above embodiment, the information processing device 10 and the projection device 30[1] were separate entities. However, the information processing device 10 and the projection device 30[1] may be implemented as a single device contained within the same housing. Similarly, the information processing device 10 and the projection device 30[2] were separate entities. However, the information processing device 10 and the projection device 30[2] may be implemented as a single device contained within the same housing.

[0128] 2-7: Variation 7 In the above embodiment, each of the projection device 30[1] and projection device 30[2] was equipped with an imaging device 320. However, at least one of these two imaging devices 320 may be provided in the information processing device 10.

[0129] 2-8: Variation 8 In the above embodiment, the flow shown in Figures 4 and 5 was executed by the processing unit 110. However, some or all of the flow shown in Figures 4 and 5 may be executed by the processing unit 330.

[0130] 3: Summary of this disclosure A summary of this disclosure is provided below.

[0131] (Note 1) A display method characterized by including: displaying a first image on a display device showing the position on the projection surface of a feature portion that serves as a marker on the projection surface, which is extracted based on a first image obtained by imaging the projection surface with an imaging device in a first period; and displaying a second image on the display device showing the position on the projection surface of a feature portion that serves as a marker on the projection surface, which is extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period.

[0132] With the above configuration, the user can understand whether the points extracted as corners of the projection surface are the same or nearly the same in two different time periods.

[0133] More specifically, with the above configuration, by the user checking the first and second images, the user can easily grasp the degree of agreement between the position of the feature portion on the projection surface in the first period and the position of the feature portion on the projection surface in the second period; that is, whether the location extracted as, for example, the corner of the projection surface is the same or nearly the same in the two periods.

[0134] (Note 2) The display method according to Note 1, wherein the first image is displayed superimposed on the first captured image including the projection surface, and the second image is displayed superimposed on the second captured image including the projection surface.

[0135] With the above configuration, users can more easily understand whether the areas extracted, for example, as corners of the projection surface, are the same or nearly the same in two different time periods.

[0136] (Note 3) The display method according to Note 1 or Note 2, wherein the first image is an image showing the coordinate values ​​of the position of the feature portion in the coordinate system of the first image, and the second image is an image showing the coordinate values ​​of the position of the feature portion in the coordinate system of the second image.

[0137] With the above configuration, users can more easily understand whether the areas extracted as, for example, the corners of the projection surface are the same or nearly the same in two different time periods.

[0138] (Note 4) The first image and the second image are displayed side by side on the display screen of the display device, using the display method described in any one of Notes 1 to 3.

[0139] With the above configuration, users can more easily understand whether the areas extracted, for example, as corners of the projection surface, are the same or nearly the same in two different time periods.

[0140] (Note 5) The display method according to any one of Notes 1 to 3, wherein the first image is displayed in response to a first operation for displaying the first image by the display device, and the second image is displayed in response to a second operation for displaying the second image by the display device.

[0141] According to the above configuration, the display method described in Appendix 5 allows the first and second images to be displayed at a timing that is convenient for the user.

[0142] (Appendix 6) The display method according to Appendix 5, further comprising displaying a first operation image that accepts the first operation by the display device, and displaying a second operation image that accepts the second operation by the display device, wherein the first operation image includes first information indicating the first period, and the second operation image includes second information indicating the second period.

[0143] With the above configuration, users can easily understand which operation image to select to display the desired image.

[0144] (Note 7) The display method described in any one of Notes 1 to 6, wherein the method of calculation for extracting feature portions as markers for the projection surface based on the first captured image is the same as the method of calculation for extracting feature portions as markers for the projection surface based on the second captured image.

[0145] According to the above configuration, the display method in Appendix 7 can improve the validity of the comparison results by standardizing the calculation method.

[0146] (Note 8) The display method according to any one of Notes 1 to 7, further comprising displaying an overall image of the projection surface by the display device, wherein displaying the first image by the display device means displaying the first image at the position closest to the first feature point among a plurality of feature points included in the overall image in response to an instruction operation for a first feature point included in the overall image, and displaying the second image by the display device means displaying the second image at the position closest to the first feature point among a plurality of feature points in response to the instruction operation.

[0147] When a user views the first and second images, displaying the first and second images containing the feature point indicated by the user near that feature point makes it easier for the user to understand the correspondence between that feature point and the first and second images.

[0148] (Note 9) An information processing apparatus comprising one or more processors that perform: a first display control which displays on a display device a first image indicating the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a first image obtained by imaging the projection surface with an imaging device in a first period; and a second display control which displays on a display device a second image indicating the position on the projection surface of a feature portion that serves as a marker for the projection surface, extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period.

[0149] With the above configuration, the user can understand whether the points extracted as corners of the projection surface are the same or nearly the same in two different time periods.

[0150] More specifically, with the above configuration, by the user checking the first and second images, the user can easily grasp the degree of agreement between the position of the feature portion on the projection surface in the first period and the position of the feature portion on the projection surface in the second period; that is, whether the location extracted as, for example, the corner of the projection surface is the same or nearly the same in the two periods.

[0151] (Note 10) A program that causes a processor to perform the following actions: display a first image on a display device showing the position on the projection surface of a feature portion that serves as a marker for the projection surface, which is extracted based on a first image obtained by imaging the projection surface with an imaging device during a first period; and display a second image on the display device showing the position on the projection surface of a feature portion that serves as a marker for the projection surface, which is extracted based on a second image obtained by imaging the projection surface with the imaging device during a second period following the first period; and display a second image on the display device showing the position on the projection surface of a feature portion that serves as a marker for the projection surface, which is extracted based on a second image obtained by imaging the projection surface with an imaging device during a second period following the first period.

[0152] With the above configuration, the user can understand whether the points extracted as corners of the projection surface are the same or nearly the same in two different time periods.

[0153] More specifically, with the above configuration, by the user checking the first and second images, the user can easily grasp the degree of agreement between the position of the feature portion on the projection surface in the first period and the position of the feature portion on the projection surface in the second period; that is, whether the location extracted as, for example, the corner of the projection surface is the same or nearly the same in the two periods. [Explanation of symbols]

[0154] 1…Projection system, 10…Information processing device, 30…Projection device, 110…Processing device, 111…Communication control unit, 112…Reception unit, 113…Setting unit, 114…Acquisition unit, 115…Extraction unit, 116…Measurement unit, 117…Calculation unit, 118…Display control unit, 118[1]…First display control unit, 118[2]…Second display control unit, 120…Storage device, 130…Display device, 140…Input device, 150…Communication device, 310…Projector, 320…Imaging device, 330…Processing device, 331…Communication control 332... Acquisition unit, 333... Projection control unit, 334... Imaging control unit, 340... Storage device, 350... Communication device, AC... Anchor image, AI... Overall image, AP... Overall image, BT... Button, CM... Candidate, CN... Corner, DI... Display screen, DT... Image, ED... Edge, FG... Image, KI... Partial image, LI... Image, MK... Image, NET... Communication network, PF... Projection surface, PI... Projected image, PP... Partial image, PR... Control program, RM1... Area, SB... Slide bar, TI... Acquired image.

Claims

1. A first image indicating the position on the projection surface of a feature portion that serves as a marker on the projection surface, which is extracted based on a first image obtained by imaging the projection surface with an imaging device during a first period, is displayed on a display device; The display device displays a second image indicating the position on the projection surface of a feature portion of the projection surface that serves as a marker, which is extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period. A method of display characterized by including

2. The first image is displayed superimposed on the first captured image which includes the projection surface. The second image is displayed superimposed on the second image, which includes the projection surface. The display method according to claim 1.

3. The first image is an image showing the coordinate values ​​of the position of the feature portion in the coordinate system of the first captured image, The second image is an image showing the coordinate values ​​of the position of the feature portion in the coordinate system of the second captured image. The display method according to claim 1.

4. The first image and the second image are displayed side by side on the display screen of the display device. The display method according to claim 1.

5. The first image is displayed in response to receiving a first operation for displaying the first image by the display device. The second image is displayed in response to receiving a second operation for displaying the second image by the display device. The display method according to claim 1.

6. The first operation image that accepts the first operation is displayed by the display device, The further includes displaying a second operation image that accepts the second operation using the display device, The first operational image includes first information indicating the first period, The second operational image includes second information indicating the second period, The display method according to claim 5.

7. The method for extracting feature portions as markers for the projection surface based on the first captured image is the same as the method for extracting feature portions as markers for the projection surface based on the second captured image. The display method according to claim 1.

8. The method further includes displaying the entire image of the projection surface using the display device, Displaying the first image by the display device means displaying the first image at the position closest to the first feature point among a plurality of feature points included in the overall image, in response to an instruction operation on a first feature point included in the overall image. Displaying the second image by the display device means displaying the second image at the position closest to the first feature point among the plurality of feature points in response to the instruction operation. The display method according to claim 1.

9. A first display control that displays a first image on a display device showing the position on the projection surface of a feature portion of the projection surface that serves as a marker, extracted based on a first image obtained by imaging the projection surface with an imaging device during a first period, within the first image obtained. A second display control that displays a second image on the display device showing the position on the projection surface of a feature portion of the projection surface that serves as a marker, extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period, An information processing device characterized by comprising one or more processors that perform the following.

10. A first image indicating the position on the projection surface of a feature portion that serves as a marker on the projection surface, which is extracted based on a first image obtained by imaging the projection surface with an imaging device during a first period, is displayed on a display device; The processor is instructed to display on the display device a second image indicating the position on the projection surface of a feature portion of the projection surface that serves as a marker, which is extracted based on a second image obtained by imaging the projection surface with the imaging device in a second period following the first period. program.

Images