Calibration method and projection-type display system

Abstract

Provided is a calibration method wherein: a first image captured by a camera at a first timing and showing an area in which an image is projected by a projection-type display device, is acquired; a plurality of feature points in the acquired first image are detected; the first image with the detected plurality of feature points superimposed thereon is displayed on a monitor; a user operation for selecting, as a plurality of reference points, some of the detected plurality of feature points is accepted; a second image captured by the camera at a second timing, which is later than the first timing, and showing said area is acquired; and the selected plurality of reference points and the acquired second image are used to detect deviation in the relative positions of the camera and the area.

Description

Calibration method and projection display system

[0001] The present disclosure relates to a method for calibrating a projection display system.

[0002] Japanese Patent Application Laid-Open No. 2003-124222 discloses a technique for detecting misalignment of an imaging device using an image obtained by the imaging device.

[0003] Japanese Patent Application Publication No. 11-261993

[0004] The present disclosure provides a calibration method that can suppress deterioration in calibration accuracy related to image projection.

[0005] A calibration method according to one aspect of the present disclosure includes acquiring a first image taken by a camera at a first timing, the first image showing an area onto which an image is projected by a projection display device, detecting a plurality of feature points in the acquired first image, superimposing the detected plurality of feature points on the first image and displaying it on a monitor, accepting a user operation to select some of the detected plurality of feature points as a plurality of reference points, acquiring a second image taken by the camera at a second timing later than the first timing, the second image showing the area, and detecting a deviation in the relative position between the camera and the area using the selected plurality of reference points and the acquired second image.

[0006] A calibration method according to an aspect of the present disclosure can suppress deterioration in calibration accuracy related to image projection.

[0007] FIG. 1 is a diagram showing the configuration of a projection display system according to an embodiment. FIG. 2 is a flowchart of an initial calibration process. FIG. 3 is a flowchart of a first example of a feature point detection process. FIG. 4 is a diagram showing an example of a graphical user interface used in the first example of a feature point detection process. FIG. 5 is a flowchart of a second example of a feature point detection process. FIG. 6 is a diagram showing an example of a graphical user interface used in the second example of a feature point detection process. FIG. 7 is a flowchart of a calibration process. FIG. 8 is a diagram showing the configuration of a projection display system according to a first modification. FIG. 9 is a diagram showing the configuration of a projection display system according to a second modification.

[0008] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not described in independent claims will be described as optional components.

[0009] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0010] (Embodiment) [Configuration] First, the configuration of a projection display system according to an embodiment will be described. Fig. 1 is a diagram showing the configuration of a projection display system according to an embodiment.

[0011] The projection display system 10 is a system that supports multi-projection, in which a single image is constructed from images projected by two projection display devices 20. The projection display system 10 also performs a calibration process using an image captured by a camera 30 to project an image aligned with a screen 70 (along the area inside the frame 70a of the screen 70). The projection display system 10 includes two projection display devices 20, a camera 30, and an information processing device 40. The projection display system 10 is only required to include at least one projection display device 20, and does not necessarily need to support multi-projection. The projection display device 20 may also include multiple cameras 30.

[0012] The projection display device 20 projects an image onto the screen 70 under the control of the information processing device 40. The projection display device 20 is realized by an optical system including, for example, a laser light source, a phosphor wheel, an image display element, and a projection lens. Specifically, the image display element is a digital micromirror device (DMD) or a reflective liquid crystal panel (LCOS: Liquid Crystal On Silicon).

[0013] When multi-projection is achieved using two projection display devices 20, a technique called edge blending is used, for example. The two images projected by the two projection display devices 20 have an overlapping portion 70b where they overlap each other, and edge blending projects the images so that the brightness of the two images cross-fade in the overlapping portion 70b. This prevents the seam between the two images from being noticeable.

[0014] Under the control of the information processing device 40, the camera 30 captures an image that shows the entire screen 70. The screen 70 is an example of an area onto which an image is projected by the projection display device 20. The camera 30 is realized by an image sensor, a lens, and the like.

[0015] The information processing device 40 performs information processing to realize the above-described multi-projection. Such information processing includes a calibration process using images captured by the camera 30 to project images aligned with the screen 70. The information processing device 40 is, for example, a general-purpose device such as a PC (personal computer) installed with an application program for executing the above-described information processing, but may also be a device dedicated to the projection display system 10. The information processing device 40 may also be a server containing the application program. Specifically, the information processing device 40 includes an input device 41, a monitor 42, a communication circuit 43, a control circuit 44, and a storage device 45.

[0016] The input device 41 receives operations from a user. The input device 41 is, for example, a keyboard and a mouse, but may also be a touch panel or the like.

[0017] The monitor 42 displays an image. The monitor 42 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel. The monitor 42 may be a device separate from the information processing device 40.

[0018] The communication circuit 43 is a communication circuit that enables the information processing device 40 to communicate with the two projection display devices 20 and the camera 30. The communication circuit 43 communicates with the two projection display devices 20 and the camera 30, for example, via a local communication network. The communication performed by the communication circuit 43 is, for example, wired communication, but may also be wireless communication. There are no particular limitations on the communication standard used for communication.

[0019] Furthermore, when the information processing device 40 is a server, the communication circuit 43 may communicate with the two projection display devices 20 and the camera 30 via a WAN (Wide Area Network) or the Internet instead of a local communication network. In this case, a user can access the information processing device 40, which is a server, from a PC via the network and control the information processing device 40. In this case, the input device 41 and the monitor 42 do not need to be provided in the information processing device 40, but may be provided in a local user PC.

[0020] The control circuit 44 performs the above-described information processing. Specifically, the control circuit 44 is realized by a processor or a microcomputer. The functions of the control circuit 44 are realized when the processor or the microcomputer constituting the control circuit 44 executes a computer program stored in the storage device 45.

[0021] The storage device 45 is a storage device that stores information necessary for the above-described information processing, such as a computer program executed by the control circuit 44. Specifically, the storage device 45 is realized by a semiconductor memory or an HDD (Hard Disk Drive).

[0022] [Initial Calibration Processing] Next, a description will be given of the initial calibration processing of the projection display system 10. The initial calibration processing is processing that is performed when the projection display system 10 starts operating. FIG. 2 is a flowchart of the initial calibration processing of the projection display system 10.

[0023] First, the control circuit 44 sets the projection positions so that the projection positions of the images from the two projection display devices 20 are aligned with the screen 70 (S11). Here, the projection position refers to a rectangular area on the screen 70 where the image projected by the projection display device 20 is displayed, and includes the coordinates of the four vertices of the rectangular area. In this embodiment, the projection position is specified by coordinates including the four vertices of the screen 70. In other words, the position of the screen 70 is set as the projection position. In step S11, the control circuit 44 transmits control signals to each of the two projection display devices 20 via the communication circuit 43, thereby causing each of the two projection display devices 20 to project an image.

[0024] At this time, the control circuit 44 adjusts the projection direction, lens shift amount, zoom magnification, focal length, etc. of each of the two projection display devices 20 so that the projection range of the two images projected by the two projection display devices 20 encompasses the screen 70.

[0025] After step S11, the control circuit 44 performs a process to store the projection positions (specifically, the coordinates of the feature points in the test images) of the test images of each of the two projection display devices 20. The process to store the projection positions of the test images of one projection display device 20 will be described below, but this process is performed for each of the two projection display devices 20.

[0026] The control circuit 44 transmits a control signal to the projection display device 20 via the communication circuit 43, thereby causing the projection display device 20 to project a test image (S12). The test image may be any image suitable for detecting feature points in the next step S13, such as an image having a predetermined color pattern.

[0027] Next, the control circuit 44 detects feature points in the test image projected onto the screen 70 and stores the coordinates of the detected feature points in the storage device 45 (S13). The detection of feature points in step S13 is a process for identifying the projection position of the image of the projection display device 20. In other words, it is a process for identifying the relative positions of the projection display device 20 and the screen 70.

[0028] Specifically, the control circuit 44 transmits a control signal to the camera 30 via the communication circuit 43, thereby causing the camera 30 to capture an image (still image) including the screen 70 with the test image projected thereon. The control circuit 44 acquires the captured image (more specifically, image information of the captured image) from the camera 30 via the communication circuit 43, and processes the acquired image to detect a plurality of feature points. At this time, the control circuit 44 uses, for example, an algorithm optimized for detecting feature points appearing in the test image.

[0029] For example, in the test image, the intersections of areas of different colors are detected as feature points. In the following, the coordinates of the detected N feature points are collectively expressed as FP(N) = {FP １ , FP ２ , FP ３ , ..., FP ｎ , ..., FP Ｎ}, where 1, 2, 3, ... N are the identification information (ID) of the feature points, and FP １ , FP ２ , FP ３ are the coordinates of each feature point. FP(N) is stored in the storage device 45.

[0030] Next, the control circuit 44 performs geometric correction and edge blending (S14). If necessary, the control circuit 44 performs geometric correction on each of the two images projected by the two projection display devices 20 so that the edges of the projected images conform to the frame 70a of the screen 70. The control circuit 44 also performs edge blending on the overlapping portion 70b of the two images projected by the two projection display devices 20. The control circuit 44 automatically performs this series of processes using the images captured by the camera 30 (more specifically, image information obtained from the camera 30 via the communication circuit 43).

[0031] Immediately after step S14, the control circuit 44 performs a feature point detection process (S15). The feature point detection process here is a process for identifying the relative positions of the camera 30 and the screen 70. The feature point detection process will be described in detail later.

[0032] [Example 1 of Feature Point Detection Processing] A specific example of the feature point detection processing performed in step S15 will now be described. Fig. 3 is a flowchart of Example 1 of feature point detection processing. Fig. 4 is a diagram showing an example of a graphical user interface (GUI) displayed on the monitor 42 by the control circuit 44 in Example 1 of feature point detection processing.

[0033] Initially, the main area 42a in Fig. 4 displays in real time an image currently being captured by the camera 30. In this state, the user performs an operation to select a marker type, and the input device 41 accepts this operation (S15a). Specifically, the user operates the pull-down button 42b provided in the Type of Markers column of the GUI in Fig. 4 to select one of "Non-Luminous," "Luminous," and "Reflective," and the input device 41 accepts this selection operation.

[0034] 1, when no markers are provided around the periphery of the screen and feature points are to be detected for the screen 70, "Non-Luminous" is selected. Furthermore, as will be described later, when feature points are to be detected in a state where non-luminous markers are provided around the periphery of the screen 70, "Non-Luminous" is selected, and when feature points are to be detected in a state where luminous markers are provided around the periphery of the screen 70, "Luminous" is selected. Furthermore, when feature points are to be detected in a state where reflective markers are provided around the periphery of the screen 70, "Reflective" is selected.

[0035] Next, the user operates the auto-shooting button 42c, and the input device 41 accepts this operation (S15b). In response to this acceptance of the operation, the control circuit 44 transmits a control signal to the camera 30 via the communication circuit 43, thereby causing the camera 30 to capture an image (still image) (S15c). At this time, the image capture conditions (white balance, shutter speed, ISO sensitivity, F-number, etc.) instructed by the information processing device 40 (control circuit 44) to the camera 30 by the control signal are automatically set according to the selection result in step S15a. Hereinafter, the image captured in step S15c will also be referred to as the "first image."

[0036] In addition, instead of operating steps S15a and S15b, the user can manually set the shooting conditions by operating multiple pull-down buttons provided in the shooting condition setting section 42d, and then take the first image by operating the shooting button 42e provided in the shooting condition setting section 42d.

[0037] When the first image is captured by the camera 30, the control circuit 44 acquires the first image (more specifically, image information of the first image) from the camera 30 via the communication circuit 43 and detects a plurality of feature points from the acquired first image as a processing target (S15d). At this time, the control circuit 44 uses, for example, an algorithm for detecting feature points that appear at the four corners (vertices of the rectangle) of the rectangular screen 70. The control circuit 44 may detect points with large contrast differences, i.e., points with relatively high spatial frequencies, as feature points in the first image acquired by the camera 30. Note that any existing algorithm may be used to detect the feature points, and open source software or the like may be used as appropriate.

[0038] Next, the control circuit 44 superimposes a plurality of objects 42j corresponding to the detected plurality of feature points on the first image and displays them in the main area 42a (monitor 42) (S15e). In the present embodiment, the objects 42j representing each feature point are formed by combining a circle and four lines. However, any shape can be used for the objects representing each feature point. At this time, the control circuit 44 detects the four corners of the screen 70 projected in the first image and sets specific areas 42f at each of the four corners of the screen 70. The specific areas 42f are set based on the projection position of the image. Specifically, for each of the specific areas 42f corresponding to each of the four corners of the screen 70, at least a portion of the specific area 42f is located outside the corresponding vertex of the screen 70. In other words, it is sufficient that the specific area 42f includes a vertex of the screen 70 and at least a portion of the specific area 42f is located outside the screen 70. Most preferably, when the specific area 42f is rectangular, each specific area 42f is positioned so that, with the point where two diagonals of the screen 70 intersect as the center, the vertex 42fv of the specific area 42f closest to the center is adjacent to the outside of the adjacent vertex of the screen 70.

[0039] Furthermore, the control circuit 44 displays the set specific region 42f on the first image in a manner that allows the region 42f to be distinguished from other regions. Specifically, the control circuit 44 displays an object (indicator) indicating the specific region 42f superimposed on the first image. The object indicating the specific region 42f is, for example, a rectangular frame, and is illustrated by a dashed line in Fig. 4. Note that the specific form (shape, size, color, etc.) of the object indicating the specific region 42f is not particularly limited.

[0040] To ensure the accuracy of the calibration, it is necessary to detect a predetermined number of feature points (for example, four or more) in each of the four specific regions 42 f. Therefore, the control circuit 44 changes the display mode of the specific region 42 f depending on whether the predetermined number of feature points or more has been detected.

[0041] For example, the control circuit 44 displays an object (more specifically, the inside of the rectangular frame of the object) indicating a specific region 42f (first region) in which a predetermined number or more of feature points have been detected in a semi-transparent green color (first aspect). The control circuit 44 then displays an object indicating a specific region 42f (second region) in which a predetermined number or more of feature points have not been detected in a semi-transparent red color (second aspect). This allows the control circuit 44 to notify the user whether each of the four specific regions 42f meets the required number of feature points. In other words, the control circuit 44 can notify the user of the number of feature points in each of the four specific regions 42f.

[0042] It is not essential that the display mode of the object indicating the specific region 42f be changed based on color, but may be changed based on the color, shape, size, line type of the frame, or blinking cycle of the object. The change in the display mode of the object indicating the specific region 42f may be achieved by combining two or more of the color, shape, size, line type of the frame, and blinking cycle of the object. The number of feature points included in the specific region 42f may be displayed numerically in the specific region 42f.

[0043] Incidentally, feature points are used to check the relative positional deviation between the camera 30 and the screen 70, but they may be detected at unexpected positions unrelated to the screen 70. For example, feature points may be detected at positions unrelated to the screen 70 depending on the lighting conditions when the first image is captured. Also, feature points may be detected at positions unrelated to the screen 70 if an obstacle (such as a cable or a stepladder) is captured in the first image. In this way, using feature points detected at positions unrelated to the screen 70 for calibration contributes to a deterioration in the accuracy of the calibration.

[0044] Therefore, the projection display system 10 has a function for excluding unnecessary feature points. Specifically, the user operates the masking area designation section 42g1, and the input device 41 accepts this operation (S15f1). As a result, the user can designate a masking area 42h1 at a desired position in the first image. The control circuit 44 excludes (invalidates) feature points included in the designated masking area 42h1 from the plurality of feature points detected in step S15d (S15g1). As a result, the feature points remaining after excluding one or more feature points that belong to the masking area 42h1 from the plurality of feature points detected in step S15d are selected as final feature points. Hereinafter, the final feature point will also be referred to as a reference point.

[0045] The control circuit 44 changes the display mode of the reference point and the excluded feature points. For example, the control circuit 44 may not display the excluded feature points in the main area 42a (cancel the display), but may display them in a mode different from that of the reference point. A mode different from that of the reference point means displaying them in a different color from that of the reference point or in a different icon shape from that of the reference point.

[0046] Next, the control circuit 44 determines whether each of the four specific regions 42f has the required number of reference points (final feature points) (S15h). If the control circuit 44 determines that each of the four specific regions 42f has the required number of reference points (Yes in S15h), it activates the Complete button 42i (S15i). The user operates the Complete button 42i, and the input device 41 accepts this operation (S15j). Upon accepting this operation, the control circuit 44 associates the first image with the coordinates of all of the reference points within the first image and stores them in the storage device 45 (S15k). Note that the coordinates of the reference points are assigned with identification information (ID) of the reference points.

[0047] On the other hand, if the control circuit 44 determines that at least one of the four specific regions 42f does not satisfy the required number of reference points (No in S15h), it disables the done button 42i (S15l) and does not accept any operation of the done button by the user. In this case, the user may change the shooting conditions, etc., and then shoot the first image again (step S15c).

[0048] As described above, in Example 1 of the feature point detection process, the projection display system 10 acquires a first image captured by the camera 30, the first image showing the area onto which the image will be projected by the projection display device 20 (the area surrounded by the frame 70a of the screen 70), and detects multiple feature points in the acquired first image. The projection display system 10 then superimposes the detected multiple feature points on the first image and displays them on the monitor 42, and accepts an operation to specify a masking area 42h1 within the first image when the multiple feature points are displayed superimposed on the first image. As a result, the feature points excluding one or more feature points that belong to the specified masking area 42h1 are selected from the multiple feature points as multiple reference points.

[0049] Such a projection display system 10 has a function of excluding unnecessary feature points from the reference points, thereby making it possible to prevent deterioration in the accuracy of the calibration process and failure of the calibration process.

[0050] In step S15f1, an operation of designating a masking region 42h1 was performed as an operation of selecting only a portion of the plurality of feature points as a plurality of reference points. In step S15f1, the user may also perform an operation of directly designating feature points that the user wants to exclude using a pointer or the like. In other words, in step S15f1, an operation of directly designating a portion of the plurality of feature points may be performed as an operation of selecting a portion of the plurality of feature points as a plurality of reference points. In this case, the feature points excluding the one or more designated feature points from among the plurality of feature points are selected as a plurality of reference points.

[0051] Furthermore, the projection display system 10 may accept both an operation of specifying a masking area 42h1 and an operation of directly specifying the feature points to be excluded using a pointer or the like as an operation of selecting only a portion of the feature points as multiple reference points.

[0052] [Example 2 of Feature Point Detection Processing] Another specific example of the feature point detection processing performed in step S15 will now be described. Fig. 5 is a flowchart of Example 2 of feature point detection processing. Fig. 6 is a diagram showing an example of a graphical user interface displayed on the monitor 42 by the control circuit 44 in Example 2 of feature point detection processing.

[0053] The processing of steps S15a to S15e is the same as that of Example 1 of the feature point detection processing, and therefore a description thereof will be omitted. After step S15e, the user operates the valid area designation unit 42g2, and the input device 41 accepts this operation (S15f2). As a result, the user can designate the valid area 42h2 at a desired position in the first image. The control circuit 44 excludes (invalidates) feature points that are not included within the designated valid area 42h2 from the multiple feature points detected in step S15d (S15g2). As a result, only feature points that belong within the valid area 42h2 are selected as reference points from among the multiple feature points detected in step S15d.

[0054] Next, the control circuit 44 determines whether each of the four specific regions 42f has the required number of reference points (S15h). The subsequent processes of steps S15i to S15l are the same as those in example 1 of the feature point detection process, and therefore, a description thereof will be omitted.

[0055] As described above, in Example 2 of the feature point detection process, the projection display system 10 acquires a first image captured by the camera 30, the first image showing the area onto which the image will be projected by the projection display device 20 (the area surrounded by the frame 70a of the screen 70), and detects multiple feature points in the acquired first image. The projection display system 10 then superimposes the detected multiple feature points on the first image and displays them on the monitor 42, and accepts an operation to specify an effective area 42h2 within the first image when the multiple feature points are displayed superimposed on the first image. As a result, only those feature points within the specified effective area 42h2 are selected as multiple reference points from among the multiple feature points.

[0056] In this way, the projection display system 10 has the function of excluding unnecessary feature points from the reference points, thereby making it possible to prevent deterioration in the accuracy of the calibration process and failure of the calibration process.

[0057] In step S15f2, an operation of specifying the valid region 42h2 was performed as an operation of selecting only some of the multiple feature points as multiple reference points. In step S15f2, the user may perform an operation of directly specifying the feature points that the user wants to select as reference points using a pointer or the like. In other words, in step S15f2, an operation of directly specifying multiple feature points may be performed as an operation of selecting some of the multiple feature points as multiple reference points. In this case, only the specified feature points from the multiple feature points are selected as multiple reference points.

[0058] Furthermore, the projection display system 10 may accept both an operation of specifying the effective area 42h2 and an operation of directly specifying the feature points to be selected as reference points using a pointer or the like, as an operation of selecting only a portion of the feature points as reference points.

[0059] [Calibration Process] Next, the calibration process performed after the first timing at which the initial calibration process is performed will be described. Fig. 7 is a flowchart of the calibration process.

[0060] When a predetermined second timing arrives, which is later than the first timing, the control circuit 44 uses this opportunity to send a control signal to the camera 30 via the communication circuit 43, thereby causing the camera 30 to capture an image (still image) (S21). For example, the control circuit 44 stores the shooting conditions during the initial calibration process in the storage device 45, and by referring to the storage device 45, causes the camera to capture an image under the same shooting conditions as when the initial calibration process was performed. Hereinafter, the image captured in step S21 will also be referred to as the second image.

[0061] Note that the second image is captured automatically without user operation. For example, the control circuit 44 manages schedule information for the calibration process (e.g., once per predetermined period, such as once a day, once a week, or once a month) and periodically captures the second image based on the schedule information. The schedule information is stored (registered) in advance in the storage device 45 by, for example, a user operating the input device 41. Note that the capture schedule may be managed by a device higher than the information processing device 40, such as a cloud server (not shown), and the information processing device 40 may capture the second image based on a command from the higher-level device.

[0062] Furthermore, the second image may be captured based on a user operation. For example, when the user views the image projected on the screen 70 and determines that calibration is necessary, the user may perform a predetermined operation on the input device 41 to input a command to capture the second image, which then causes the second image to be captured.

[0063] When the second image is captured by the camera 30, the control circuit 44 acquires the second image (more specifically, image information of the second image) from the camera 30 via the communication circuit 43, and detects a plurality of feature points from the acquired second image as a processing target (S22). The method for detecting the plurality of feature points is the same as the method described in step S15d, except that the processing target is the second image.

[0064] Next, the control circuit 44 reads the coordinates of the first image and the multiple reference points stored in the storage device 45 in step S15k (S23). The control circuit 44 then calculates the coordinate deviation amount at the current time (second timing) for each of the multiple reference points read (S24). Specifically, the control circuit 44 identifies the feature amount of each of the multiple reference points read by comparing the coordinates of each of the multiple reference points with the first image. For each of the multiple reference points, the control circuit 44 selects one feature point from the multiple feature points detected in step S22 that has a feature amount most similar to the feature amount of the reference point. That is, the control circuit 44 selects some of the multiple feature points of the second image as multiple reference points of the second image based on the multiple reference points of the first image. The control circuit 44 then calculates the coordinate deviation amount from the coordinates of the selected reference point.

[0065] The coordinate deviation amount of each of the multiple reference points indicates a deviation (change) in the relative position between the camera 30 and the screen 70. In other words, it can be said that in step S23, a deviation in the relative position between the camera 30 and the screen 70 is detected. That is, the deviation (change) in the relative position between the camera 30 and the screen 70 is detected based on the deviation (coordinate deviation amount) between the reference point of the first image and the reference point of the second image.

[0066] Next, the control circuit 44 calculates a correction coefficient for the coordinate system of the camera 30 based on the coordinate shift amount of each of the plurality of reference points (S25). Based on the coordinate shift amount of each of the plurality of reference points, the control circuit 44 calculates, as a correction coefficient, a homography matrix H for converting the coordinate system of the camera 30 at the current time (second timing) into the coordinate system of the camera 30 at the time when the initial calibration process was executed (first timing). The homography matrix H is expressed by the following equation.

[0067]

[0068] Here, if the coordinates of the second image (the image captured at the current time) are (x, y), the corresponding coordinates (x', y') of the first image (the image captured at the time the initial calibration process was performed) are expressed by the following equation:

[0069]

[0070] Note that a method for calculating the homography matrix H from the deviation amounts of four or more reference points in a two-axis coordinate system is well known, and therefore a detailed description thereof will be omitted here.

[0071] Next, the control circuit 44 performs a process to correct the projection position of the image for each of the two projection display devices 20. The following describes the process to correct the projection position of the image for one projection display device 20, but this process is performed for each of the two projection display devices 20.

[0072] First, the control circuit 44 causes the projection display device 20 to project a test image (S26) by transmitting a control signal to the projection display device 20 via the communication circuit 43. The test image is the same as the test image projected in step S12 of the initial calibration process.

[0073] Next, the control circuit 44 detects feature points in the test image projected onto the screen 70, and stores the coordinates FP'(N) of the detected feature points in the storage device 45 (S27). This process is similar to step S13 of the initial calibration process.

[0074] Next, the control circuit 44 corrects the coordinates FP'(N) of the feature points detected in step S27 based on the correction coefficients (homography matrix H) calculated in step S25 (S28). That is, the control circuit 44 corrects the coordinates of the feature points detected in the current positional relationship between the camera 30 and the screen 70 to coordinates that would be obtained if the feature points were detected in the positional relationship between the camera 30 and the screen 70 when the initial calibration process was performed. Specifically, the control circuit 44 calculates the corrected coordinates FP''(N) using the following equation:

[0075]

[0076] Next, the control circuit 44 calculates the amount of deviation between the coordinate FP(N) of the feature point stored in step S13 of the initial calibration process and the coordinate FP"(N) of the feature point obtained in step S28 (S29). The amount of deviation (change) in the relative position between the camera 30 and the screen 70, which is included in the coordinate FP'(N) before correction, has been removed from the corrected coordinate FP"(N). Therefore, the amount of deviation between the coordinate FP(N) and the coordinate FP"(N) indicates the deviation (change) in the relative position between the projection display device 20 and the screen 70. In other words, in step S29, the deviation in the relative position between the projection display device 20 and the screen 70 has been detected.

[0077] Next, the control circuit 44 calculates new geometric correction parameters based on the coordinate deviation amount calculated in step S29 (S30). Specifically, the control circuit 44 calculates the coordinates FP'' for each feature point n. ｎ is the coordinate FP ｎ The homography matrix H ｎ Calculate the current geometric correction parameters (homography matrix) Hold ｎ The homography matrix H ｎ By multiplying the new geometric correction parameters (homography matrix) H ｎ That is, the new geometric correction parameter H ｎ Is Hnew ｎ =Hold ｎ ×H ｎ It is calculated based on the formula:

[0078] The control circuit 44 then sets the new geometric correction parameters calculated in step S30 in the projection display device 20 (S31). Specifically, the control circuit 44 transmits a setting command including the new geometric correction parameters to the projection display device 20 via the communication circuit 43. As a result, the geometric correction parameters in the projection display device 20 are changed. In other words, the image projected by the projection display device 20 is corrected.

[0079] If the control circuit 44 determines that the amount of deviation calculated in step S29 is smaller than a predetermined value, it may omit the processes of steps S30 and S31. For example, if the coordinate FP"(N) and the coordinate FP(N) are substantially the same and there is no deviation in the relative positions of the projection display device 20 and the screen 70, there is no need to correct the image projected by the projection display device 20.

[0080] As described above, in the calibration process, the projection display system 10 acquires a second image captured by the camera 30, the second image showing the area surrounded by the frame 70a of the screen 70, and detects the deviation in the relative position between the camera 30 and the area using the multiple reference points selected in the initial calibration process and the acquired second image. Furthermore, the projection display system 10 corrects the image projected by the projection display device 20 based on the detected deviation in the relative position.

[0081] If the calibration process is performed periodically without requiring user operation as described above, it is possible to prevent misalignment of the image projection positions of the two projection display devices 20 without requiring human intervention. Furthermore, as described above, because unnecessary feature points are excluded from the reference points in the initial calibration process, the projection display system 10 can prevent a deterioration in the accuracy of the calibration process. Since the exclusion of unnecessary feature points from the reference points reduces the possibility of the calibration process failing, it becomes easier to automate the calibration process. Even if the calibration process is not automated, there is the advantage that even a user who is not familiar with the operation can instruct the calibration process with fewer operations.

[0082] [Modification 1] Next, a description will be given of the configuration of a projection display system according to Modification 1. Fig. 8 is a diagram showing the configuration of a projection display system according to Modification 1.

[0083] The projection display system 10a differs from the projection display system 10 in that it projects an image onto a wall surface 80 instead of a screen 70, but other configurations are the same as those of the projection display system 10, and therefore a description thereof will be omitted.

[0084] Wall surface 80 is used as an area onto which an image is projected by projection display device 20. Four non-luminous markers 81 are provided on wall surface 80. The number and arrangement of non-luminous markers 81 are not particularly limited.

[0085] In the feature point detection process executed by the projection display system 10a, feature points appearing in (or around) the plurality of non-luminous markers 81 displayed in the first image are detected, and the control circuit 44 uses an algorithm suitable for detecting such feature points. The process for detecting feature points in the second image is similar to the feature point detection process (the process for detecting feature points in the first image).

[0086] This projection display system 10a also has a function of excluding unnecessary feature points from reference points in the initial calibration process (feature point detection process), similar to the projection display system 10. Therefore, the projection display system 10a can prevent the accuracy of the calibration process from deteriorating.

[0087] [Modification 2] Next, a description will be given of the configuration of a projection display system according to Modification 2. Fig. 9 is a diagram showing the configuration of a projection display system according to Modification 2.

[0088] The projection display system 10b projects an image onto a screen 90. Unlike the screen 70, the screen 90 does not have a frame 70a, and instead has luminous markers 91 at positions corresponding to the four vertices (corners) of the screen 90.

[0089] Unlike the projection display system 10, the projection display system 10b includes a marker control device 50. The other configurations of the projection display system 10b are the same as those of the projection display system 10, and therefore a description thereof will be omitted.

[0090] The marker control device 50 turns on and off the light-emitting markers 91 under the control of the information processing device 40. The light-emitting markers 91 are realized by, for example, LED (Light Emitting Diode) elements. The number and arrangement of the light-emitting markers 91 are not particularly limited.

[0091] In the feature point detection process executed by the projection display system 10b, the marker control device 50 captures a first image with the plurality of light-emitting markers 91 illuminated, and detects feature points appearing in the plurality of light-emitting markers 91 (or their surroundings) that appear in the first image. The control circuit 44 uses an algorithm suitable for detecting such feature points. Note that "Luminous" is selected in the marker type selection in step S15a. The process of detecting feature points in the second image is similar to the feature point detection process (the process of detecting feature points in the first image).

[0092] This projection display system 10b also has a function of excluding unnecessary feature points from reference points in the initial calibration process (feature point detection process), similar to the projection display system 10. Therefore, the projection display system 10b can prevent the accuracy of the calibration process from deteriorating.

[0093] [Variation 3] Next, the configuration of a projection display system according to Variation 3 will be described. The projection display system according to Variation 3 differs from that according to Variation 1 in that it uses reflective markers instead of the non-luminous markers 81. In the feature point detection process performed by the projection display system according to Variation 3, the projection display device projects an all-white image and detects light reflected by the reflective markers. Here, an all-white image is an image in which the pixel value is 255 in the case of a grayscale image, or in which the pixel value of each color is 255 in the case of an RGB image. This makes it possible to detect feature points with a simple configuration even in a dark environment where the projection screen is placed.

[0094] [Effects, etc.] As described above, the calibration method executed by a computer such as the projection display system 10 (information processing device 40) acquires a first image taken by the camera 30 at a first timing, the first image showing an area onto which an image is projected by the projection display device 20, detects a plurality of feature points in the acquired first image, superimposes the detected plurality of feature points on the first image and displays it on the monitor 42, accepts a user operation to select some of the detected plurality of feature points as a plurality of reference points, acquires a second image taken by the camera 30 at a second timing after the first timing, the second image showing the area, and detects a deviation in the relative position between the camera 30 and the area using the selected plurality of reference points and the acquired second image.

[0095] Such a calibration method can eliminate unnecessary feature points from the reference points, and therefore can prevent deterioration in the accuracy of calibration related to image projection (the calibration process described above).

[0096] Furthermore, for example, the above operation is an operation of designating an area within the first image when a plurality of feature points are displayed superimposed on the first image, and feature points excluding one or more feature points that belong to the designated area are selected as the plurality of reference points from among the plurality of feature points. Such an area is, for example, the masking area 42h1 in the above embodiment.

[0097] According to this calibration method, the user can exclude unnecessary feature points from the reference points by specifying the masking region 42h1.

[0098] Furthermore, for example, the above operation is an operation of designating an area within the first image when a plurality of feature points are displayed superimposed on the first image, and feature points within the designated area are selected as a plurality of reference points from among the plurality of feature points. Such an area is, for example, the effective area 42h2 in the above embodiment.

[0099] According to this calibration method, the user can exclude unnecessary feature points from the reference points by specifying the valid area 42h2.

[0100] Also, for example, the above operation is an operation of specifying one or more of the multiple feature points when the multiple feature points are displayed superimposed on the first image, and the feature points excluding the specified one or more feature points from the multiple feature points are selected as multiple reference points.

[0101] According to this calibration method, the user can directly specify unnecessary feature points, thereby excluding the unnecessary feature points from the reference points.

[0102] Also, for example, the above operation is an operation of specifying some of the multiple feature points when the multiple feature points are displayed superimposed on the first image, and the specified feature points among the multiple feature points are selected as multiple reference points.

[0103] According to such a calibration method, the user can directly specify the necessary feature points, thereby excluding unnecessary feature points from the reference points.

[0104] For example, the calibration method further includes displaying on the monitor 42 a specific area 42f in the first image where feature points need to be detected.

[0105] According to this calibration method, the user can grasp the specific region 42f in which feature points need to be detected.

[0106] For example, the calibration method further notifies the user of the number of feature points detected within the specific region 42f.

[0107] According to this calibration method, the user can grasp the number of feature points in the specific region 42f.

[0108] Also, for example, the number of feature points detected within the specific region 42f is notified to the user by the display mode of the object indicating the specific region 42f.

[0109] According to this calibration method, the user can grasp the number of feature points in the specific region 42f.

[0110] Furthermore, for example, the calibration method further corrects the image projected by the projection display device 20 based on the detected deviation in relative position.

[0111] According to this calibration method, the projection display device 20 can project an image along the above-mentioned area, taking into account the detected deviation in relative position.

[0112] The projection display system 10 also includes a projection display device 20, a camera 30, and an information processing device 40. The information processing device 40 acquires a first image captured by the camera 30 at a first timing, the first image showing an area onto which an image is projected by the projection display device 20, detects a plurality of feature points in the acquired first image, superimposes the detected plurality of feature points on the first image and displays it on a monitor 42, accepts a user operation to select some of the detected plurality of feature points as a plurality of reference points, acquires a second image captured by the camera 30 at a second timing after the first timing, the second image showing the area, and detects a deviation in the relative position between the camera 30 and the area using the selected plurality of reference points and the acquired second image.

[0113] Such a projection display system 10 can exclude unnecessary feature points from the reference points, and therefore can prevent deterioration in the accuracy of calibration (the calibration process described above) related to image projection.

[0114] Other Embodiments Although the calibration method and projection display system according to the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0115] For example, in the above embodiment, the frame of the screen or a marker is used to detect the feature points, but other objects may be used to detect the feature points. As long as the relative position between the camera and the surface onto which the image is projected can be identified, the object is not particularly limited.

[0116] For example, in the above embodiment, the projection display system is realized by multiple devices, but it may also be realized as a single device. For example, the projection display system may be realized as a single device equivalent to an information processing device. When the projection display system is realized by multiple devices, the components of the projection display system may be distributed among the multiple devices in any manner.

[0117] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0118] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0119] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0120] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, device, method, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM. Also, the present disclosure may be realized as any combination of a system, device, method, integrated circuit, computer program, and recording medium. For example, the present disclosure may be realized as the projection display system or information processing device 40 of the above-described embodiment. The present disclosure may be realized as a program (computer program product) for causing a computer to execute the calibration method of the above-described embodiment, or as a computer-readable non-transitory recording medium on which such a program is stored.

[0121] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0122] The calibration method of the present disclosure can prevent deterioration in the accuracy of calibration of a projection display system.

[0123] 10, 10a, 10b Projection display system 20 Projection display device 30 Camera 40 Information processing device 41 Input device 42 Monitor 42a Main area 42b Pull-down button 42c Auto shooting button 42d Shooting condition setting section 42e Shooting button 42f Specific area 42fv Vertex 42g1 Masking area designation section 42g2 Effective area designation section 42h1 Masking area 42h2 Effective area 42i Complete button 43 Communication circuit 44 Control circuit 45 Storage device 50 Marker control device 70, 90 Screen 70a Frame 70b Superimposed portion 80 Wall surface 81 Non-luminous marker 91 Luminous marker

Claims

1. acquiring a first image captured by a camera at a first timing, the first image showing an area onto which an image is projected by the projection display device; Detecting a plurality of feature points in the acquired first image; superimposing the detected plurality of feature points on the first image and displaying the same on a monitor; accepting a user operation to select some of the detected feature points as a plurality of reference points; acquiring a second image captured by the camera at a second timing that is later than the first timing, the second image including the area; Detecting a deviation in the relative position between the camera and the area using the selected reference points and the acquired second image. Calibration method.

2. the operation is an operation of designating an area within the first image when the plurality of feature points are displayed superimposed on the first image, Among the plurality of feature points, feature points excluding one or more feature points belonging to a designated area are selected as the plurality of reference points. The calibration method according to claim 1 .

3. the operation is an operation of designating an area within the first image when the plurality of feature points are displayed superimposed on the first image, Among the plurality of feature points, feature points that belong to a designated area are selected as the plurality of reference points. The calibration method according to claim 1 .

4. the operation is an operation of designating one or more of the plurality of feature points when the plurality of feature points are displayed superimposed on the first image, Feature points excluding one or more designated feature points from the plurality of feature points are selected as the plurality of reference points. The calibration method according to claim 1 .

5. the operation is an operation of designating some of the plurality of feature points when the plurality of feature points are displayed superimposed on the first image, A designated part of the plurality of feature points is selected as the plurality of reference points. The calibration method according to claim 1 .

6. moreover, a first area in the first image in which a predetermined number of feature points or more have been detected is displayed on the monitor in a first manner; A second area in the first image in which less than the predetermined number of feature points is detected is displayed on the monitor in a second manner different from the first manner. The calibration method according to claim 1 .

7. Furthermore, the number of feature points detected in the second region is notified to the user. The calibration method according to claim 6 .

8. The number of the feature points detected in the second area is notified to the user by a display mode of an object indicating the second area. The calibration method according to claim 7 .

9. Furthermore, the image projected by the projection display device is corrected based on the detected deviation of the relative position. The calibration method according to any one of claims 1 to 8.

10. moreover, Detecting a plurality of feature points in the second image; selecting a portion of the plurality of feature points of the second image as a plurality of reference points of the second image based on a plurality of reference points of the first image; The deviation of the relative position is detected based on deviations between a plurality of reference points of the first image and a plurality of reference points of the second image. The calibration method according to claim 1 .

11. A program for causing a computer to execute the calibration method according to any one of claims 1 to 8.

12. a projection display device; A camera and an information processing device; The information processing device includes: acquiring a first image captured by a camera at a first timing, the first image showing an area onto which an image is projected by the projection display device; Detecting a plurality of feature points in the acquired first image; superimposing the detected plurality of feature points on the first image and displaying the same on a monitor; accepting a user operation to select some of the detected feature points as a plurality of reference points; acquiring a second image captured by the camera at a second timing that is later than the first timing, the second image including the area; Detecting a deviation in the relative position between the camera and the area using the selected reference points and the acquired second image. Projection display system.