Image processing method, image processing system, and program

The image processing method corrects projection image quality by capturing and using feature points outside the display area to avoid marker superimposition, ensuring consistent image quality during marker display.

JP7947898B2Active Publication Date: 2026-11-11SEIKO EPSON CORP
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2022166060
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-10-17
Publication Date
2026-11-11
Estimated Expiration
2042-10-17

AI Technical Summary

Technical Problem

The quality of a projection image is degraded due to the superimposition of markers used for correction, which is not present in conventional techniques.

Method used

An image processing method that captures a projection image and feature points outside the display area, correcting the color or luminance based on these points to avoid marker superimposition.

Benefits of technology

Maintains image quality by correcting color or luminance without marker superimposition, allowing for continuous image projection without recalibration when positional changes are minor.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007947898000001
    Figure 0007947898000001
  • Figure 0007947898000002
    Figure 0007947898000002
  • Figure 0007947898000003
    Figure 0007947898000003
Patent Text Reader

Abstract

To perform correction of a color or brightness of a projection image to be projected on a colored or patterned projection surface without reducing quality of the projection image.SOLUTION: A projector 10 comprises an optical device 120, a camera 130, and a processor 110. The processor 110 acquires, from the camera 130, a picked-up image including a projection image to be projected on a projection surface SS from the optical device 120 on the basis of an image signal and a plurality of feature points located in an adjustment area on an outside of an area where the projection image is displayed. The processor 110 determines the range of correction data, in a correction data group, which is used for correction of the color of the projection image on the basis of the position of the area corresponding to the projection image in the picked-up image, and corrects the color of the projection image by using the correction data included in the determined range.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an image processing method, an image processing system, and a program.

Background Art

[0002] As a conventional technique for detecting the position of an image or video displayed on a projection surface, the technique disclosed in Patent Document 1 can be cited. In Patent Document 1, markers are superimposed on four corner portions of a video, and the coordinates of the corner portions on the projection surface are calculated based on a captured image of the projection surface on which the video with the markers superimposed on each corner portion is displayed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] When a marker is superimposed on a projection image for correction of the projection image, the quality of the projection image is degraded due to the superimposition of the marker during the period in which the marker is displayed, compared to the case where the marker is not displayed.

Means for Solving the Problem

[0005] An image processing method according to one aspect of the present disclosure includes acquiring a captured image that includes a projection image projected from an optical device onto the projection surface based on an image signal, and a plurality of feature points located in a region outside the region where the projection image is displayed on the projection surface, by imaging the projection surface, and correcting the color or luminance of the projection image based on the captured image.

[0006] Furthermore, an image processing system according to one aspect of the present disclosure includes a processing unit which captures a projection surface to obtain an image including a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed, and corrects the color or brightness of the projection image based on the image.

[0007] Furthermore, a program according to one aspect of the present disclosure causes a computer to capture an image of a projection surface, thereby acquiring an image of the projection surface including a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed, and to correct the color or brightness of the projection image based on the image. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of a projector 10 according to one embodiment of the image processing system disclosed herein. [Figure 2] This figure shows an example of a projection surface SS onto which an image is projected from projector 10. [Figure 3] This diagram shows the relationship between the display area and the adjustment area on the projection surface SS. [Figure 4] This figure shows an example of a marker image projected from the projector 10 onto the projection surface SS along with the projected image. [Figure 5] This diagram illustrates the range of correction data used after a change in the position of the display area. [Figure 6] This is a flowchart showing the flow of the image processing method executed by the processing unit 110 of the projector 10 according to the program PRA. [Modes for carrying out the invention]

[0009] The embodiments described below are subject to various technically preferred limitations. However, the embodiments of this disclosure are not limited to those described below. (A: Embodiment) Figure 1 shows an example configuration of a projector 10 that performs an image processing method according to one embodiment of the present disclosure. An image supply device (not shown) is connected to the projector 10 by wire or wireless. The projector 10 projects an image represented by an image signal supplied from the image supply device onto a projection surface SS. A specific example of the image supply device is, for example, a personal computer. In this embodiment, the projection surface SS is the wall surface of the room where the projector 10 is installed. Unlike a projection screen, the projection surface SS has various patterns drawn on it in various colors. Figure 2 shows an example of a projection surface SS. In the projection surface SS shown in Figure 2, for example, a polka dot pattern is drawn on a white background in a color other than white, such as light blue. The diagonal hatching in Figure 2 represents a color other than white.

[0010] As shown in Figure 1, the projector 10 includes a processing unit 110, an optical device 120, a camera 130, and a storage device 140. In addition to the processing unit 110, optical device 120, camera 130, and storage device 140, the projector 10 also includes a communication device that communicates with an image supply device by wire or wireless connection, and an input device such as a keyboard that accepts various user input operations. However, the communication device and the input device are not shown in Figure 1 because they are not very relevant to this disclosure.

[0011] The processing unit 110 is comprised of a processor, such as a CPU (Central Processing Unit), i.e., a computer. The processing unit 110 may consist of a single processor or multiple processors. The processing unit 110 functions as the control center for the projector 10 by operating according to the program PRA stored in the storage device 140.

[0012] The optical device 120 includes a projection lens, a liquid crystal drive unit, a liquid crystal panel, and a light source unit. Note that the projection lens, liquid crystal drive unit, liquid crystal panel, and light source unit are not shown in Figure 1. The liquid crystal drive unit drives the liquid crystal panel according to an image signal supplied from the processing unit 110, thereby drawing the image represented by this image signal onto the liquid crystal panel. The light source unit includes, for example, a halogen lamp or a laser diode. Light from the light source unit is modulated pixel by pixel in the liquid crystal panel and projected as image light through the projection lens. This image light is projected onto the projection surface SS, thereby displaying an image on the projection surface SS. Note that the optical device 120 only needs to be configured to project image light onto the projection surface SS, and may include a digital mirror device (DMD) instead of the liquid crystal drive unit and liquid crystal panel.

[0013] In this embodiment, an image signal representing the largest image that the optical device 120 can project is supplied to the projector 10 from the image supply device, while the optical device 120, under the control of the processing device 110, projects an image smaller than the image represented by the image signal onto the projection surface SS. For example, as shown in Figure 3, if the outline of the largest image that the optical device 120 can project is represented by frame Z1, the optical device 120 reduces the image represented by the image signal provided from the image supply device to an image whose outline is represented by frame Z2 and projects it onto the projection surface SS. Hereinafter, the area inside frame Z2 will be referred to as the display area, and the image represented by the image signal provided from the image supply device will be referred to as the image to be processed, and the image projected onto the display area will be referred to as the projected image. The area outside frame Z2, between frame Z2 and frame Z1, will be referred to as the adjustment area. In this embodiment, the optical device 120 has a resolution of 1920 × 1080 pixels, and the resolution of the image to be processed is also 1920 × 1080 pixels, but the resolution of the projected image, i.e., the size of the display area, is 1600 × 800 pixels. For this reason, in this embodiment, adjustment areas with a width of 160 pixels are provided on the left and right sides of the display area, and adjustment areas with a width of 90 pixels are provided on the top and bottom of the display area.

[0014] Camera 130 is equipped with an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), which converts light focused by an optical system such as a lens into an electrical signal. Camera 130 is positioned so that its optical axis is directed towards the center of the projection surface SS. Camera 130 is fitted with a filter that transmits visible light in order to receive visible light. Camera 130 captures an image of the projection surface SS under the control of the processing unit 110. Each time camera 130 captures an image of the projection surface SS, it outputs an image signal representing the captured image to the processing unit 110.

[0015] The storage device 140 is a recording medium that can be read by the processing unit 110. The storage device 140 includes, for example, non-volatile memory and volatile memory. The non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, RAM (Random Access Memory). The non-volatile memory of the storage device 140 is pre-stored with a program PRA that causes the processing unit 110 to execute the image processing method of the present disclosure. The volatile memory of the storage device 140 is used by the processing unit 110 as a work area when executing the program PRA.

[0016] Furthermore, the volatile memory stores conversion data for mutually converting the position on the captured image captured by the camera 130 and the position on the liquid crystal panel in the optical device 120, as well as a set of correction data used for color correction of the image projected onto the projection surface SS. The correction data set is a collection of correction data that shows the two-dimensional coordinates indicating each position on the liquid crystal panel and the correction amounts for R, G, and B at that position; specifically, it is a 3D-LUT, or three-dimensional lookup table. The conversion data and the correction data set are generated by performing calibration and stored in the volatile memory.

[0017] Calibration refers to a process for associating a camera coordinate system that defines positions on a captured image captured by a camera 130 with a panel coordinate system that defines positions on a liquid crystal panel in an optical device 120. A specific example of the conversion data includes a first conversion matrix for mutually converting between the camera coordinate system and the panel coordinate system. The conversion data is generated, for example, by comparing a captured image obtained by imaging, with the camera 130, the projection surface SS on which a pattern image such as Gaussian dots is projected from the optical device 120, with the pattern image. By this conversion data, positions on the projection surface SS are associated with positions on the liquid crystal panel.

[0018] The correction data group is generated based on 125 captured images obtained by sequentially projecting a total of 125 images with R, G, and B color values of (0,0,0), (0,0,64), (0,0,128)... (255,255,192), (255,255,255) onto the display area, and imaging the projection surface SS in a state where each image is projected onto the display area with the camera 130. More specifically, the correction data group is obtained by performing projective transformation or the like using the conversion data on each of the 125 captured images to extract a portion corresponding to the display area, and applying to the extracted portion exist It is generated by calculating the difference between the pixel value of the pixel and the pixel value of the pixel corresponding to the relevant pixel in the liquid crystal panel. Therefore, the correction data group can also be said to be a collection of correction data indicating two-dimensional coordinates representing each position on the display area on the projection surface SS and the correction amounts of each of the R, G, and B colors at the relevant position. Note that when generating the correction data group, existing techniques may be appropriately used, including color reduction processing for reducing the number of colors. In addition, in the process of sequentially capturing the 125 captured images, it is preferable that the exposure and shutter speed of the camera 130 are fixed. Further, in order to remove white noise of the camera 130 and fine patterns on the projection surface SS, it is preferable that each of the 125 captured images is subjected to noise removal processing by a median filter.

[0019] The processing device 110 reads the program PRA from the non-volatile memory to the volatile memory triggered by power-on of the projector 10, and starts executing the read program PRA. The processing device 110 operating in accordance with the program PRA functions as the initial setting unit 110a, projection control unit 110b, and correction control unit 110c shown in FIG. 1. The initial setting unit 110a, projection control unit 110b, and correction control unit 110c shown in FIG. 1 are software modules implemented by operating the processing device 110 in accordance with the program PRA. The functions each of the initial setting unit 110a, projection control unit 110b, and correction control unit 110c shown in FIG. 1 are responsible for are as follows.

[0020] The initial setting unit 110a performs the aforementioned calibration, generates conversion data and a correction data group, and stores them in the volatile memory.

[0021] The projection control unit 110b reduces the processing target image to an image of a size corresponding to the display area, further performs color correction using the correction data group, and causes the optical device 120 to project the reduced and color-corrected image as a projection image.

[0022] The correction control unit 110c detects a change in the relative positional relationship of the projector 10 with respect to the projection surface SS in a state where the projection image is projected onto the projection surface SS. The change in the relative positional relationship of the projector 10 with respect to the projection surface SS occurs, for example, when the user accidentally pushes the projector 10. The correction control unit 110c periodically executes a process of projecting marker images representing markers corresponding to each of the plurality of feature points onto the adjustment area, and causing the camera 130 to capture an image of the projection surface SS in a state where the projection image and the marker images are projected, under the circumstance that the projection image is projected onto the projection surface SS. In other words, the correction control unit 110c projects the marker images onto the adjustment area prior to acquiring the captured image.

[0023] Figure 4 shows an example of a projection surface SS on which a projection image and a marker image are projected. As shown in Figure 4, the marker in this embodiment is a dot drawn in white on a black background, and the dot pattern image is the marker image. Since the marker image is projected onto the adjustment area, i.e., the area outside the contour of the display area, the projection image and the marker image do not overlap. The correction control unit 110c causes the camera 130 to capture an image of the projection surface SS on which both the projection image and the marker image are projected, and acquires an image signal from the camera 130 that represents the captured image, which includes the projection image and images of multiple feature points located in an area outside the display area.

[0024] The correction control unit 110c detects changes in the relative position of the projector 10 with respect to the projection surface SS based on the captured image taken by the camera 130. Changes in the relative position of the projector 10 with respect to the projection surface SS appear as changes in the area occupied by the display area on the projection surface SS. Therefore, the correction control unit 110c detects changes in the relative position of the projector 10 with respect to the projection surface SS by detecting changes in the position of feature points present in the adjustment area in the captured image taken periodically by the camera 130. Specifically, the correction control unit 110c detects how much and in which direction the display area has moved from its initial position by tracking changes in the position of each feature point located closest to the four corners of the contour of the display area.

[0025] When the correction control unit 110c detects a change in the position of the display area on the projection surface SS, it generates a second transformation matrix that associates the position of the display area after the change with the position of the display area before the change. For detecting changes in the position of feature points and deriving the second transformation matrix, known image registration techniques, such as SIFT (Scale-invariant feature transform) or SURF (Speeded Up Robust Features), can be appropriately employed.

[0026] Next, the correction control unit 110c determines whether the amount of change in the position of the display area on the projection surface SS is less than a predetermined threshold, and if the amount of change is greater than or equal to the predetermined threshold, it issues a notification prompting the system to re-execute the calibration. The index indicating the amount of change in the position of the display area is the matrix element that constitutes the second transformation matrix, i.e., the largest component of the matrix.

[0027] In contrast, if the change in the position of the display area on the projection surface SS is not zero and is less than a predetermined threshold, the correction control unit 110c determines the range of correction data from the correction data group to be used for color correction of the image projected onto the changed display area. Subsequently, the projection control unit 110b performs color correction using the correction data within the range determined by the correction control unit 110c. For example, suppose that due to a change in the relative positional relationship between the projector 10 and the projection surface SS, the position of the display area shown by the solid line in Figure 5 changes to the position shown by the dotted line. In this case, the correction data group has already been created by calibration for the area shown by hatching in Figure 5, that is, the overlapping area between the display area before the position change and the display area after the position change. Therefore, for the pixels on the liquid crystal panel corresponding to that area, the projection control unit 110b uses the first and second transformation matrices to determine the position on the projection surface SS corresponding to the pixel after the position of the display area has changed, and performs color correction using the correction data from the created correction data that corresponds to the determined position. In contrast, the projection control unit 110b does not perform color correction on pixels outside the overlapping area between the display area before the position change and the display area after the position change. The reason why the system prompts for re-calibration when the amount of change in the position of the display area on the projection surface SS exceeds a predetermined threshold is that the overlapping area between the display area before the position change and the display area after the position change is narrow, and therefore there is little correction data that can be reused.

[0028] Furthermore, the processing unit 110, which operates according to the program PRA, executes an image processing method whose flow is shown in the flowchart of Figure 6. As shown in Figure 6, this image processing method includes an initial setup process SA100, a projection control process SA110, a first determination process SA120, a second determination process SA130, a notification process SA140, a final decision process SA150, a third determination process SA160, and a sleep process SA170.

[0029] In the initial setup process SA100, the processing unit 110 functions as an initial setup unit 110a. In the initial setup process SA100, the processing unit 110 performs the aforementioned calibration and stores the conversion data and correction data group in volatile memory.

[0030] In projection control processing SA110, the processing unit 110 functions as projection control unit 110b. In projection control processing SA110, the processing unit 110 applies image reduction processing and color correction using correction data to the image signal supplied from the image supply device and supplies it to the optical device 120, which then projects the reduced and color-corrected image onto the display area.

[0031] In the first determination process SA120, the second determination process SA130, the notification process SA140, the decision process SA150, and the third determination process SA160, the processing unit 110 functions as a correction control unit 110c. In the first determination process SA120, the processing unit 110 projects a marker image onto the adjustment area and causes the camera 130 to capture an image including the marker image and the projected image, and acquires the captured image. Then, in the first determination process SA120, it is determined, based on the captured image, whether or not the position of the display area on the projection surface SS has changed.

[0032] If the result of the first determination process SA120 is "No," that is, if it is determined that the position of the display area on the projection surface SS has not changed, the processing unit 110 executes the third determination process SA160. In the third determination process SA160, the processing unit 110 determines whether or not an input operation to instruct the end of projection has been performed. If the result of the third determination process SA160 is "Yes," that is, if it is determined that an input operation to instruct the end of projection has been performed, the processing unit 110 terminates the execution of this image processing and ends the projection of the image onto the projection surface SS. If the result of the third determination process SA160 is "No," that is, if it is determined that an input operation to instruct the end of projection has not been performed, the processing unit 110 executes the sleep process SA170. 。 During the sleep process SA170, the processing unit 110 enters a sleep state, i.e., a dormant state, for a certain period of time, such as a few milliseconds. After the sleep process SA170 is completed, the processing unit 110 resumes the processing from the projection control process SA110 onward.

[0033] If the result of the first determination process SA120 is "Yes," that is, if it is determined that the position of the display area on the projection surface SS has changed, the processing unit 110 executes the second determination process SA130. In the second determination process SA130, the processing unit 110 determines whether the amount of change in the position of the display area on the projection surface SS is less than a predetermined threshold. If the result of the second determination process SA130 is "No," that is, if it is determined that the amount of change in the position of the display area on the projection surface SS is greater than or equal to a predetermined threshold, the processing unit 110 executes the notification process SA140. In the notification process SA140, the processing unit 110 issues a notification prompting the user to re-execute the calibration. After completing the execution of the notification process SA140, the processing unit 110 executes the third determination process SA160 described above.

[0034] If the result of the second determination process SA130 is "Yes," that is, if it is determined that the amount of change in the position of the display area on the projection surface SS is less than a predetermined threshold, the processing unit 110 executes the decision process SA150. In the decision process SA150, the processing unit 110 determines the range of correction data from the correction data group to be used for color correction of the image to be projected onto the changed display area. After completing the execution of the decision process SA150, the processing unit 110 executes the third determination process SA160. As described above, if the result of the third determination process SA160 is "Yes," that is, if it is determined that an input operation to instruct the end of projection has been made, the processing unit 110 terminates the execution of this image processing and ends the projection of the image onto the projection surface SS. If the result of the third determination process SA160 is "No," that is, if it is determined that no input operation to instruct the end of projection has been performed, the processing unit 110 executes the sleep process SA170, and after the sleep process SA170 is completed, it executes the projection control process SA110 and subsequent processes again. In the projection control process SA110, which is executed after the determination process SA150, color correction is performed using the correction data of the range determined in the determination process SA150.

[0035] As described above, according to this embodiment, the marker image for detecting the relative positional relationship between the projection surface SS and the projector 10 does not overlap with the projected image. Therefore, when this positional relationship changes, it becomes possible to perform color correction of the projected image projected onto a projection surface that has color or pattern applied, without degrading the quality of the projected image. In addition, according to this embodiment, even if the relative positional relationship of the projector 10 with respect to the projection surface SS changes, if the amount of change is small, it becomes possible to perform color correction of the projected image without performing calibration again.

[0036] (B: Transformation) Each of the above embodiments can be modified as follows. (1) In the above embodiment, the correction data set was a collection of correction data indicating two-dimensional coordinates indicating a position on the projection surface SS and correction amounts for each of the R, G, and B colors at that position. However, the correction data set in this disclosure may be a collection of correction data indicating two-dimensional coordinates indicating a position on the projection surface SS and correction amounts for the brightness of the pixels displayed at that position. Alternatively, the correction data set may be a collection of correction data indicating the color correction amount and correction data indicating the brightness correction amount. In short, in the image processing method in this disclosure, the color or brightness of the projected image should be corrected based on an captured image that includes a projected image projected from the optical device 120 onto the projection surface SS based on an image signal and a plurality of feature points located in an adjustment area outside the display area. In addition to the marker image, an image of a color correction pattern may also be projected into the adjustment area.

[0037] (2) For example, when a strongly blue image is projected onto the display area and a strongly red marker image is displayed on the adjustment area, the color or brightness of the image displayed on the display area and the marker image displayed on the adjustment area are so different that it may cause discomfort to the viewer. To avoid causing such discomfort to the viewer, a histogram of the color or brightness of the entire image projected onto the display area or the area around the outer edge of the image may be obtained, and the marker image may be projected at a timing when the difference between the histogram of the marker image and the histogram of the marker image falls below a predetermined threshold. Alternatively, the marker image may be projected using a marker image from among several marker images with different colors, where the difference between the color or brightness histogram of the above image and the color or brightness histogram of the marker image falls below a predetermined threshold.

[0038] (3) The size of the image represented by the image signal output from the image supply device to the projector 10 may be smaller than the maximum size of the image that the optical device 120 can project onto the projection surface SS. In addition, the non-display area generated during WARP processing such as trapezoidal correction may be considered as the adjustment area. Furthermore, although an adjustment area is always provided in the above embodiment, similar to the L-shaped screen used in television broadcasts for displaying election results or disaster information, an adjustment area may not be provided under normal circumstances, and may be provided only when the user instructs the detection of the relative positional relationship with the projection surface SS.

[0039] (4) In the above embodiment, the processing device 110 projected the marker image onto the optical device 120 along with the projected image. However, the markers for extracting feature points may be pre-placed on the projection surface SS by, for example, being attached to the projection surface SS, or a pattern drawn on the projection surface SS may be used as a marker. If markers are pre-placed on the projection surface SS, the projection of the marker image from the projector 10 onto the projection surface SS can be omitted. In other words, the projection of the marker image is not an essential component of the image processing method of this disclosure and can be omitted.

[0040] (5) In the above embodiment, the camera 130 was included in the projector 10, but the camera 130 may not be included in the projector 10 and may be a separate device. Also, the optical device 120 may be a separate device from the processing device 110. In short, the present disclosure can be applied to any image processing system that includes an optical device 120 that projects an image onto a projection surface SS, a camera 130 that captures an image of the projection surface SS, and a processing device 110 that controls the operation of the optical device 120 and the camera 130.

[0041] (6) In the above embodiment, the optical device that projects the marker image onto the adjustment area and the optical device that projects the projected image onto the display area were the same, but they may be separate optical devices. In this case, since the projection position of the optical device that projects the marker image onto the adjustment area needs to be changed in synchronization with the positional shift of the display area, the optical device must be fixed to the projector body and configured not to interfere with the display area. In addition to the positional shift of the projector body, the projection position of the optical device that projects the marker image onto the adjustment area needs to move in accordance with user operations such as enlarging or reducing the projected image or lens shift. If these conditions are met, it is not necessarily required to track feature points near the display area, and some positional correction can be performed by illuminating or photographing any point on the projection surface SS.

[0042] (7) In the above embodiment, the initial setting unit 110a, the projection control unit 110b, and the correction control unit 110c were software modules. However, one or more of the initial setting unit 110a, the projection control unit 110b, and the correction control unit 110c, or all of them, may be hardware modules such as ASICs (Application Specific Integrated Circuits). Even if one or more of the initial setting unit 110a, the projection control unit 110b, and the correction control unit 110c, or all of them, are hardware modules, the same effects as in the above embodiment will be achieved.

[0043] (8) Program PRA may be manufactured as a standalone product and may be provided for a fee or free of charge. Specific forms of providing Program PRA include providing it by writing it to a computer-readable recording medium such as flash ROM, or providing it by downloading it via a telecommunications line such as the Internet. By operating a general-purpose computer according to Program PRA provided in these forms, it becomes possible to have the computer execute the image processing method of the present disclosure.

[0044] (C: Summary of this disclosure) This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms. The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate. A summary of this disclosure is provided below.

[0045] (Note 1) An image processing method according to one aspect of the present disclosure includes: acquiring an image of a projection surface by imaging the projection surface, which includes a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed; and correcting the color or brightness of the projection image based on the image. According to the image processing method of (Note 1), since the color or brightness of the projection image is corrected based on a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed, there is no need to display markers corresponding to the feature points in the area where the projection image is displayed, and the quality of the projection image does not deteriorate even during the period when markers are displayed compared to when markers are not displayed.

[0046] (Note 2) A more preferred embodiment of the image processing method is the image processing method described in (Note 1), which includes correcting the color or brightness of the projected image by determining a range of coordinates used to correct the color of the projected image from among correction data represented by two-dimensional coordinates used to correct the color of the projected image, based on the position of the region in the captured image corresponding to the projected image, and correcting the color of the projected image using the correction data included in the range. According to the image processing method in (Note 2), based on the position of the region in the captured image corresponding to the projected image, a range of coordinates used to correct the color of the projected image from among correction data represented by two-dimensional coordinates used to correct the color of the projected image can be determined, and the color of the projected image can be corrected using the correction data included in the range.

[0047] (Note 3) Another preferred embodiment of the image processing method is the image processing method described in (Appendix 1), wherein the projection surface includes a marker corresponding to each of the plurality of feature points. According to the image processing method in (Appendix 3), the color or brightness of the projected image can be corrected based on the markers corresponding to each of the plurality of feature points without degrading the quality of the projected image compared to when the markers are not displayed, even during the period when the markers are displayed.

[0048] (Note 4) A further preferred embodiment of the image processing method is the image processing method of (Note 3), wherein the size of the projected image is smaller than the size of the largest image that the optical device can project onto the projection surface, and prior to acquiring the captured image, the optical device projects a marker image representing the marker between the contour of the largest image and the contour of the projected image. According to the image processing method of (Note 4), the optical device can correct the color or brightness of the projected image based on the image of the marker displayed between the contour of the largest image that can be projected onto the projection surface and the contour of the projected image.

[0049] (Note 5) A further preferred embodiment of the image processing method is the image processing method of (Note 3), wherein the size of the projected image is smaller than the size of the image indicated by the image signal, and prior to acquiring the captured image, a marker image representing the marker is projected from the optical device between the contour of the image when the image is projected at the size indicated by the image signal and the contour of the projected image. According to the image processing method of (Note 5), the color or brightness of the projected image can be corrected based on the image of the marker displayed between the contour of the image when the image is projected at the size indicated by the image signal and the contour of the projected image.

[0050] (Note 6) A more preferred embodiment of the image processing method is the image processing method described in (Appendix 3) or (Appendix 4), wherein the projected image and the marker image do not overlap. In the image processing method of (Appendix 6), since the projected image and the marker image do not overlap, the color or brightness of the projected image can be corrected based on the marker image without degrading the quality of the projected image compared to when the marker image is not displayed, even during the period when the marker image is displayed.

[0051] (Note 7) A more preferred embodiment of the image processing method is an image processing method according to any one of (Appendix 1) to (Appendix 6), further comprising projecting an image of a color correction pattern from the optical device onto an area outside the projected image on the projection surface. According to the image processing method of (Appendix 7), a plurality of feature points can be extracted from the image of the color correction pattern projected from the optical device onto an area outside the projected image on the projection surface, and the color or brightness of the projected image can be corrected based on the plurality of feature points.

[0052] (Note 8) An image processing system according to one aspect of the present disclosure includes a processing unit which captures an image of a projection surface, thereby acquiring an image of the projection surface which includes a projection image projected from an optical device onto the projection surface based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed, and correcting the color or brightness of the projection image based on the image of the captured image. According to the image processing system (Note 8), similar to the image processing method (Note 1), the color or brightness of the projection image is corrected based on a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed. Therefore, it is not necessary to display markers corresponding to the feature points in the area where the projection image is displayed, and the quality of the projection image does not deteriorate even during the period when markers are displayed compared to when markers are not displayed.

[0053] (Note 9) A program according to one aspect of the present disclosure causes a computer to capture an image of a projection surface, thereby acquiring an image of the projection surface including a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed; and correct the color or brightness of the projection image based on the image. According to the program in (Note 9), similar to the image processing method in (Note 1), the color or brightness of the projection image is corrected based on a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed. Therefore, it is not necessary to display markers corresponding to the feature points in the area where the projection image is displayed, and the quality of the projection image does not deteriorate even during the period when markers are displayed compared to when markers are not displayed. [Explanation of Symbols]

[0054] 10...Projector, 110...Processing device, 110a...Initial setup unit, 110b...Projection control unit, 110c...Correction control unit, 120...Optical device, 130...Camera, 140...Storage device, PRA...Program, SS...Projection surface.

Claims

1. By imaging the projection surface, an image is obtained that includes a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed. Correcting the color or brightness of the projected image based on the captured image, Includes, When correcting the color or brightness of a projected image based on the captured image, if a change in the position of the area where the projected image is displayed is detected based on the captured image, and the amount of the change in position is less than a predetermined threshold, the color or brightness of the projected image is corrected using correction data corresponding to the overlapping area between the area before the position change and the area after the position change, and no color or brightness correction is performed on pixels outside the overlapping area. Image processing methods.

2. Correcting the color or brightness of the aforementioned projected image is Based on the position of the region corresponding to the projection image in the captured image, the range of coordinates used to correct the color of the projection image is determined from the correction data represented by two-dimensional coordinates used to correct the color of the projection image. The color of the projected image is corrected using the correction data included in the range, The image processing method according to claim 1, including the method described in claim 1.

3. The image processing method according to claim 1, wherein the projection surface includes markers corresponding to each of the plurality of feature points.

4. The size of the projected image is smaller than the maximum image size that the optical device can project onto the projection surface. Prior to acquiring the aforementioned image, a marker image representing the marker is projected from the optical device between the contour of the largest image and the contour of the projected image. The image processing method according to claim 3, further comprising:

5. The size of the projected image is smaller than the size of the image indicated by the image signal. Prior to acquiring the aforementioned captured image, a marker image representing the marker is projected from the optical device between the contour of the image when the image is projected at the magnitude indicated by the image signal and the contour of the projected image. The image processing method according to claim 3, further comprising:

6. The image processing method according to claim 4 or claim 5, wherein the projected image and the marker image do not overlap.

7. The image processing method according to any one of claims 1 to 5, further comprising projecting an image of a color correction pattern from the optical device onto an area outside the projected image on the projection surface.

8. Including a processing device, The aforementioned processing apparatus is By imaging the projection surface, an image is obtained that includes a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed. The process involves correcting the color or brightness of the projected image based on the captured image, When correcting the color or brightness of a projected image based on the captured image, if a change in the position of the area where the projected image is displayed is detected based on the captured image, and the amount of the change in position is less than a predetermined threshold, the color or brightness of the projected image is corrected using correction data corresponding to the overlapping area between the area before the position change and the area after the position change, and no color or brightness correction is performed on pixels outside the overlapping area. Image processing system.

9. On the computer, By imaging the projection surface, an image is obtained that includes a projection image projected onto the projection surface from an optical device based on an image signal, and a plurality of feature points located in an area outside the area on the projection surface where the projection image is displayed. The system corrects the color or brightness of the projected image based on the captured image, and performs the following actions: When correcting the color or brightness of a projected image based on the captured image, if a change in the position of the area where the projected image is displayed is detected based on the captured image, and the amount of the change in position is less than a predetermined threshold, the color or brightness of the projected image is corrected using correction data corresponding to the overlapping area between the area before the position change and the area after the position change, and no color or brightness correction is performed on pixels outside the overlapping area. program.

Citation Information

Patent Citations

  • Device for detecting projection image area

    JP2011048773A

  • Image processing apparatus, image projection system, and program

    JP2014060611A

  • Projection system, image processing system, projection method and program

    JP2015056834A

  • Control device, image projection system, program, and correction method

    JP2016212357A

  • Projector and method for controlling projector

    JP2020005095A