Image processing method, program, and projector

Abstract

To provide an image processing method, a program, and a projector making it possible to easily select feature points.SOLUTION: An image processing method includes: acquiring a picked-up image obtained by picking up an image of a projection surface Sp on which a projection image is projected; executing image processing on the picked-up image to detect a first feature point and a second feature point; and projecting, on the projection surface Sp, a code image C including information indicating the first feature point and a code image C including information indicating the second feature point.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an image processing method, a program, and a projector.

Background Art

[0002] When the installation position of a projector that projects a projection image onto a projection surface moves, the position of the projection image on the projection surface will deviate from the initial position. For this reason, as shown in Patent Document 1, a projector that can return the position of the projection image to the initial position even when the installation position moves has been proposed. The projector described in Patent Document 1, as an initial setting process, images an outdoor scene including the projection surface with an imaging device, and extracts a plurality of first feature points from the captured image. Further, this projector also images the outdoor scene again after the installation position has moved, and extracts a plurality of second feature points. Then, using the projective transformation parameters for performing coordinate transformation from the first feature points to the corresponding second feature points, the position of the projection image is returned to the initial position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The projector described in Patent Document 1 extracts feature points using a predetermined algorithm, but the plurality of extracted feature points may include feature points that are inappropriate as reference points for the projection image. For this reason, it is desirable to enable a user to select appropriate feature points as reference points from among the plurality of feature points extracted by the projector. However, Patent Document 1 does not describe selecting appropriate feature points from among the extracted feature points. For this reason, a technique that enables a user to easily select feature points is desired.

Means for Solving the Problem

[0005] The image processing method includes acquiring a captured image obtained by capturing a projection surface onto which a projection image is projected, detecting a first feature point and a second feature point by performing image processing on the captured image, and projecting a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface.

[0006] The program causes a computer to acquire a captured image obtained by capturing a projection surface onto which a projection image is projected, detect a first feature point and a second feature point by performing image processing on the captured image, and project a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface.

[0007] The projector includes an optical device, and a processing device that acquires a captured image obtained by capturing a projection surface onto which a projection image is projected by the optical device, detects a first feature point and a second feature point by performing image processing on the captured image, and projects a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface using the optical device.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Embodiments for Carrying Out the Invention

[0009] 1. Embodiment Hereinafter, the image projection system 100 of the present embodiment will be described with reference to the drawings. FIG. 1 is a perspective view showing the schematic configuration of the image projection system 100. The image projection system 100 is an example of an image processing system.

[0010] As shown in FIG. 1, the image projection system 100 includes a projector 1 that projects a projection image Ep onto a projection plane Sp, and a terminal device 2 used by a user of the projector 1. The projector 1 of the present embodiment has a projection maintenance function that maintains the position and shape of the projection image Ep on the projection plane Sp even when the position and orientation of the projector 1 move slightly. The terminal device 2 is used by the user when utilizing the projection maintenance function. Details of the projection maintenance function will be described later. On the projection plane Sp, if the maximum range within which the installed projector 1 can project an image is defined as a projectionable range Aa, the projector projects the projection image Ep, which is a significant image, within a projection range Ap that is a partial range within the projectionable range Aa.

[0011] FIG. 2 is a block diagram showing the internal configuration of the projector 1, and FIG. 3 is a schematic diagram showing the schematic configuration of the image projection unit 15 included in the projector 1.

[0012] As shown in FIG. 2, the projector 1 includes a control unit 10, a storage unit 11, a communication unit 12, an image input unit 13, an image correction unit 14, an image projection unit 15, an operation unit 16, and an imaging unit 17. The projector 1 projects the projection image Ep onto the projection plane Sp from the image projection unit 15 based on the image data input to the image input unit 13.

[0013] The control unit 10 includes one or more processors such as a CPU (Central Processing Unit). The control unit 10 controls the operation of the projector 1 by operating according to the control program 11a stored in the storage unit 11. The control unit 10 is an example of a processing device and corresponds to a computer.

[0014] The memory unit 11 includes memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The RAM is used for temporary storage of various data, etc., and the ROM stores a control program 11a for controlling the operation of the projector 1, control data, and various image data, etc. The control program 11a is an example of a program. The control program 11a causes the control unit 10, which is a computer, to execute various operations described later.

[0015] The communication unit 12 is connected to a network NW such as a LAN (Local Area Network). The communication unit 12 includes an interface circuit for performing communication via the network NW, and communicates with other devices connected to the network NW. In the network NW of the present embodiment, in addition to the projector 1, a terminal device 2 is connected. The communication unit 12 can perform input / output of information with the terminal device 2 via the network NW. The connection between the communication unit 12 and the network NW may be a wired connection or a wireless connection.

[0016] The image input unit 13 includes a plurality of input terminals (not shown), and an external image supply device 3 such as a computer or an image playback device is connected to each input terminal. The image input unit 13 receives the supply of image data from the image supply device 3, and performs processing such as converting the data format and converting the resolution on the supplied image data as necessary, and outputs the processed image data to the image correction unit 14. Note that the image input unit 13 may receive the supply of image data from an external device via the network NW and the communication unit 12.

[0017] Based on the control of the control unit 10, the image correction unit 14 performs various correction processes on the image data input from the image input unit 13, and outputs the processed image data to the light valve driving unit 24 (see FIG. 3) of the image projection unit 15. For example, the image correction unit 14 performs processes such as adjusting the image quality, correcting trapezoidal distortion, and superimposing an OSD (On Screen Display) image such as a message or menu on the projected image on the image data. Further, the image correction unit 14 of the present embodiment executes a correction process by a projection maintenance function, that is, a correction process for maintaining the position and shape of the projected image Ep on the projection plane Sp. The image correction unit 14 may be configured by one or a plurality of integrated circuits. The image correction unit 14 may be included in the control unit 10. Hereinafter, the image data input from the image input unit 13 to the image correction unit 14 is also referred to as first image data D1, and the corrected image data output from the image correction unit 14 to the image projection unit 15 is also referred to as second image data D2.

[0018] As shown in FIG. 3, the image projection unit 15 includes a light source 21, three liquid crystal light valves 22 as a light modulation device, a projection optical system 23, a light valve driving unit 24, and the like. The image projection unit 15 modulates the light emitted from the light source 21 with the liquid crystal light valve 22 to form image light, and projects this image light from the projection optical system 23 including a lens onto the projection plane Sp to display the projection image Ep. The liquid crystal light valve 22 includes a liquid crystal light valve 22R for red light, a liquid crystal light valve 22G for green light, and a liquid crystal light valve 22B for blue light. The image projection unit 15 is an example of an optical device.

[0019] The light source 21 includes a solid light source such as a semiconductor laser or a light emitting diode, or a discharge type light source lamp such as an ultra-high pressure mercury lamp or a metal halide lamp. The light emitted from the light source 21 is converted into light with a substantially uniform luminance distribution by an integrator optical system (not shown). Then, after being separated into each color light component of red, green, and blue, which are the three primary colors of light, by a color separation optical system (not shown), they are incident on the liquid crystal light valves 22R, 22G, and 22B, respectively.

[0020] Each of the liquid crystal light valves 22R, 22G, and 22B is configured with a transmissive liquid crystal panel in which liquid crystal is sealed between a pair of transparent substrates. A rectangular pixel area 22i consisting of a plurality of pixels arranged in a matrix is formed on each liquid crystal panel, and a drive voltage can be selectively applied to the liquid crystal elements that make up each pixel. The pixel area 22i corresponds to the projectable range Aa.

[0021] The light valve driver 24 receives second image data D2 from the image corrector 14. The second image data D2 input from the image corrector 14 to the light valve driver 24 is image data representing the brightness of each pixel in the pixel region 22i. The light valve driver 24 forms an image in the pixel region 22i of the liquid crystal light valve 22 based on the second image data D2. Specifically, the light valve driver 24 applies a drive voltage corresponding to the second image data D2 input from the image corrector 14 to each pixel in the pixel region 22i, thereby setting the light transmittance of each pixel to the second image data D2. The light emitted from the light source 21 is modulated for each pixel as it passes through the pixel region 22i of the liquid crystal light valves 22R, 22G, and 22B, and image light corresponding to the second image data D2 is formed for each color light. The formed image light for each color is combined for each pixel by a color combining optical system (not shown) to form image light representing a color image, which is then enlarged and projected onto the projection surface Sp by the projection optical system 23.

[0022] In the pixel region 22i of the liquid crystal light valve 22, the projected image Ep is formed within the effective region 22e inside the pixel region 22i. In other words, the effective region 22e is the region corresponding to the projection range Ap. The region outside the effective region 22e is set to black, i.e., the minimum light transmittance. In other words, in the second image data D2 supplied from the image corrector 14 to the light valve driver 24, the region outside the effective region 22e is set to a value corresponding to black. Therefore, on the projection surface Sp, the projected image Ep is displayed within the projection range Ap, and almost no light is irradiated outside the projection range Ap.

[0023] Hereinafter, the image based on the second image data D2 input from the image correction unit 14 to the image projection unit 15 is also referred to as a panel image. That is, the panel image is an image formed over the entire pixel region 22i including the inside and outside of the effective region 22e.

[0024] Returning to FIG. 2, the operation unit 16 includes a plurality of operation keys (not shown) for the user to give various instructions to the projector 1. When the user operates the operation keys of the operation unit 16, the operation unit 16 outputs an operation signal corresponding to the user's operation content to the control unit 10. Note that a configuration may be adopted in which a remote controller (not shown) capable of remote operation is used as the operation unit 16. In this case, the remote controller transmits an infrared operation signal corresponding to the user's operation content, and a remote controller signal receiving unit (not shown) receives this and transmits it to the control unit 10. The operation unit 16 can also be referred to as an operation device or an input device.

[0025] The imaging unit 17 images the front, which is the direction in which an image is projected from the image projection unit 15, and generates image data representing the captured image. Hereinafter, the image data representing the captured image is also simply referred to as a captured image. The imaging unit 17 images an imaging range including at least the projectable range Aa. The imaging unit 17 outputs the generated captured image to the control unit 10. The imaging unit 17 is an imaging device including an imaging element 18 such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and an imaging lens (not shown). The imaging element 18 is provided with a light receiving region including a plurality of pixels, and the captured image is generated based on the light reception amount of each pixel.

[0026] The storage unit 11 of the present embodiment stores a WEB page W0 that is referred to at the time of initial setting of the projection maintenance function. And the control unit 10 also functions as a WEB server and provides this WEB page W0 to the terminal device 2 etc. connected to the network NW. Also, the control unit 10 updates the content of the WEB page W0 based on the user's operation etc.

[0027] In addition, the storage unit 11 stores image data representing a plurality of code images C. The code image C is, for example, a QR code (registered trademark), which is a matrix-type two-dimensional code. This code image C is used at the time of initial setting of the projection maintenance function. The code image C contains information representing the URL (Uniform Resource Locator) of the above-mentioned WEB page W0, and different parameters are added to this URL for each code image C. Specifically, the parameter includes a number represented by one or more natural numbers. When accessing the WEB page W0 using the code image C, the control unit 10 can acquire the number added to the URL.

[0028] FIG. 4 is a block diagram showing a schematic configuration of the terminal device 2. The terminal device 2 is a portable information processing device used by the user of the projector 1, and examples include a smartphone or a tablet-type terminal device.

[0029] The terminal device 2 includes a terminal control unit 30, a terminal storage unit 31, a terminal communication unit 32, a terminal operation unit 33, a terminal display unit 34, and a terminal imaging unit 35. The terminal control unit 30 includes a processor such as a CPU, and operates according to an OS (Operating System) and application programs (not shown) stored in the terminal storage unit 31 to control the operation of the terminal device 2.

[0030] The terminal storage unit 31 is composed of a storage device such as a memory, and stores the above-mentioned OS, application programs, and various setting data.

[0031] The terminal communication unit 32 includes an interface circuit for performing communication via the network NW, and can perform communication with the projector 1 via the network NW. The connection between the terminal communication unit 32 and the network NW is usually a wireless connection, but a wired connection may also be used.

[0032] The terminal operation unit 33 receives the input operations of the user. The terminal operation unit 33 outputs operation information corresponding to the received input operations to the terminal control unit 30. The terminal display unit 34 includes a display device such as a liquid crystal display or an organic EL display, and displays various information based on the control of the terminal control unit 30. Note that the terminal operation unit 33 may be integrally configured with the terminal display unit 34 like a touch panel.

[0033] The terminal imaging unit 35 is an imaging device including an imaging element (not shown) such as a CCD sensor or a CMOS sensor, and an imaging lens (not shown) or the like. The terminal imaging unit 35 images an imaging range according to the orientation and posture of the terminal device 2, and generates an imaging image. The terminal imaging unit 35 outputs the generated imaging image to the terminal control unit 30.

[0034] The terminal storage unit 31 stores various application programs (not shown). The terminal control unit 30 can execute various operations according to the activated application programs. The application programs stored in the terminal storage unit 31 include a reading program for reading the code image C imaged by the terminal imaging unit 35, a WEB browser for displaying the WEB page W0, and the like. These application programs may be general-purpose programs provided by the manufacturing company or sales company of the terminal device 2, or the OS development company, etc., or may be dedicated programs provided by the manufacturing company or sales company of the projector 1, or other third parties.

[0035] Next, the image processing method of the projector 1 will be described. FIG. 5 is a flowchart for explaining the image processing method of the projector 1, and shows the operation of the projector 1 when performing the initial setting of the projection maintenance function. FIGS. 6 to 9 are diagrams showing the state of the projection plane Sp at the time of the initial setting of the projection maintenance function. After the user activates the projector 1 and performs an operation for performing the initial setting of the projection maintenance function, the control unit 10 operates according to the flow shown in FIG. 5.

[0036] The position of the projector 1 is roughly adjusted in advance by the user so that the projected image Ep is projected in a desired projection range Ap on the projection surface Sp. Specifically, the user installs the projector 1 so that the projectable range Aa of the projector 1 covers the desired projection range Ap. In this embodiment, the desired projection range Ap is set inside a rectangular frame F provided on the projection surface Sp.

[0037] First, in step S101, the control unit 10 causes the image projection unit 15 to project a pattern image Pp, in which a plurality of dot-shaped marks M1 are arranged, onto the projection surface Sp within the projectable range Aa. Image data representing the pattern image Pp is stored in advance in the storage unit 11, and the control unit 10 reads this image data from the storage unit 11 and outputs it to the image correction unit 14. When the image correction unit 14 outputs this image data to the image projection unit 15 without correction, the pattern image Pp is projected from the image projection unit 15 onto the projection surface Sp.

[0038] In step S102, the control unit 10 causes the imaging unit 17 to capture an image of the imaging range including the projectable range Aa. That is, the control unit 10 causes the imaging unit 17 to capture an image of the projection surface Sp on which the pattern image Pp is projected, and acquires the captured image from the imaging unit 17.

[0039] In step S103, the control unit 10 detects, in the captured image, a plurality of marks M1 in the captured pattern image Pp. Then, the control unit 10 generates first correspondence information representing the correspondence relationship between coordinates on the panel image and coordinates on the captured image based on the positions of the plurality of marks M1 in the captured image and the positions of the plurality of marks M1 in the panel image. The positions of the plurality of marks M1 in the panel image are known information.

[0040] In step S104, as shown in FIG. 7, the control unit 10 causes the image projection unit 15 to project the adjustment image Pc onto the projection plane Sp. A reference image Pb of a predetermined size is arranged at the center of the adjustment image Pc. The reference image Pb is an image for determining the projection range Ap where the projection image Ep is displayed. The shape of the reference image Pb is, for example, a rectangular shape on the pixel region 22i of the liquid crystal light valve 22, that is, on the panel image. The image data representing the adjustment image Pc is stored in the storage unit 11 in advance, and the control unit 10 reads out this image data from the storage unit 11 and outputs it to the image correction unit 14. When the image correction unit 14 outputs this image data to the image projection unit 15 without correcting it, the adjustment image Pc including the reference image Pb is projected from the image projection unit 15 onto the projection plane Sp. Cross-shaped marks M2 are attached to the four corners of the reference image Pb, for example.

[0041] In step S105, the control unit 10 determines the projection range Ap based on the user's operation. Specifically, the control unit 10 determines the positions of the four corners of the reference image Pb on the panel image based on the user's operation. Since the position and shape of the quadrangular reference image Pb are determined by the positions of the corners of the reference image Pb, determining the positions of the corners of the reference image Pb is synonymous with determining the position and shape of the reference image Pb. The user can individually specify the positions of the four corners of the reference image Pb, that is, the positions of the marks M2, by operating the operation unit 16. The control unit 10 deforms the reference image Pb by moving the mark M2 specified by the user in the direction specified by the user. The user adjusts the positions of the respective marks M2 so that the position and shape of the reference image Pb projected on the projection surface Sp match the position and shape of the desired projection range Ap. Then, on the projection surface Sp, the range of the quadrangle with the positions of the four adjusted marks M2 as vertices is determined as the projection range Ap. For example, as shown in FIG. 8, the user adjusts the corners of the reference image Pb, that is, the corners of the projection range Ap, so that they are parallel to the respective sides of the frame F inside the rectangular frame F on the projection surface Sp. In FIG. 8, the projection range Ap is adjusted to be smaller than the inner edge of the frame F, but it may be adjusted to match the inner edge of the frame F, for example.

[0042] In step S106, the control unit 10 determines the quadrangular region with the positions of the four adjusted marks M2 on the panel image as vertices as the valid region 22e. Then, the control unit 10 generates a first conversion formula for converting the first image data D1 so that the position and shape of the projection image Ep on the panel image match the position and shape of the valid region 22e.

[0043] In step S107, with the image not being projected from the image projection unit 15, the control unit 10 causes the imaging unit 17 to image the projection surface Sp. Then, the control unit 10 acquires the captured image of the projection surface Sp from the imaging unit 17. Note that the image projection unit 15 may project a predetermined image such as an image in which the entire projection possible range Aa has a substantially uniform brightness.

[0044] In step S108, the control unit 10 detects a plurality of feature points by performing image processing on the acquired captured image. As an algorithm for detecting feature points, for example, AKAZE can be used, and feature points are detected based on feature amounts obtained by quantifying features. Here, the feature points to be detected are points that serve as a reference when maintaining the position and shape of the projected image Ep by the projection maintenance function. Therefore, the feature points are points existing on the projection plane Sp and need to be points whose positions do not move during the period of using the projector 1. And it is desirable that each feature point exists in the vicinity of the four corners of the projection range Ap. In the present embodiment, the four corners of the frame F are detected as feature points, and these four feature points are used as reference points for the projection maintenance function. Also, in this step, feature points that are inappropriate as reference points for the projection maintenance function, such as feature points existing at positions separated from the projection plane Sp and feature points caused by foreign objects or shadows, may also be detected. Instead of the frame F, a plurality of markers may be arranged on the projection plane Sp so that these markers are detected as feature points.

[0045] In step S109, the control unit 10 assigns a unique number consisting of one or more natural numbers to each of the detected feature points. Then, for each feature point, the control unit 10 stores the position in the captured image and the feature amount in the storage unit 11 in association with the assigned number. The number assigned to the feature point is an example of information indicating the feature point. Note that the control unit 10 can determine which corner of the four corners of the projection range Ap each feature point is located near based on the position of each feature point in the captured image.

[0046] In step S110, as shown in FIG. 9, the control unit 10 causes the image projection unit 15 to project a selected image Ps1 onto the projection plane Sp. The selected image Ps1 is an image in which a code image C is arranged at a position on the projection plane Sp corresponding to the position of the feature point in the captured image. That is, the control unit 10 causes the code image C to be projected at the position of the feature point on the projection plane Sp. In other words, the control unit 10 causes a plurality of code images C to be projected onto the projection plane Sp.

[0047] Specifically, the control unit 10 converts the position of the feature point in the captured image to the position in the panel image based on the first correspondence information representing the correspondence between the coordinates on the panel image and the coordinates on the captured image. Then, the control unit 10 generates the image data of the selection image Ps1 in which the code image C is arranged at this position and outputs it to the image correction unit 14. At this time, the control unit 10 arranges the code image C with the parameter of the same number as the number assigned to this feature point added at the position of each feature point. That is, each of the plurality of code images C projected onto the projection plane Sp includes the number assigned to the corresponding feature point. Also, the control unit 10 arranges the code image C so that the center of the code image C coincides with the position of the feature point. When the image correction unit 14 outputs the image data of the selection image Ps1 generated by the control unit 10 to the image projection unit 15 without correction, the selection image Ps1 including the plurality of code images C is projected from the image projection unit 15 onto the projection plane Sp. Note that the selection image Ps1 may include an image indicating the projection range Ap.

[0048] Also, as shown in FIG. 9, when a plurality of feature points are detected near one corner of the projection range Ap, the user needs to select an appropriate feature point as the reference point for the projection maintenance function from among the plurality of feature points. That is, the user needs to select the feature points corresponding to the four corners of the frame F. Specifically, the user selects the feature point closest to the corner at each of the four corners of the frame F. In the present embodiment, the user can specify the feature point to be selected by reading the code image C with the terminal device 2. The selection of the feature point will be described later.

[0049] In step S111, the control unit 10 acquires the number of the feature point selected by the user from the terminal device 2. Then, the control unit 10 specifies the position of the selected feature point in the captured image based on the acquired number and stores the information representing this position in the storage unit 11.

[0050] FIG. 10 and FIG. 11 are diagrams showing the terminal display unit 34 of the terminal device 2, and show the display content displayed on the terminal display unit 34 when the user selects a feature point. As shown in FIG. 10, when selecting a feature point, the user activates a reading program on the terminal device 2. Then, among the plurality of code images C projected on the projection plane Sp, the user captures the code image C of the feature point to be selected with the terminal imaging unit 35. The terminal control unit 30 analyzes the captured code image C by the reading program and reads the information included in the code image C, that is, the URL with parameters added. Then, the terminal control unit 30 activates a WEB browser and accesses the WEB page W0 represented by the read URL. At this time, the parameter added to the URL, that is, the number of the selected feature point, is acquired by the control unit 10 of the projector 1.

[0051] As shown in FIG. 11, the WEB page W0 includes a first display unit W1, a second display unit W2, and a button image W3 for canceling an operation. The first display unit W1 includes four images respectively corresponding to the four corners of the projection range Ap. Among these, the image corresponding to the corner where the feature point has been selected is displayed in a different color from the other images, making it possible to determine which corner's vicinity the feature point has been selected. In addition, the second display unit W2 shows a message indicating how many more feature points should be selected according to the current selection status. In this way, the user can know the progress of the process of selecting feature points based on the display content of the first display unit W1 and the second display unit W2.

[0052] For example, the user first captures, using the terminal device 2, the code image C closest to the upper left corner of the frame F. Then, based on the captured code image C, the terminal device 2 accesses the web page W0. The control unit 10 of the projector 1 can determine, based on the number obtained from the terminal device 2, that the selected feature point is the upper left feature point. Then, according to the user's selection result, the control unit 10 updates the first display section W1 and the second display section W2 of the web page W0. That is, the control unit 10 displays on the web page W0 that the upper left feature point has been selected and that there are three remaining feature points to be selected. As a result, as shown in FIG. 11, the updated web page W0 is displayed on the terminal display section 34. In this way, when the control unit 10 of the projector 1 obtains the number of the feature point selected by the user from the terminal device 2, it transmits, in the form of the web page W0, information indicating the progress of the process of selecting the feature point to the terminal device 2.

[0053] For the remaining three corners, the user also specifies the feature points to be selected by reading the code image C with the terminal device 2. Then, when all four feature points are selected by the user and the numbers of the four feature points are obtained from the terminal device 2, the control unit 10 ends the flow.

[0054] With the above, the initial setting of the projection maintenance function is completed. Thereafter, when projecting the projection image Ep from the image projection unit 15, the image correction unit 14 corrects the first image data D1 input from the image input unit 13 using the first conversion formula. As a result, on the projection surface Sp, the projection image Ep is projected in a state where the position and shape are corrected according to the projection range Ap determined in step S105. Note that the correction process performed on the first image data D1 using the first conversion formula is also referred to as the first correction process. And thereafter, for example, when the position of the projector 1 has moved or is assumed to have moved, a restoration process for restoring the position and shape of the projection image Ep is executed.

[0055] FIG. 12 is a flowchart for explaining the image processing method of the projector 1, and shows the operation of the projector 1 during the restoration process. In this restoration process, the control unit 10 generates a second conversion formula for correcting the position and shape of the projection image Ep on the projection plane Sp using the four feature points selected by the user.

[0056] As shown in FIG. 12, steps S201 to S203 are the same steps as steps S101 to S103 described above. That is, in step S201, the control unit 10 causes the image projection unit 15 to project a pattern image Pp in which a plurality of dot-shaped marks M1 are arranged within the projection range Aa of the projection plane Sp. In step S202, the control unit 10 causes the imaging unit 17 to image the imaging range including the projection range Aa. In step S203, the control unit 10 detects a plurality of marks M1 in the captured pattern image Pp in the captured image. Then, the control unit 10 generates second correspondence information representing the correspondence relationship between the coordinates on the panel image and the coordinates on the captured image based on the positions of the plurality of marks M1 in the captured image and the positions of the plurality of marks M1 in the panel image. When the projector 1 moves after the initial setting, the second correspondence information is different from the first correspondence information generated in step S103.

[0057] The subsequent step S204 is the same step as step S107 described above. That is, the control unit 10 causes the imaging unit 17 to image the projection plane Sp in a state where no image is being projected from the image projection unit 15. Then, the control unit 10 acquires the captured image of the projection plane Sp from the imaging unit 17.

[0058] In step S205, the control unit 10 detects a plurality of feature points by executing image processing on the newly acquired captured image, similar to step S108 described above. Then, based on the feature amounts of the respective feature points, the control unit 10 extracts four feature points selected by the user at the time of initial setting, that is, four feature points corresponding to the four corners of the frame F. Specifically, the control unit 10 extracts four feature points having feature amounts similar to those of the four feature points selected by the user.

[0059] In step S206, the control unit 10 generates a second conversion formula for correcting the position and shape of the projected image Ep on the projection plane Sp and ends the flow. Specifically, the control unit 10 generates the second conversion formula based on the first correspondence information generated in step S103, the second correspondence information generated in step S203, the information representing the positions of the four feature points in the captured image acquired in step S111, and the information representing the positions of the four feature points extracted in step S205. The second conversion formula is an example of correction information.

[0060] Thereafter, when projecting the projected image Ep, the image correction unit 14 executes the above-described first correction process on the input first image data D1 and also executes a conversion using the generated second conversion formula. As a result, the position and shape of the projected image Ep are corrected to the position and shape of the projection range Ap determined in step S105. Note that the correction process performed using the second conversion formula generated in step S206 is also referred to as the second correction process. Each time the control unit 10 executes the restoration process, it regenerates the second conversion formula. Then, when projecting the projected image Ep, the image correction unit 14 is made to execute the first correction process and the second correction process using the newly generated second conversion formula. Thereby, the projector 1 maintains the position and shape of the projected image Ep.

[0061] The restoration process described above may be started by a user instruction when the projector 1 moves. Alternatively, the projector 1 may be provided with a detection means for detecting movement, and when the detection means detects movement of the projector 1, the control unit 10 may start the restoration process.

[0062] Furthermore, when the projector 1 is used for a long period of time, the restoration process may be set to be executed periodically regardless of whether or not the projector 1 is moving. In this case, it is desirable that the control unit 10 start up the projector 1 and execute the restoration process when the projector 1 is not being used, such as at night. In this way, when the restoration process is executed periodically, it is possible to prevent the position and shape of the projected image Ep from gradually changing over time.

[0063] As described above, the projector 1, control program 11a, and image processing method of this embodiment can provide the following effects.

[0064] According to this embodiment, multiple code images C including the numbers of the feature points are projected onto the projection surface Sp, so the user can specify the feature point to be selected by reading any of the code images C with the terminal device 2. As a result, it becomes easy to select an appropriate feature point from among multiple feature points.

[0065] According to this embodiment, the projector 1 generates a second conversion formula for correcting the position and shape of the projected image Ep using the feature points of the numbers acquired from the terminal device 2. Therefore, the position and shape of the projected image Ep can be corrected using appropriate feature points selected by the user.

[0066] According to this embodiment, the code image C is projected at the position of the feature point on the projection surface Sp, so the user can intuitively determine the feature point to be selected.

[0067] According to the present embodiment, when the projector 1 acquires the numbers of the feature points from the terminal device 2, the projector 1 transmits information indicating the progress of the process of selecting the feature points to the terminal device 2. Therefore, the convenience of the user who is selecting the feature points is improved.

[0068] In the present embodiment, among the plurality of feature points detected in step S108, one of the feature points selected by the user on the terminal device 2 corresponds to the first feature point, and another feature point different from the first feature point corresponds to the second feature point. Also, in step S109, the unique number assigned to the first feature point corresponds to the information indicating the first feature point, and the unique number assigned to the second feature point corresponds to the information indicating the second feature point. Further, the code image C including the number assigned to the first feature point corresponds to the first code image, and the code image C including the number assigned to the second feature point corresponds to the second code image. Therefore, projecting the plurality of code images C onto the projection plane Sp in step S110 corresponds to projecting the first code image and the second code image onto the projection plane Sp, and includes projecting the first code image at the position of the first feature point on the projection plane Sp and projecting the second code image at the position of the second feature point on the projection plane Sp.

[0069] 2. Modification Example The above-described embodiment is based on the above-described configuration, but within the scope not departing from the gist of the present disclosure, partial changes or omissions of the configuration, etc. can also be made. Also, the present embodiment and the modification examples described below can be implemented in combination with each other as long as there is no technical contradiction. Hereinafter, the modification examples will be described.

[0070] In the above-described embodiment, the code image C projected onto the projection plane Sp in step S110 is arranged such that its center coincides with the position of the feature point on the projection plane Sp, but is not limited to this aspect. For example, as shown in FIG. 13, a mark M3 smaller than the code image C is arranged at the position of the feature point on the projection plane Sp, and the code image C may be arranged in the vicinity of the mark M3, that is, in the vicinity of the feature point. This aspect also corresponds to projecting the code image C at the position of the feature point. According to this aspect, the position of the feature point can be more clearly indicated. Further, in this aspect, it is desirable that the code image C be arranged at a position shifted in the direction toward the approximate center of the projection range Ap from the position of the feature point. According to this aspect, it is possible to suppress the code image C to be selected, that is, the code image C closest to the corner of the frame F from overlapping the frame F. Therefore, it is possible to suppress the occurrence of a reading failure when the terminal device 2 reads the code image C. Further, it is possible to suppress the code image C from being incomplete and not fitting within the projectable range Aa.

[0071] Also, for example, as shown in FIG. 14, a mark M3 is arranged at the position of the feature point on the projection plane Sp, and an identification image Pi representing information capable of identifying the feature point may be displayed in the vicinity thereof. The identification image Pi is an image that corresponds one-to-one with the code image C, and is, for example, an image representing the number of the feature point assigned in step S109. The code image C is arranged below the projection range Ap in association with the identification image Pi. Specifically, the code image C is arranged at the center of the projection range Ap, that is, at a position lower than the center of the projection image Ep. That is, in this aspect, the control unit 10 generates image data of a selection image Ps2 in step S110 by arranging the identification image Pi in the vicinity of the feature point and arranging a plurality of code images C below the projection range Ap. Then, the control unit 10 causes the image projection unit 15 to project the selection image Ps2 onto the projection plane Sp. According to this aspect, even when the projection range Ap is large and the upper feature point is at a position considerably higher than the user's height, the user can easily select the feature point by the code image C at a low position. Further, in this aspect, it is desirable that the identification image Pi be arranged at a position shifted in a direction toward the approximate center of the projection range Ap from the position of the feature point. According to this aspect, since the identification image Pi is suppressed from overlapping with the frame F, the visibility of the identification image Pi is improved. Also, it is possible to suppress the identification image Pi from being missing without fitting within the projectable range Aa.

[0072] Note that, among the plurality of displayed identification images Pi, the identification image Pi that corresponds one-to-one with the first code image described above corresponds to the first identification image, and the identification image Pi that corresponds one-to-one with the second code image described above corresponds to the second identification image. That is, in this modification, projecting a plurality of code images C onto the projection plane Sp corresponds to projecting the first code image and the second code image onto the projection plane Sp, and projecting the first code image and the second code image at a position lower than the center of the projection image Ep, projecting the first identification image corresponding one-to-one with the first code image at the position of the first feature point on the projection plane Sp, and projecting the second identification image corresponding one-to-one with the second code image at the position of the second feature point on the projection plane Sp.

[0073] Also, a mode of projecting a selection image Ps1 in which a code image C is arranged at the position of a feature point on the projection surface Sp and a mode of projecting a selection image Ps2 in which a plurality of code images C are arranged below the projection range Ap may be made switchable according to a user's operation. For example, as shown in FIGS. 15 and 16, the control unit 10 may project a code image Ca for switching the display mode onto the projection surface Sp. The code image Ca is, for example, a QR code, similar to the code image C. The information represented by the code image Ca is the URL of the web page W0, and a parameter indicating switching of the display mode is added to this URL. Then, when access to the web page W0 is made from the terminal device 2 using the code image Ca, that is, when a parameter indicating switching of the display mode is acquired from the terminal device 2, the control unit 10 switches the selection images Ps1 and Ps2 to be projected. Specifically, in step S110, the control unit 10 generates two types of selection images Ps1 and Ps2 including the code image Ca and causes either one of them to be projected from the image projection unit 15. Then, when the control unit 10 acquires a parameter indicating switching of the display mode from the terminal device 2, the control unit 10 switches the image to be projected between the selection image Ps1 and the selection image Ps2. The code image Ca corresponds to a third code image.

[0074] In this modification, the mode of projecting the selection image Ps1 corresponds to the first mode, and the mode of projecting the selection image Ps2 corresponds to the second mode. That is, when the projector 1 projects a plurality of code images C including the first code image and the second code image onto the projection surface Sp, the first code image is projected at the position of the first feature point on the projection surface Sp, and the second code image is projected at the position of the second feature point on the projection surface Sp in the first mode, and a code image Ca for switching between the first mode and the second mode in which the first code image and the second code image are projected at a position lower than the center of the projected image Ep is projected onto the projection surface Sp. According to this configuration, the user can cause the plurality of code images C to be displayed in a desired mode by reading the code image Ca with the terminal device 2.

[0075] Further, when a predetermined operation is performed on the operation unit 16 such as a remote controller, the control unit 10 may switch the selection images Ps1 and Ps2. Alternatively, as shown in FIG. 17, a button image W4 for switching the display mode may be arranged in the WEB page W0, and the selection images Ps1 and Ps2 may be switched when this button image W4 is operated.

[0076] In the above embodiment, although the mode in which the user selects four feature points existing in the vicinity of the four corners of the projection range Ap is shown, feature points at positions away from the corners of the projection range Ap may be selected. Also, the number of feature points to be selected by the user is not limited to four. For example, it may be three, or five or more. Also, any number of feature points may be selected by the user. In this case, a code image with a parameter indicating the end of the selection of the feature points may be arranged in the selection images Ps1 and Ps2, or a button image indicating the end of the selection of the feature points may be arranged in the WEB page W0.

[0077] In the above embodiment, a configuration in which a plurality of code images C are stored in the storage unit 11 in advance is shown. In this configuration, it is necessary to store a large number of code images C assuming that a large number of feature points can be detected. Therefore, instead of this configuration, the control unit 10 may generate the code image C according to the detection result of the feature points. In this case, the control unit 10 may generate the same number of code images C as the number of detected feature points. Also, for a predetermined number of code images C, they may be stored in the storage unit 11 in advance, and when more than the predetermined number of feature points are detected, the control unit 10 may generate the corresponding number of code images C.

[0078] In the above embodiment, the control unit 10 assigns a number consisting of one or more natural numbers as information indicating the feature point to each of the detected feature points. However, as long as it is information unique to each feature point, it does not have to be a number. For example, it may be a character, or it may be coordinates representing the position of the feature point in the captured image.

[0079] In the above embodiment, a configuration for projecting a projection image Ep based on image data input to the image input unit 13 from the outside is shown, but the configuration is not limited thereto. For example, a configuration in which an image based on the image data stored in the storage unit 11 is projected as the projection image Ep may be used. In this case, the control unit 10 outputs the image data stored in the storage unit 11 to the image correction unit 14 as the first image data D1.

[0080] In the above embodiment, as information indicating the progress of the process of selecting feature points, information indicating which corner's feature points have been selected and information indicating how many remaining feature points should be selected are displayed, but only one of them may be displayed. Further, information other than the above, such as information indicating which corner's feature points are unselected and information indicating the number of selected feature points, may be displayed.

[0081] In the above embodiment, the code images C and Ca are not limited to QR codes. For example, a two-dimensional code other than a QR code may be used, or a one-dimensional bar code may be used.

[0082] In the above embodiment, a transmissive liquid crystal light valve 22 is used as the optical modulation device, but a reflective optical modulation device such as a reflective liquid crystal light valve can also be used. Further, by controlling the emission direction of the incident light for each micromirror as a pixel, a digital micromirror device or the like that modulates the light emitted from the light source 21 can also be used. Further, the configuration is not limited to a configuration including a plurality of optical modulation devices for each color light, and a configuration in which a plurality of color lights are modulated in a time-division manner by one optical modulation device may be used.

[0083] 3. Summary of the Present Disclosure The summary of the present disclosure is appended below.

[0084] (Appendix 1) Obtaining a captured image that captures a projection plane onto which a projected image is projected, detecting a first feature point and a second feature point by performing image processing on the captured image, a first code image including information indicating the first feature point, and a second code image including information indicating the second feature point, and projecting the first code image and the second code image onto the projection plane. An image processing method including the above steps.

[0085] According to the configuration of Appendix 1, since a plurality of code images including information indicating feature points are projected onto the projection plane, the user can specify the feature points to be selected by reading any of the code images with a terminal device. As a result, it becomes easy to select appropriate feature points from among a plurality of feature points.

[0086] (Appendix 2) The image processing method according to Appendix 1, further including obtaining information indicating the first feature point from a terminal device, and generating correction information for correcting the position and shape of the projected image on the projection plane using the first feature point. An image processing method including the above steps.

[0087] According to the configuration of Appendix 2, correction information for correcting the position and shape of the projected image is generated using the feature points indicated by the information obtained from the terminal device. Therefore, the position and shape of the projected image can be corrected using the appropriate feature points selected by the user.

[0088] (Appendix 3) The image processing method according to Appendix 1 or 2, wherein projecting the first code image and the second code image onto the projection plane includes projecting the first code image at the position of the first feature point on the projection plane, and projecting the second code image at the position of the second feature point on the projection plane. An image processing method including the above steps.

[0089] According to the configuration of Appendix 3, since the code image is projected at the position of the feature point on the projection plane, the user can intuitively distinguish the feature points to be selected.

[0090] (Appendix 4) The image processing method according to Appendix 1 or 2, wherein projecting the first code image and the second code image onto the projection plane includes projecting the first code image and the second code image at a position lower than the center of the projected image, projecting a first identification image corresponding one-to-one to the first code image at the position of the first feature point on the projection plane, and projecting a second identification image corresponding one-to-one to the second code image at the position of the second feature point on the projection plane.

[0091] According to the configuration of Appendix 4, an identification image corresponding to the code image is displayed at the position of the feature point on the projection plane, and the code image is displayed below the center of the projected image. Therefore, even when the feature point is at a high position, the user can easily select the feature point by the code image at a low position.

[0092] (Appendix 5) The image processing method according to any one of Appendices 1 to 4, wherein when projecting the first code image and the second code image onto the projection plane, a third code image for switching between a first mode in which the first code image is projected at the position of the first feature point on the projection plane and the second code image is projected at the position of the second feature point on the projection plane, and a second mode in which the first code image and the second code image are projected at a position lower than the center of the projected image, is projected onto the projection plane.

[0093] According to the configuration of Appendix 5, a third code image for switching between a first mode in which the code image is displayed at the position of the feature point and a second mode in which the code image is displayed below the projected image is displayed on the projection plane. Therefore, the user can cause the first code image and the second code image to be displayed in a desired mode by reading the third code image with the terminal device.

[0094] (Appendix 6) An image processing method according to any one of Appendices 1 to 5, further including transmitting, to the terminal device, information indicating the progress of the process of selecting feature points when information indicating the first feature points is obtained from the terminal device.

[0095] According to the configuration of Appendix 6, when information indicating feature points is obtained from the terminal device, information indicating the progress of the process of selecting feature points is transmitted to the terminal device. Therefore, the convenience of the user who is selecting feature points is improved.

[0096] (Appendix 7) A program causing a computer to perform: acquiring a captured image obtained by capturing an image of a projection surface on which a projection image is projected; detecting a first feature point and a second feature point by performing image processing on the captured image; and projecting a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface.

[0097] According to the configuration of Appendix 7, since a plurality of code images including information indicating feature points are projected onto the projection surface, the user can specify the feature points to be selected by reading any one of the code images with the terminal device. As a result, it becomes easy to select appropriate feature points from among a plurality of feature points.

[0098] (Appendix 8) A projector including: an optical device; a processing device that performs: acquiring a captured image obtained by capturing an image of a projection surface on which a projection image is projected by the optical device; detecting a first feature point and a second feature point by performing image processing on the captured image; and projecting, using the optical device, a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface.

[0099] According to the configuration of Supplementary Note 8, since a plurality of code images including information indicating feature points are projected onto the projection surface, the user can specify the feature points to be selected by reading any of the code images with the terminal device. As a result, it becomes easy to select appropriate feature points from among the plurality of feature points.

Explanation of Signs

[0100] 1…Projector, 2…Terminal device, 3…Image supply device, 10…Control unit, 11…Storage unit, 11a…Control program, 12…Communication unit, 13…Image input unit, 14…Image correction unit, 15…Image projection unit, 16…Operation unit, 17…Imaging unit, 18…Image sensor, 21…Light source, 22, 22R, 22G, 22B…Liquid crystal light valve, 22e…Effective area, 22i…Pixel area, 23…Projection optical system, 24…Light valve drive unit, 30…Terminal control unit, 31…Terminal storage unit, 32…Terminal communication unit, 33…Terminal operation unit, 34…Terminal display unit, 35…Terminal imaging unit, 100…Image projection system, Aa…Projectable range, Ap…Projection range, C, Ca…Code image, D1…First image data, D2…Second image data, Ep…Projection image, F…Frame, M1…Mark, M2…Mark, M3…Mark, NW…Network, Pb…Reference image, Pc…Adjustment image, Pi…Identification image, Pp…Pattern image, Ps1, Ps2…Selected image, Sp…Projection surface, W0…WEB page, W1…First display unit, W2…Second display unit, W3…Button image, W4…Button image.

Claims

1. Obtaining a captured image obtained by capturing a projection surface onto which a projected image is projected; Detecting a first feature point and a second feature point by performing image processing on the captured image; Projecting a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface, An image processing method.

2. The image processing method according to claim 1, Obtaining information indicating the first feature point from a terminal device; Further including generating correction information for correcting the position and shape of the projected image on the projection surface using the first feature point, An image processing method.

3. The image processing method according to claim 1 or 2, Projecting the first code image and the second code image onto the projection surface includes: Projecting the first code image at the position of the first feature point on the projection surface; Projecting the second code image at the position of the second feature point on the projection surface, An image processing method.

4. The image processing method according to claim 1 or 2, Projecting the first code image and the second code image onto the projection surface includes: Projecting the first code image and the second code image at a position lower than the center of the projected image; Projecting a first identification image that corresponds one-to-one with the first code image at the position of the first feature point on the projection surface; Projecting a second identification image that corresponds one-to-one with the second code image at the position of the second feature point on the projection surface, An image processing method.

5. The image processing method according to claim 1 or 2, When projecting the first code image and the second code image onto the projection surface, A first mode in which the first code image is projected at the position of the first feature point on the projection surface and the second code image is projected at the position of the second feature point on the projection surface; A second mode in which the first code image and the second code image are projected at a position lower than the center of the projected image; Projecting a third code image for switching between the two onto the projection surface, An image processing method.

6. The image processing method according to claim 2, When information indicating the first feature point is acquired from the terminal device, further including transmitting information indicating the progress of the process of selecting the feature point to the terminal device. Image processing method. **Claim 7** Causing a computer to acquire a captured image obtained by imaging a projection surface onto which a projection image is projected; detect a first feature point and a second feature point by performing image processing on the captured image; cause a first code image including information indicating the first feature point and a second code image including information indicating the second feature point to be projected onto the projection surface. Program. **Claim 8** An optical device, acquiring a captured image obtained by imaging a projection surface onto which a projection image is projected by the optical device; detecting a first feature point and a second feature point by performing image processing on the captured image; projecting a first code image including information indicating the first feature point and a second code image including information indicating the second feature point onto the projection surface using the optical device; and a processing device that executes the above. Projector.

Images