Image processing system, image processing method, and program

Abstract

To provide an image processing system, an image processing method, and a program for allowing a user to easily select a feature point.SOLUTION: An image projection system 100 includes a projector 1 and a terminal device 2. The projector 1 includes a control unit that executes acquiring a first picked-up image obtained by picking up an image of a projection surface Sp, executing image processing on the first picked-up image to detect a plurality of feature points, and transmitting, to the terminal device 2, feature point information including information indicating, of the plurality of feature points, the positions of the plurality of feature points included in a partial image obtained by cutting out a portion of the first picked-up image. The terminal device 2 includes a terminal control unit that executes displaying a user interface image on a terminal display on the basis of the feature point information, acquiring selection information indicating a feature point that is selected from the plurality of feature points included in the partial image by using the user interface image, and transmitting the selection information to the projector 1.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an image processing method, and a program. [Background technology]

[0002] When a projector that projects a projection image onto a projection surface moves, the position of the projection image on the projection surface shifts from its initial position. For this reason, a projector that can return the position of the projection image to its initial position even if the installation position moves has been proposed, as shown in Patent Document 1. As an initial setting process, the projector described in Patent Document 1 captures an image of the outside scene including the projection surface with an imaging device and extracts multiple first feature points from the captured image. Furthermore, even after the installation position moves, this projector again captures an image of the outside scene and extracts multiple second feature points. Then, the position of the projection image is returned to its initial position using a projection transformation parameter that performs coordinate transformation from the first feature points to the corresponding second feature points. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-92169 Summary of the Invention [Problem to be solved by the invention]

[0004] The projector described in Patent Document 1 extracts feature points using a predetermined algorithm, but the extracted feature points may include feature points that are inappropriate as reference points for the projected image. For this reason, it is desirable to enable the user to select an appropriate feature point as a reference point from the multiple feature points extracted by the projector. However, Patent Document 1 does not describe how to select an appropriate feature point from the extracted feature points. Therefore, a technology that enables the user to easily select feature points is desired. [Means for solving the problem]

[0005] The image processing system includes a projector including a first communication device and a first processing device that executes the following: acquiring a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; and transmitting feature point information including information indicating the positions of a plurality of feature points included in a partial image obtained by cutting out a portion of the captured image to a terminal device via the first communication device; a display device; a second communication device; and the terminal device that executes the following: displaying a user interface image on the display device based on the feature point information; acquiring selection information indicating a feature point selected using the user interface image from the plurality of feature points included in the partial image; and transmitting the selection information to the projector via the second communication device.

[0006] The image processing method includes acquiring a captured image of a projection surface onto which a projection image is projected, detecting a plurality of feature points by performing image processing on the captured image, transmitting feature point information including information indicating the positions of a plurality of feature points included in a partial image obtained by cutting out a portion of the captured image to a terminal device, and acquiring selection information from the terminal device indicating a feature point selected from the plurality of feature points included in the partial image.

[0007] The program causes a computer to perform the following operations: acquire a captured image of a projection surface onto which a projection image is projected; detect a plurality of feature points by performing image processing on the captured image; transmit feature point information to a terminal device, the feature point information including information indicating the positions of a plurality of feature points included in a partial image obtained by cutting out a portion of the captured image; and acquire selection information from the terminal device indicating a feature point selected from the plurality of feature points included in the partial image. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view showing a schematic configuration of an image projection system. [Figure 2] FIG. 1 is a block diagram showing the internal configuration of a projector. [Figure 3] FIG. 2 is a schematic diagram showing a schematic configuration of an image projection unit. [Figure 4] FIG. 2 is a block diagram showing a schematic configuration of a terminal device. [Figure 5] 10 is a flowchart for explaining the operation of the projector during initial setting of the projection maintenance function. [Figure 6] FIG. 10 is a diagram showing the state of the projection surface at the time of initial setting of the projection maintenance function. [Figure 7] FIG. 10 is a diagram showing the state of the projection surface at the time of initial setting of the projection maintenance function. [Figure 8] FIG. 10 is a diagram showing the state of the projection surface at the time of initial setting of the projection maintenance function. [Figure 9] FIG. 10 is a diagram showing the state of the projection surface at the time of initial setting of the projection maintenance function. [Figure 10] FIG. 10 is a diagram showing a first captured image of the projection surface during initial setting of the projection maintenance function. [Figure 11] FIG. 10 is a diagram showing a first captured image of the projection surface during initial setting of the projection maintenance function. [Figure 12] FIG. 4 is a diagram showing a partial image cut out from the first captured image. [Figure 13] 10 is a flowchart for explaining the operation of the terminal device when selecting feature points. [Figure 14] FIG. 2 is a diagram showing a terminal display unit of the terminal device. [Figure 15] 10 is a flowchart for explaining the operation of the projector during restoration processing. [Figure 16] FIG. 10 is a diagram showing a partial image according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1. Embodiment The image projection system 100 of this embodiment will be described below with reference to the drawings. 1 is a perspective view showing a schematic configuration of an image projection system 100. The image projection system 100 is an example of an image processing system.

[0010] As shown in FIG. 1, the image projection system 100 includes a projector 1 that projects a projection image Ep onto a projection surface Sp, and a terminal device 2 that is used by a user of the projector 1. The projector 1 of this embodiment has a projection maintenance function that maintains the position and shape of the projection image Ep on the projection surface Sp even if the position or attitude of the projector 1 moves slightly. The terminal device 2 is used by the user when using the projection maintenance function. Details of the projection maintenance function will be described later. If the maximum range onto the projection surface Sp onto which the projector 1 can project an image after installation is defined as a projectable range Aa, the projector projects the projection image Ep, which is a significant image, into a projection range Ap that is a portion of the projectable range Aa.

[0011] FIG. 2 is a block diagram showing the internal configuration of the projector 1, and FIG. 3 is a schematic diagram showing the general configuration of an image projection unit 15 provided in the projector 1. As shown in FIG.

[0012] 2, the projector 1 includes a control unit 10, a storage unit 11, a communication unit 12, an image input unit 13, an image correction unit 14, an image projection unit 15, an operation unit 16, and an imaging unit 17. The projector 1 projects a projection image Ep from the image projection unit 15 onto a projection surface Sp based on image data input to the image input unit 13.

[0013] The control unit 10 includes one or more processors such as a CPU (Central Processing Unit). The control unit 10 operates in accordance with a control program 11a stored in the storage unit 11 to control the operation of the projector 1. The control unit 10 is an example of a first processing device, and corresponds to a computer.

[0014] The storage unit 11 includes memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The RAM is used for temporary storage of various data, and the ROM stores a control program 11a for controlling the operation of the projector 1, control data, various image data, and the like. The control program 11a is an example of a program. The control program 11a causes the control unit 10, which is a computer, to perform various operations, which will be described later.

[0015] The communication unit 12 is connected to a network NW such as a LAN (Local Area Network). The communication unit 12 includes an interface circuit for communicating via the network NW, and communicates with other devices connected to the network NW. In addition to the projector 1, a terminal device 2 is connected to the network NW of this embodiment. The communication unit 12 can input and output information to and from the terminal device 2 via the network NW. The connection between the communication unit 12 and the network NW may be wired or wireless. The communication unit 12 is an example of a first communication device.

[0016] The image input unit 13 has a plurality of input terminals (not shown), each of which is connected to an external image supply device 3 such as a computer or an image playback device. The image input unit 13 receives image data from the image supply device 3, performs processing such as converting the data format or the resolution of the supplied image data as necessary, and outputs the processed image data to the image correction unit 14. The image input unit 13 may also receive image data from an external device via the network NW and the communication unit 12.

[0017] Under the control of the control unit 10, the image correction unit 14 performs various correction processes on the image data input from the image input unit 13 and outputs the processed image data to the light valve driving unit 24 (see FIG. 3) of the image projection unit 15. For example, the image correction unit 14 performs processes on the image data, such as adjusting the image quality, correcting trapezoidal distortion, and superimposing an OSD (On Screen Display) image such as a message or menu on the projected image. Furthermore, in this embodiment, the image correction unit 14 performs correction processes using a projection maintenance function, i.e., correction processes for maintaining the position and shape of the projected image Ep on the projection surface Sp. The image correction unit 14 may be configured with one or more integrated circuits. The image correction unit 14 may be included in the control unit 10. Hereinafter, the image data input from the image input unit 13 to the image correction unit 14 will also be referred to as first image data D1, and the corrected image data output from the image correction unit 14 to the image projection unit 15 will also be referred to as second image data D2.

[0018] As shown in Fig. 3, the image projection unit 15 includes a light source 21, three liquid crystal light valves 22 as light modulation devices, a projection optical system 23, a light valve driver 24, etc. The image projection unit 15 modulates light emitted from the light source 21 with the liquid crystal light valves 22 to form image light, and projects this image light from the projection optical system 23, which includes a lens, to display a projection image Ep on a projection surface Sp. The liquid crystal light valves 22 include a liquid crystal light valve 22R for red light, a liquid crystal light valve 22G for green light, and a liquid crystal light valve 22B for blue light. The image projection unit 15 is an example of an optical device.

[0019] Light source 21 includes a solid-state light source such as a semiconductor laser or a light-emitting diode, or a discharge-type light source lamp such as an ultra-high pressure mercury lamp or a metal halide lamp. The light emitted from light source 21 is converted into light with a substantially uniform luminance distribution by an integrator optical system (not shown). The light is then separated into red, green, and blue color light components, which are the three primary colors of light, by a color separation optical system (not shown), and then these color light components are incident on liquid crystal light valves 22R, 22G, and 22B, respectively.

[0020] Each of the liquid crystal light valves 22R, 22G, and 22B is configured with a transmissive liquid crystal panel in which liquid crystal is sealed between a pair of transparent substrates. A rectangular pixel area 22i consisting of a plurality of pixels arranged in a matrix is formed on each liquid crystal panel, and a drive voltage can be selectively applied to the liquid crystal elements that make up each pixel. The pixel area 22i corresponds to the projectable range Aa.

[0021] The light valve driver 24 receives second image data D2 from the image corrector 14. The second image data D2 input from the image corrector 14 to the light valve driver 24 is image data representing the brightness of each pixel in the pixel region 22i. The light valve driver 24 forms an image in the pixel region 22i of the liquid crystal light valve 22 based on the second image data D2. Specifically, the light valve driver 24 applies a drive voltage corresponding to the second image data D2 input from the image corrector 14 to each pixel in the pixel region 22i, thereby setting the light transmittance of each pixel to the second image data D2. The light emitted from the light source 21 is modulated for each pixel as it passes through the pixel region 22i of the liquid crystal light valves 22R, 22G, and 22B, and image light corresponding to the second image data D2 is formed for each color light. The formed image light for each color is combined for each pixel by a color combining optical system (not shown) to form image light representing a color image, which is then enlarged and projected onto the projection surface Sp by the projection optical system 23.

[0022] In the pixel region 22i of the liquid crystal light valve 22, the projected image Ep is formed within the effective region 22e inside the pixel region 22i. In other words, the effective region 22e is the region corresponding to the projection range Ap. The region outside the effective region 22e is set to black, i.e., the minimum light transmittance. In other words, in the second image data D2 supplied from the image corrector 14 to the light valve driver 24, the region outside the effective region 22e is set to a value corresponding to black. Therefore, on the projection surface Sp, the projected image Ep is displayed within the projection range Ap, and almost no light is irradiated outside the projection range Ap.

[0023] Hereinafter, the image formed in the pixel region 22i of the liquid crystal light valve 22 and the image based on the second image data D2 input from the image correction unit 14 to the image projection unit 15 will also be referred to as a panel image. In other words, the panel image is an image formed in the entire pixel region 22i, including the inside and outside of the effective area 22e.

[0024] Returning to FIG. 2, the operation unit 16 has a plurality of operation keys (not shown) that the user uses to give various instructions to the projector 1. When the user operates an operation key on the operation unit 16, the operation unit 16 outputs an operation signal corresponding to the user's operation to the control unit 10. Note that a remote control (not shown) that allows remote operation may also be used as the operation unit 16. In this case, the remote control emits an infrared operation signal corresponding to the user's operation, which is received by a remote control signal receiving unit (not shown) and transmitted to the control unit 10. The operation unit 16 can also be called an operation device or an input device.

[0025] The imaging unit 17 captures an image of the front, which is the direction in which an image is projected from the image projection unit 15, and generates image data representing the captured image. Hereinafter, the image data representing the captured image will also be simply referred to as the captured image. The imaging unit 17 captures an image of an imaging range that includes at least the projectable range Aa. The imaging unit 17 outputs the generated captured image to the control unit 10. The imaging unit 17 is an imaging device that includes an imaging element 18 such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, an imaging lens (not shown), and the like. The imaging element 18 has a light-receiving area including multiple pixels, and the captured image is generated based on the amount of light received by each pixel.

[0026] FIG. 4 is a block diagram showing a schematic configuration of the terminal device 2. As shown in FIG. The terminal device 2 is a portable information processing device used by a user of the projector 1, and corresponds to, for example, a smartphone or a tablet terminal device. However, the terminal device 2 is not limited to a portable device, and may be, for example, a stationary personal computer.

[0027] The terminal device 2 includes a terminal control unit 30, a terminal storage unit 31, a terminal communication unit 32, a terminal operation unit 33, and a terminal display unit 34. The terminal control unit 30 includes a processor such as a CPU, and controls the operation of the terminal device 2 by operating in accordance with an operating system (OS) and application programs (not shown) stored in the terminal storage unit 31. The terminal control unit 30 is an example of a second processing device.

[0028] The terminal storage unit 31 is configured by a storage device such as a memory, and stores the above-mentioned OS, application programs, various setting data, and the like.

[0029] The terminal communication unit 32 includes an interface circuit for communicating via the network NW, and is capable of communicating with the projector 1 via the network NW. The connection between the terminal communication unit 32 and the network NW is usually wireless, but may be wired. The terminal communication unit 32 is an example of a second communication device.

[0030] The terminal operation unit 33 is an input device that accepts input operations from the user. The terminal operation unit 33 outputs operation information corresponding to the accepted input operations to the terminal control unit 30. The terminal display unit 34 is equipped with a display device such as a liquid crystal display or an organic EL display, and displays various information based on the control of the terminal control unit 30. The terminal display unit 34 is an example of a display device. The terminal operation unit 33 may be configured integrally with the terminal display unit 34, such as a touch panel.

[0031] The terminal storage unit 31 stores various application programs (not shown), and the terminal control unit 30 can execute various operations in accordance with the launched application programs. The application programs stored in the terminal storage unit 31 include a feature point selection program used when performing initial setup of the projection maintenance function. This program is provided, for example, by the manufacturer or distributor of the projector 1, or by some other third party.

[0032] Next, the image processing method of the projector 1 will be described. FIG. 5 is a flowchart for explaining the image processing method of the projector 1, and shows the operation of the projector 1 when initializing the projection maintenance function. Also, FIGS. 6 to 9 are diagrams showing the state of the projection surface Sp when initializing the projection maintenance function, and FIGS. 10 and 11 are diagrams showing a first captured image Pm obtained by capturing an image of the projection surface Sp when initializing the projection maintenance function. Also, FIG. 12 is a diagram showing a partial image Pn cut out from the first captured image Pm. When the user starts up the projector 1 and then performs an operation to initialize the projection maintenance function, the control unit 10 operates according to the flow shown in FIG. 5.

[0033] The position of the projector 1 is roughly adjusted in advance by the user so that the projected image Ep is projected in a desired projection range Ap on the projection surface Sp. Specifically, the user installs the projector 1 so that the projectable range Aa of the projector 1 covers the desired projection range Ap. In this embodiment, the desired projection range Ap is set inside a rectangular frame F provided on the projection surface Sp.

[0034] First, in step S101, the control unit 10 causes the image projection unit 15 to project a pattern image Pp, in which a plurality of dot-shaped marks M1 are arranged, onto the projection surface Sp within the projectable range Aa. Image data representing the pattern image Pp is stored in advance in the storage unit 11, and the control unit 10 reads this image data from the storage unit 11 and outputs it to the image correction unit 14. When the image correction unit 14 outputs this image data to the image projection unit 15 without correction, the pattern image Pp is projected from the image projection unit 15 onto the projection surface Sp.

[0035] In step S102, the control unit 10 causes the imaging unit 17 to capture an image of the imaging range including the projectable range Aa. That is, the control unit 10 causes the imaging unit 17 to capture an image of the projection surface Sp on which the pattern image Pp is projected, and acquires the captured image from the imaging unit 17.

[0036] In step S103, the control unit 10 detects, in the captured image, a plurality of marks M1 in the captured pattern image Pp. Then, the control unit 10 generates first correspondence information representing the correspondence relationship between coordinates on the panel image and coordinates on the captured image based on the positions of the plurality of marks M1 in the captured image and the positions of the plurality of marks M1 in the panel image. The positions of the plurality of marks M1 in the panel image are known information.

[0037] In step S104, the control unit 10 controls the image projection unit 15 to project the adjusted image Pc onto the projection surface Sp, as shown in FIG. 7. A reference image Pb of a predetermined size is placed in the center of the adjusted image Pc. The reference image Pb is an image for determining the projection range Ap in which the projected image Ep is displayed. The shape of the reference image Pb is, for example, a rectangular shape on the pixel region 22i of the liquid crystal light valve 22, i.e., the panel image. Image data representing the adjusted image Pc is pre-stored in the storage unit 11, and the control unit 10 reads this image data from the storage unit 11 and outputs it to the image correction unit 14. When the image correction unit 14 outputs this image data to the image projection unit 15 without correction, the adjusted image Pc including the reference image Pb is projected from the image projection unit 15 onto the projection surface Sp. For example, cross marks M2 are attached to the four corners of the reference image Pb.

[0038] In step S105, the control unit 10 determines the projection range Ap based on the user's operation. Specifically, the control unit 10 determines the positions of the four corners of the reference image Pb on the panel image based on the user's operation. Because the position and shape of the rectangular reference image Pb are determined by the positions of the corners of the reference image Pb, determining the positions of the corners of the reference image Pb is equivalent to determining the position and shape of the reference image Pb. The user can individually specify the positions of the four corners of the reference image Pb, i.e., the positions of the marks M2, by operating the operation unit 16. The control unit 10 deforms the reference image Pb by moving the marks M2 specified by the user in the direction specified by the user. The user adjusts the position of each mark M2 so that the position and shape of the reference image Pb projected on the projection surface Sp match the position and shape of the desired projection range Ap. Then, on the projection surface Sp, the rectangular range whose vertices are the adjusted positions of the four marks M2 is determined as the projection range Ap. For example, as shown in Fig. 8, the user adjusts the positions of the corners of the reference image Pb, i.e., the corners of the projection range Ap, inside a rectangular frame F on the projection surface Sp so that each side is parallel to the frame F. Note that in Fig. 8, the projection range Ap is adjusted to be smaller than the inner edge of the frame F, but it may also be adjusted to coincide with the inner edge of the frame F, for example.

[0039] In step S106, the control unit 10 determines a rectangular area having vertices at the positions of the four adjusted marks M2 on the panel image as the effective area 22e. Then, the control unit 10 generates a first conversion formula for converting the first image data D1 so that the position and shape of the projected image Ep on the panel image coincide with the position and shape of the effective area 22e.

[0040] In step S107, the control unit 10 causes the imaging unit 17 to capture an image of the projection surface Sp in a state where no image is being projected from the image projection unit 15. Then, as shown in FIG. 10, the control unit 10 acquires a captured image of the projection surface Sp from the imaging unit 17. Note that the image projection unit 15 may project a predetermined image, such as an image in which the entire projectible range Aa has substantially uniform brightness. Hereinafter, of the captured images captured by the imaging unit 17, the captured image captured in step S107 will also be referred to as a first captured image Pm.

[0041] In step S108, the control unit 10 detects multiple feature points by performing image processing on the acquired first captured image Pm. For example, AKAZE can be used as an algorithm for detecting feature points, and feature points are detected based on feature quantities that quantify features. The feature points to be detected here are points that serve as reference points when maintaining the position and shape of the projected image Ep using the projection maintenance function. For this reason, feature points must be points on the projection surface Sp that do not move during the period in which the projector 1 is used. It is desirable that each feature point be located near the four corners of the projection range Ap. In this embodiment, the four corners of the frame F are detected as feature points, and these four feature points are used as reference points for the projection maintenance function. Furthermore, in this step, feature points that are located far from the projection surface Sp, feature points caused by foreign objects or shadows, or other feature points that are inappropriate as reference points for the projection maintenance function may also be detected. Instead of the frame F, multiple markers may be placed on the projection surface Sp, and these markers may be detected as feature points.

[0042] The control unit 10 associates the coordinates on the first captured image Pm with the feature amount for each detected feature point and stores them in the storage unit 11. Based on the coordinates of each feature point on the first captured image Pm, the control unit 10 can determine which of the four corners of the projection range Ap the feature point is located near.

[0043] In step S109, the control unit 10 causes the image projection unit 15 to project a feature point image Pf onto the projection surface Sp, as shown in Fig. 9. The feature point image Pf is an image in which marks M3 indicating the positions of the feature points are arranged at positions on the projection surface Sp that correspond to the positions of the feature points in the first captured image Pm. In other words, the feature point image Pf includes a plurality of marks M3 corresponding to the plurality of feature points detected in step S108.

[0044] Specifically, the control unit 10 converts the positions of feature points in the first captured image Pm into positions on the panel image based on first correspondence information that indicates the correspondence between coordinates on the panel image and coordinates on the captured image. Then, the control unit 10 generates image data of a feature point image Pf in which marks M3 are placed at these positions, and outputs the image data to the image correction unit 14. The marks M3 are, for example, round marks. When the image correction unit 14 outputs the image data of the feature point image Pf generated by the control unit 10 to the image projection unit 15 without correction, the feature point image Pf including the multiple marks M3 is projected from the image projection unit 15 onto the projection surface Sp. Note that, as shown in FIG. 9, the feature point image Pf may include at least one of an image indicating the boundary of the projection range Ap and marks M2 indicating the corners of the projection range Ap.

[0045] In step S110, the control unit 10 determines whether feature points have been determined at all four corners of the projection range Ap. For example, if one feature point is detected near each of the four corners of the projection range Ap, feature points are determined at all corners. On the other hand, as shown in FIG. 9, if multiple feature points are detected near any corner of the projection range Ap, there is a possibility that one of the feature points is inappropriate as a reference point for the projection maintenance function. Therefore, in this case, the feature point remains undetermined until the user selects a feature point appropriate as a reference point for the projection maintenance function. The selection of a feature point by the user will be described later. If all feature points have been determined (step S110: YES), the control unit 10 ends the flow. If no feature point has been determined at any corner of the projection range Ap (step S110: NO), the control unit 10 proceeds to step S111. For example, in the example shown in FIG. 9, multiple feature points are detected at the upper left corner and the lower right corner, so the control unit 10 proceeds to step S111.

[0046] When the process proceeds to step S111, the control unit 10 cuts out a partial image Pn included in a rectangular area near a corner where multiple feature points were detected from the first captured image Pm captured in step S107, as shown in FIG. 11. For example, in the example shown in FIG. 11, the control unit 10 cuts out the partial image Pn from near the upper left corner of the corners where multiple feature points were detected. Then, as shown in FIG. 12, the control unit 10 performs a superimposition process to superimpose an image indicating the boundary of the projection range Ap and a mark M2 indicating the corner of the projection range Ap on the cut-out partial image Pn. In this way, the partial image Pn is an image cut out from a part of the first captured image Pm and includes a corner of the projection range Ap, i.e., a corner of the projected image Ep. Furthermore, although the mark M3 is not added to the partial image Pn, the partial image Pn includes multiple feature points.

[0047] In step S112, the control unit 10 transmits feature point information regarding a plurality of feature points included in the partial image Pn to the terminal device 2 via the communication unit 12. The feature point information includes information indicating the positions of the plurality of feature points, information indicating the position of the partial image Pn in the first captured image Pm, and image data of the partial image Pn after the superimposition process. The information indicating the positions of the plurality of feature points is, for example, information indicating the coordinates of the plurality of feature points in the first captured image Pm. In this way, the control unit 10 transmits feature point information including information indicating the positions of a plurality of feature points included in the partial image Pn obtained by cutting out a portion of the first captured image Pm, out of the plurality of feature points detected in step S108, to the terminal device 2 via the communication unit 12.

[0048] Fig. 13 is a flowchart for explaining the operation of the terminal device 2 when selecting a feature point. Fig. 14 is a diagram showing the terminal display unit 34 of the terminal device 2, illustrating the user interface image Pu displayed on the terminal display unit 34 when the user selects a feature point.

[0049] When selecting feature points on the terminal device 2, the user launches a feature point selection program on the terminal device 2 in advance. When feature point information is transmitted from the projector 1, the terminal control unit 30 receives the feature point information via the terminal communication unit 32 (step S121). Then, as shown in FIG. 14 , the terminal control unit 30 displays a user interface image Pu on the terminal display unit 34 based on the received feature point information (step S122). Specifically, the terminal control unit 30 displays a partial image Pn on the terminal display unit 34 based on image data included in the feature point information. Furthermore, the terminal control unit 30 displays a plurality of marks M4 as images indicating feature points superimposed on the partial image Pn based on information indicating the position of the partial image Pn and information indicating the positions of the feature points, both included in the feature point information. That is, the user interface image Pu includes the partial image Pn and a plurality of marks M4 indicating feature points. The user interface image Pu also includes a message Ms prompting the user to select feature points.

[0050] The user can select a feature point by specifying one of the multiple marks M4 displayed on the terminal display unit 34 using the terminal operation unit 33. For example, if the terminal operation unit 33 is configured as a touch panel, the user selects a feature point by touching one of the marks M4. In this case, the user selects the feature point that corresponds to one of the four corners of the frame F from the multiple feature points. Specifically, the user selects the feature point that is closest to the one of the four corners of the frame F.

[0051] The terminal control unit 30 receives the user's selection of feature points via the terminal operation unit 33 (step S123). When the user selects one feature point, the terminal control unit 30 acquires selection information indicating the selected feature point (step S124). The selection information is, for example, information indicating the position of the selected feature point. In this way, the terminal control unit 30 acquires selection information indicating the feature point selected using the user interface image Pu from among the multiple feature points included in the partial image Pn. The terminal control unit 30 transmits the acquired selection information to the projector 1 via the terminal communication unit 32 (step S125).

[0052] Returning to FIG. 5, in step S113, the control unit 10 receives the selection information transmitted from the terminal device 2, i.e., the coordinates of the selected feature point, via the communication unit 12. That is, the control unit 10 acquires the selection information from the terminal device 2. The control unit 10 then stores the acquired selection information in the storage unit 11. As a result, a feature point is determined for one corner of the projection range Ap. Thereafter, the control unit 10 returns the process to step S110. If there are other corners at which multiple feature points have been detected, the above-described steps S111 to S113 are repeated for those corners to determine the feature points. On the other hand, if feature points have been determined for all four corners, the control unit 10 ends the flow.

[0053] This completes the initial setting of the projection maintenance function. Thereafter, when the image projection unit 15 projects the projection image Ep, the image correction unit 14 corrects the first image data D1 input from the image input unit 13 using the first conversion formula. As a result, the projection image Ep is projected onto the projection surface Sp with its position and shape corrected according to the projection range Ap determined in step S105. The correction process performed on the first image data D1 using the first conversion formula is also referred to as the first correction process. Thereafter, for example, if the position of the projector 1 has moved or is expected to have moved, a restoration process is executed to restore the position and shape of the projection image Ep.

[0054] 15 is a flowchart for explaining the image processing method of the projector 1, and shows the operation of the projector 1 when performing restoration processing. In this restoration processing, the control unit 10 generates a second conversion formula for correcting the position and shape of the projected image Ep on the projection surface Sp, using the feature points determined during initial setup, i.e., four feature points including the feature points indicated by the acquired selection information.

[0055] As shown in FIG. 15, steps S201 to S203 are the same as steps S101 to S103 described above. That is, in step S201, the control unit 10 causes the image projection unit 15 to project a pattern image Pp, in which a plurality of dot-shaped marks M1 are arranged, into the projectable range Aa of the projection surface Sp. In step S202, the control unit 10 causes the imaging unit 17 to capture an image of an imaging range including the projectable range Aa. In step S203, the control unit 10 detects the plurality of marks M1 in the captured pattern image Pp in the captured image. Then, based on the positions of the plurality of marks M1 in the captured image and the positions of the plurality of marks M1 in the panel image, the control unit 10 generates second correspondence information that indicates the correspondence between coordinates on the panel image and coordinates on the captured image. If the projector 1 has moved after the initial setting, the second correspondence information will be different from the first correspondence information generated in step S103.

[0056] The next step S204 is the same as step S107 described above. That is, the control unit 10 causes the imaging unit 17 to capture an image of the projection surface Sp in a state where no image is projected from the image projection unit 15. Then, the control unit 10 acquires a captured image of the projection surface Sp from the imaging unit 17. Hereinafter, of the captured images captured by the imaging unit 17, the captured image captured in step S204 will also be referred to as a second captured image.

[0057] In step S205, the control unit 10 detects a plurality of feature points by performing image processing on the newly acquired second captured image, similar to step S108 described above. Then, the control unit 10 extracts four feature points determined at the time of initial setup, i.e., four feature points corresponding to the four corners of the frame F, based on the feature amounts of each feature point. Specifically, the control unit 10 extracts four feature points having feature amounts comparable to the feature amounts of the four feature points determined at the time of initial setup.

[0058] In step S206, the control unit 10 generates a second conversion formula for correcting the position and shape of the projected image Ep on the projection surface Sp, and then ends the flow. Specifically, the control unit 10 generates the second conversion formula based on the first correspondence information generated in step S103, the second correspondence information generated in step S203, information indicating the positions of the four feature points in the captured image determined in steps S108 to S113, and information indicating the positions of the four feature points in the captured image extracted in step S205. The second conversion formula is an example of correction information.

[0059] Thereafter, when projecting the projection image Ep, the image correction unit 14 performs the above-described first correction process on the input first image data D1 and also performs conversion using the generated second conversion formula. As a result, the position and shape of the projection image Ep are corrected to the position and shape of the projection range Ap determined in step S105. The correction process performed using the second conversion formula generated in step S206 is also referred to as the second correction process. Each time the restoration process is performed, the control unit 10 regenerates the second conversion formula. Then, when projecting the projection image Ep, the image correction unit 14 executes the first correction process and the second correction process using the newly generated second conversion formula. As a result, the projector 1 maintains the position and shape of the projection image Ep.

[0060] The restoration process described above may be started by a user instruction when the projector 1 moves. Alternatively, the projector 1 may be provided with a detection means for detecting movement, and when the detection means detects movement of the projector 1, the control unit 10 may start the restoration process.

[0061] Furthermore, when the projector 1 is used for a long period of time, the restoration process may be set to be executed periodically regardless of whether or not the projector 1 is moving. In this case, it is desirable that the control unit 10 start up the projector 1 and execute the restoration process when the projector 1 is not being used, such as at night. In this way, when the restoration process is executed periodically, it is possible to prevent the position and shape of the projected image Ep from gradually changing over time.

[0062] As described above, the image projection system 100, the image processing method, and the control program 11a of this embodiment can provide the following effects.

[0063] According to this embodiment, feature point information regarding a plurality of feature points detected by the projector 1 is transmitted to the terminal device 2, and the user can specify the feature point to be selected on the terminal device 2. As a result, it becomes easy for the user to select the desired feature point from among the plurality of feature points.

[0064] According to this embodiment, the control unit 10 generates a second conversion formula for correcting the position and shape of the projection image Ep using the feature points indicated by the selection information received from the terminal device 2. Therefore, the projector 1 can correct the projection image Ep using appropriate feature points selected by the user.

[0065] According to this embodiment, the partial image Pn is an image that includes the corners of the projection range Ap, i.e., the corners of the projected image Ep, so the user can select feature points near the corners of the projected image Ep. When determining the position and shape of the projected image Ep, the corners of the projected image Ep are used as the reference, so by using feature points near the corners of the projected image Ep, the position and shape of the projected image Ep can be appropriately corrected.

[0066] According to this embodiment, information indicating the coordinates in the first captured image Pm is transmitted to the terminal device 2 as information indicating the positions of multiple feature points detected by the projector 1. Therefore, the feature points can be displayed at accurate positions in the user interface image Pu.

[0067] According to this embodiment, the feature point information transmitted to the terminal device 2 includes a partial image Pn, which is a part of the first captured image Pm obtained by capturing the projection surface Sp, so that a plurality of feature points can be displayed together with the actual projection surface Sp in the user interface image Pu. This improves the convenience when the user selects feature points. Furthermore, according to this embodiment, the amount of communication can be reduced compared to when the entire first captured image Pm is transmitted to the terminal device 2.

[0068] 2. Variations The above-described embodiment is based on the above-described configuration, but partial modifications or omissions of the configuration may be made without departing from the spirit of the present disclosure. Furthermore, this embodiment and the modifications described below may be combined with each other within the scope of technical compatibility. The modifications are described below.

[0069] In the above embodiment, the terminal control unit 30 superimposes marks M4 indicating feature points on the partial image Pn acquired from the projector 1, but the present invention is not limited to this. For example, as shown in FIG. 16 , the control unit 10 of the projector 1 may superimpose marks M3 indicating feature points on the partial image Pn. That is, marks M3 as images indicating feature points may be superimposed on the partial image Pn included in the feature point information transmitted from the projector 1 to the terminal device 2. According to this configuration, the partial image Pn on which marks M3 indicating multiple feature points have been superimposed in advance is transmitted to the terminal device 2. This allows the terminal device 2 to generate a user interface image Pu without performing complex processing.

[0070] In the above embodiment, the control unit 10 performs a superimposition process in which an image indicating the boundary of the projection range Ap and a mark M2 indicating a corner of the projection range Ap are superimposed on the partial image Pn cut out in step S111. However, these images do not have to be superimposed. However, if the partial image Pn does not include an image indicating the boundary of the projection range Ap, it becomes difficult for the user to identify which corner of the projection range Ap the partial image Pn displayed in the user interface image Pu corresponds to. For this reason, the control unit 10 may change the color of the mark M2 or mark M3 located near the corner from which the partial image Pn was cut out in the feature point image Pf projected onto the projection surface Sp from the color of the mark M2 or mark M3 located near the other corners. Alternatively, the color of the mark M2 or mark M3 in the feature point image Pf may be different for each corner of the projection range Ap, and the color of the mark M4 displayed in the user interface image Pu may be the same as the color corresponding to the corner from which the partial image Pn was cut out.

[0071] In the above embodiment, four feature points near the four corners of the projection range Ap are used as reference points for the projection maintenance function, but this is not limiting. For example, the number of feature points used may be three, or five or more. Therefore, the number of feature points that the user should select on the user interface image Pu is not limited to one per corner, and may be multiple.

[0072] In the above embodiment, the feature point information transmitted from the projector 1 to the terminal device 2 includes image data of the partial image Pn, but the image data of the partial image Pn does not have to be included. In this case, the feature point information may include, for example, only information indicating the positions of multiple feature points. The terminal control unit 30 may then place marks M4 indicating the feature points in a user interface image Pu that does not include the partial image Pn, and allow the user to select one. This configuration can further reduce the amount of communication when the projector 1 transmits feature point information.

[0073] In the above embodiment, the information indicating the positions of the plurality of feature points included in the feature point information is not limited to information indicating the coordinates of the plurality of feature points in the first captured image Pm, but may be information indicating the coordinates in the partial image Pn, for example.

[0074] In the above embodiment, the selection information transmitted from the terminal device 2 to the projector 1 is not limited to information indicating the position of the selected feature point, but may be any information that can identify the selected feature point. For example, it may be identification information associated with the feature point.

[0075] In the above embodiment, a configuration has been shown in which a projection image Ep is projected based on image data input from outside to the image input unit 13, but this configuration is not limiting. For example, a configuration may be used in which an image based on image data stored in the storage unit 11 is projected as the projection image Ep. In this case, the control unit 10 outputs the image data stored in the storage unit 11 to the image correction unit 14 as first image data D1.

[0076] In the above embodiment, a transmissive liquid crystal light valve 22 is used as the light modulation device, but a reflective light modulation device such as a reflective liquid crystal light valve can also be used. Also, a digital mirror device or the like can be used that modulates the light emitted from the light source 21 by controlling the emission direction of incident light for each micromirror pixel. Also, the configuration is not limited to one having multiple light modulation devices for each color of light, and a configuration in which multiple color lights are modulated in a time-division manner by one light modulation device can be used.

[0077] 3. Summary of this disclosure A summary of this disclosure is provided below.

[0078] (Supplementary Note 1) An image processing system including: a projector including a first communication device; a first processing device that executes the following: acquiring a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; and transmitting, to a terminal device via the first communication device, feature point information including information indicating positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image from the plurality of feature points; a display device; a second communication device; and the terminal device that executes the following: displaying a user interface image on the display device based on the feature point information; acquiring selection information indicating a feature point selected using the user interface image from the plurality of feature points included in the partial image; and transmitting the selection information to the projector via the second communication device.

[0079] According to the configuration of Supplementary Note 1, feature point information regarding a plurality of feature points detected by the projector is transmitted to the terminal device, allowing the user to specify the feature point to be selected on the terminal device. As a result, it becomes easy for the user to select the desired feature point from the plurality of feature points.

[0080] (Appendix 2) An image processing system according to Appendix 1, wherein the first processing device further generates correction information for correcting the position and shape of the projected image on the projection surface using the feature points indicated by the selection information.

[0081] According to the configuration of Supplementary Note 2, the first processing device generates correction information for correcting the position and shape of the projected image using the feature points indicated by the selection information received from the terminal device, so that the projector can correct the projected image using appropriate feature points selected by the user.

[0082] (Supplementary Note 3) The image processing system according to Supplementary Note 1 or 2, wherein the partial image is an image that includes a corner of the projected image.

[0083] According to the configuration of Supplementary Note 3, since the partial image is an image that includes the corner of the projected image, the user can select a feature point near the corner of the projected image. When determining the position and shape of the projected image, the corner of the projected image is used as a reference, so by using the feature point near the corner of the projected image, the position and shape of the projected image can be appropriately corrected.

[0084] (Appendix 4) An image processing system according to any one of Appendices 1 to 3, wherein the information indicating the positions of the plurality of feature points included in the feature point information is information indicating the coordinates of the plurality of feature points in the captured image.

[0085] According to the configuration of Supplementary Note 4, information indicating coordinates in the captured image is transmitted to the terminal device as information indicating the positions of the detected multiple feature points, so that the feature points can be displayed at accurate positions in the user interface image.

[0086] (Supplementary Note 5) The image processing system according to any one of Supplementary Notes 1 to 4, wherein the feature point information further includes the partial image.

[0087] According to the configuration of Supplementary Note 5, the feature point information transmitted to the terminal device includes a partial image that is a part of the captured image of the projection surface, so that a plurality of feature points can be displayed together with the actual projection surface in the user interface image, thereby improving the convenience for the user when selecting feature points.

[0088] (Supplementary Note 6) The image processing system according to Supplementary Note 5, wherein an image showing the plurality of feature points is superimposed on the partial image.

[0089] According to the configuration of Supplementary Note 6, a partial image on which an image showing a plurality of feature points is superimposed in advance is transmitted to the terminal device, which allows the terminal device to generate a user interface image without performing complex processing.

[0090] (Appendix 7) An image processing method including: acquiring a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; transmitting feature point information including information indicating positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image to a terminal device; and acquiring selection information from the terminal device indicating a feature point selected from the plurality of feature points included in the partial image.

[0091] According to the configuration of Supplementary Note 7, feature point information regarding a plurality of feature points detected from a captured image is transmitted to a terminal device, and the user can specify the feature points to be selected on the terminal device. As a result, the user can easily select the desired feature points from the plurality of feature points.

[0092] (Appendix 8) A program that causes a computer to perform the following operations: acquire a captured image of a projection surface onto which a projection image is projected; detect a plurality of feature points by performing image processing on the captured image; transmit feature point information including information indicating the positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image to a terminal device; and acquire selection information from the terminal device indicating a feature point selected from the plurality of feature points included in the partial image.

[0093] According to the configuration of Supplementary Note 8, feature point information regarding a plurality of feature points detected from a captured image is transmitted to a terminal device, and therefore the user can specify the feature points to be selected on the terminal device. As a result, it becomes easy for the user to select a desired feature point from among the plurality of feature points. [Explanation of symbols]

[0094] 1...projector, 2...terminal device, 3...image supply device, 10...control unit, 11...storage unit, 11a...control program, 12...communication unit, 13...image input unit, 14...image correction unit, 15...image projection unit, 16...operation unit, 17...imaging unit, 18...imaging element, 21...light source, 22, 22R, 22G, 22B...liquid crystal light valve, 22e...effective area, 22i...pixel area, 23...projection optical system, 24...light valve driving unit, 30...terminal control unit, 31...terminal memory unit, 3 2...terminal communication unit, 33...terminal operation unit, 34...terminal display unit, 100...image projection system, Aa...projectable range, Ap...projection range, D1...first image data, D2...second image data, F...frame, M1 to M4...mark, Ms...message, NW...network, Ep...projected image, Pb...reference image, Pc...adjusted image, Pf...feature point image, Pm...first captured image, Pn...partial image, Pp...pattern image, Pu...user interface image, Sp...projection surface.

Claims

1. a first communication device; Obtaining a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; transmitting, to the terminal device via the first communication device, feature point information including information indicating positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image, among the plurality of feature points; a first processing device that executes the above; a projector including: A display device; a second communication device; displaying a user interface image on the display device based on the feature point information; acquiring selection information indicating a feature point selected using the user interface image from among the plurality of feature points included in the partial image; transmitting the selection information to the projector via the second communication device; a second processing device that executes the and the terminal device including: Image processing system.

2. 2. The image processing system according to claim 1, the first processing device further generates correction information for correcting the position and shape of the projected image on the projection surface, using the feature points indicated by the selection information. Image processing system.

3. 2. The image processing system according to claim 1, the partial image is an image including a corner of the projected image; Image processing system.

4. 4. The image processing system according to claim 1, the information indicating the positions of the plurality of feature points included in the feature point information is information indicating coordinates of the plurality of feature points in the captured image; Image processing system.

5. 4. The image processing system according to claim 1, the feature point information further includes the partial image; Image processing system.

6. 6. The image processing system according to claim 5, an image showing the plurality of feature points is superimposed on the partial image; Image processing system.

7. Obtaining a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; transmitting, to a terminal device, feature point information including information indicating positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image, among the plurality of feature points; acquiring, from the terminal device, selection information indicating a feature point selected from the plurality of feature points included in the partial image; Image processing methods.

8. On the computer, Obtaining a captured image of a projection surface onto which a projection image is projected; detecting a plurality of feature points by performing image processing on the captured image; transmitting, to a terminal device, feature point information including information indicating positions of a plurality of feature points included in a partial image obtained by cutting out a part of the captured image, among the plurality of feature points; acquiring, from the terminal device, selection information indicating a feature point selected from the plurality of feature points included in the partial image; program.

Images