Projection system and control method for projection system

Abstract

To enable a state where an image such as a projection image of a projector is superimposed on handwritten characters or the like to be shared by a plurality of devices.SOLUTION: A projection system 100 including a projector 1 and a display device 6 receives an image from the display device 6 by the projector 1, projects the received image to a projection object OB, images a range including the projection object OB, transmits a photographed image to the display device 6, generates a drawn image based on a drawing operation by the display device 6, displays the drawn image on a display, transmits the drawn image to the projector 1, receives the photographed image from the projector 1, performs processing of excluding an image superimposed on the photographed image received from the projector 1 in the drawn image displayed on the display, and updates the display of the display on the basis of the image after processing and the photographed image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a projection system and a control method for a projection system. [Background technology]

[0002] Conventionally, there are known techniques for drawing characters or the like by superimposing them on an image projected by a projector. For example, the device disclosed in Patent Document 1 allows a user to write characters by hand and superimpose them on an image projected on a screen. This device records the handwritten character data on a recording medium in association with the projected video data, and can then play back the video data with the handwritten characters superimposed on it. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-161748 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, it has not been easy to share an image, such as an image projected by a projector, with handwritten characters or the like superimposed on the image, among a plurality of devices. [Means for solving the problem]

[0005] One aspect of the present disclosure is a projection system comprising a projector and a display device, wherein the projector comprises a projector communication unit that communicates with the display device, a projection unit that projects an image onto a projection target, and a capture unit that captures an image of an area including the projection target, wherein the projector projects an image received by the projector communication unit using the projection unit, and transmits an image captured by the capture unit using the projector communication unit, and the display device comprises a display communication unit that communicates with the projector, a display that displays an image, and an operation unit that accepts drawing operations, wherein the display device generates a drawn image based on the drawing operation accepted by the operation unit, displays the drawn image on the display, transmits the drawn image using the display communication unit, receives the captured image using the display communication unit, processes the drawn image displayed on the display to remove any images that overlap with the captured image received from the projector, and updates the display based on the processed image and the captured image.

[0006] Another aspect of the present disclosure is a control method for a projection system including a projector and a display device, in which the projector receives an image from the display device, projects the received image onto a projection target, captures an image of an area including the projection target, and transmits the captured image to the display device, the display device generates a drawing image based on a drawing operation, displays the drawing image on a display, transmits the drawing image to the projector, receives the captured image from the projector, and processes the drawn image displayed on the display to remove any images that overlap with the captured image received from the projector, and updates the display based on the processed image and the captured image. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a projection system according to an embodiment. [Figure 2] FIG. 2 is a block diagram of each device of the projection system according to the embodiment. [Figure 3] FIG. 2 is a schematic diagram showing an example of the configuration of a projection unit and an imaging unit. [Figure 4] FIG. 10 is a schematic diagram showing another example of the configuration of the projection unit and the imaging unit. [Figure 5] FIG. 4 is a sequence diagram showing the operation of the projection system according to the embodiment. [Figure 6] FIG. 4 is a sequence diagram showing the operation of the projection system according to the embodiment. [Figure 7] FIG. 2 is a schematic diagram illustrating the operation of the projection system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] [1. Projection system configuration] 1 is a diagram showing a schematic configuration of a projection system 100 according to an embodiment. The projection system 100 includes a projector 1 and a display device 6. The projector 1 and the display device 6 are connected via a communication network 3 so as to be able to communicate data with each other.

[0009] There is no limitation on the installation locations of the projector 1 and the display device 6. For example, the use location S1 of the projector 1 and the use location S2 of the display device 6 may be located far from each other or close to each other.

[0010] The communication network 3 is a network that enables data communication between devices. The communication network 3 may be, for example, a local network such as a LAN (Local Area Network) or a wide area network. The communication network 3 may also be, for example, an open network such as the Internet. The communication network 3 may include communication lines such as dedicated lines, public lines, or cellular communication lines, and communication devices such as routers and gateway devices. The projector 1 and the communication network 3 may be connected by wire via a communication cable, or may be connected wirelessly by a wireless communication path. Similarly, the display device 6 and the communication network 3 may be connected by wire via a communication cable, or may be connected wirelessly by a wireless communication path. The communication cable may be, for example, a LAN cable or a USB cable conforming to the USB (Universal Serial Bus) communication standard. The wireless communication path may be, for example, Wi-Fi or Bluetooth. Wi-Fi is a registered trademark. Bluetooth is a registered trademark.

[0011] The projector 1 projects image light PL toward the projection target OB, forming a projection image PP on the projection target OB. Projecting the image light PL by the projector 1 corresponds to displaying the projection image PP on the projection target OB. In the following description, the term "image" includes video and still images.

[0012] FIG. 1 shows a configuration in which the projector 1 is installed above the projection target OB and projects the image light PL downward from the projector 1, but this is just one example. The projection target OB may be a flat surface, or may have a curved or uneven surface. The position of the projector 1 and the orientation in which the projector 1 is installed are determined according to the positional relationship with the projection target OB. For example, the projector 1 may be installed in an orientation in which the image light PL is projected horizontally or in an orientation in which the image light PL is projected upward.

[0013] As will be described later, the projector 1 has a function of capturing an image of the projection target OB. The projector 1 transmits a captured image of the projection target OB to the display device 6 via the communication network 3.

[0014] The display device 6 is a device having a display 61. The display device 6 may be any device having the display 61, an input function, and a communication function. For example, the display device 6 is configured as a tablet computer, a laptop computer, or a smartphone. The display device 6 described in this embodiment has a function of detecting an operation of a pointer 65 on the display 61 and drawing a drawn image DP based on the operation by the pointer 65. The pointer 65 is, for example, a pen-shaped device as shown in FIG. 1. The display device 6 may be configured so that the user's finger can be used as the pointer 65.

[0015] [2. Projector and display device configuration] FIG. 2 is a block diagram of each device in the projection system 100. The projector 1 includes a projection unit 10 that projects image light PL and a drive circuit 13 that drives the projection unit 10. The projection unit 10 includes an image light formation unit 11 and a projection optical system 12.

[0016] The image light forming unit 11 generates the image light PL. The image light forming unit 11 includes self-emitting elements that emit predetermined color light. The predetermined color light is, for example, red light, blue light, and green light. The self-emitting elements include, for example, LED (Light Emitting Diode) elements or OLED (Organic LED) elements. The configuration of the image light forming unit 11 will be described later.

[0017] The image light forming unit 11 may be configured to include, for example, a light source having a lamp or a solid-state light source, and a light modulation device that modulates the light emitted by the light source. Examples of the lamp include a halogen lamp, a xenon lamp, and an ultra-high pressure mercury lamp. Examples of the solid-state light source include an LED and a laser light source. Examples of the light modulation device include a transmissive liquid crystal panel, a reflective liquid crystal panel, and a digital micromirror device (DMD).

[0018] The projection optical system 12 includes optical elements that guide the image light PL emitted by the image light formation unit 11 toward the projection target OB. The optical elements include a single lens or a lens group including multiple lenses. The optical elements may include a prism and a dichroic mirror. The optical elements may also be reflective optical elements such as mirrors.

[0019] The drive circuit 13 is connected to an image processing unit 43, which will be described later. The drive circuit 13 generates the image light PL by driving the image light formation unit 11 based on an image signal input from the image processing unit 43. For example, the drive circuit 13 causes the image light formation unit 11 to form an image on a frame-by-frame basis.

[0020] The projector 1 includes a capturing unit 15. The capturing unit 15 is a digital camera with an image sensor. The capturing unit 15 captures images under the control of a PJ control unit 20, which will be described later, and outputs the captured image to the PJ control unit 20. The capturing range of the capturing unit 15 includes the direction in which the projection unit 10 projects the image light PL. For example, the capturing range of the capturing unit 15 includes the projection target OB. The image sensor included in the capturing unit 15 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The configuration of the capturing unit 15 will be described later together with the projection unit 10.

[0021] The projector 1 includes a projector control unit 20, an operation unit 31, a remote control receiver 32, an input interface 33, a connection unit 41, a projector communication unit 42, and an image processing unit 43. In the following description and drawings, the projector may be abbreviated as PJ. For example, the projector control unit 20 will be referred to as the PJ control unit 20, and the projector communication unit 42 will be referred to as the PJ communication unit 42. The PJ control unit 20, the input interface 33, the connection unit 41, the PJ communication unit 42, and the image processing unit 43 are connected to each other via a bus 39 so that they can communicate data with each other.

[0022] The operation unit 31 includes various buttons and switches provided on the surface of the housing of the projector 1. The operation unit 31 generates operation signals corresponding to the operation of the buttons or switches, and outputs them to the input interface 33. The input interface 33 includes a circuit that outputs the operation signals input from the operation unit 31 to the PJ control unit 20.

[0023] The remote control light receiving unit 32 has a light receiving element that receives infrared light, and receives an infrared signal transmitted from the remote control 2. When a switch (not shown) provided on the remote control 2 is operated, the remote control 2 transmits an infrared signal indicating the operation. The remote control light receiving unit 32 decodes the received infrared signal to generate an operation signal. The remote control light receiving unit 32 outputs the generated operation signal to the input interface 33. The input interface 33 has a circuit that outputs the operation signal input from the remote control light receiving unit 32 to the PJ control unit 20.

[0024] There are no limitations on the specific manner in which signals are transmitted and received between the remote control 2 and the remote control light receiving unit 32. The configuration in which the remote control 2 transmits an infrared signal to the remote control light receiving unit 32 is one example. For example, the remote control 2 and the remote control light receiving unit 32 may transmit and receive signals by performing short-range wireless communication such as Bluetooth.

[0025] The connection unit 41 is an interface device that receives image data from an external device, and is connected to, for example, a player that plays back an optical disc type recording medium or a personal computer.

[0026] The PJ communication unit 42 is connected to the communication network 3 and transmits and receives image data to and from the display device 6 via the communication network 3. The PJ communication unit 42 is a communication device that includes, for example, a connector for connecting a communication cable and a communication circuit for inputting and outputting signals via the communication cable. The PJ communication unit 42 may also be a wireless communication device. In this case, the PJ communication unit 42 includes, for example, an antenna, an RF (Radio Frequency) circuit, a baseband circuit, etc.

[0027] The image processing unit 43 selects an image source under the control of the PJ control unit 20. Sources that the projector 1 can use are, for example, image data received by the connection unit 41 and image data received by the PJ communication unit 42.

[0028] The image processing unit 43 performs image processing on the image data of the selected source under the control of the PJ control unit 20. The image processing performed by the image processing unit 43 includes, for example, resolution conversion processing, geometric correction processing, digital zoom processing, and image correction processing for adjusting the color tone and brightness of the image.

[0029] The image processing unit 43 generates an image signal based on the image data after image processing and outputs it to the drive circuit 13. A frame memory (not shown) may be connected to the image processing unit 43. In this case, the image processing unit 43 loads the image data acquired from the source into the frame memory. The image processing unit 43 performs image processing on the image data loaded into the frame memory.

[0030] The image processing unit 43 can be configured, for example, by an integrated circuit. The integrated circuit is configured, for example, by an LSI (Large Scale Integration). More specifically, the image processing unit 43 is configured by an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or the like. The PLD includes, for example, an FPGA (Field-Programmable Gate Array). Furthermore, an analog circuit may be included as part of the configuration of the integrated circuit, or the integrated circuit may be a combination of a processor and an integrated circuit. The combination of a processor and an integrated circuit is called a microcontroller (MCU), a SoC (System-on-a-Chip), a system LSI, a chipset, or the like.

[0031] The PJ control unit 20 includes a processor 21 and a memory 25. The memory 25 is a storage device that stores programs and data executed by the processor 21 in a nonvolatile manner. The memory 25 is configured from a magnetic storage device, a semiconductor storage element such as a flash ROM (Read Only Memory), or other types of nonvolatile storage device. The memory 25 may include a RAM (Random Access Memory) that configures the work area of ​​the processor 21. The memory 25 stores data processed by the processor 21 and a control program 26 executed by the processor 21.

[0032] The processor 21 is configured with a CPU (Central Processing Unit), an MPU (Micro-processing unit), or the like. The processor 21 may be configured with a single processor, or multiple processors functioning as the processor 21. The processor 21 may be configured as an SoC integrated with part or all of the memory 25 and / or other circuits. As described above, the processor 21 may be configured by combining a CPU that executes programs and a DSP (Digital Signal Processor) that executes predetermined arithmetic processing. All of the functions of the processor 21 may be implemented in hardware, or may be configured using a programmable device. The processor 21 may also have the functions of the image processing unit 43. In other words, the functions of the image processing unit 43 may be executed by the processor 21.

[0033] The processor 21 controls each part of the projector 1 by executing a control program 26 stored in the memory 25 .

[0034] The processor 21 causes the image processing unit 43 to select a source and causes the image processing unit 43 to acquire image data of the selected source. The processor 21 controls the drive circuit 13 to cause the projection unit 10 to project image light PL and display an image based on the image signal output by the image processing unit 43. In the operation of the projector 1 described below, the processor 21 causes the projection unit 10 to project an image received from the display device 6 by the PJ communication unit 42.

[0035] The processor 21 controls the image capturing unit 15 to capture an image of an area including the projection target OB. The processor 21 transmits the image captured by the image capturing unit 15 to the display device 6 via the PJ communication unit .

[0036] The display device 6 includes a display 61, a touch sensor 62, a display communication unit 63, and a display control unit 70. In the following description and drawings, the display may be abbreviated as DP. For example, the display communication unit 63 will be referred to as the DP communication unit 63, and the display control unit 70 will be referred to as the DP control unit 70.

[0037] The display 61 displays an image under the control of the DP control unit 70. The display 61 includes, for example, a liquid crystal display panel, an organic EL display panel, or another display panel.

[0038] The touch sensor 62 detects operations on the display panel of the display 61. The touch sensor 62 detects contact operations or pressure operations on the display 61 and outputs a signal indicating the operation position to the DP control unit 70. The touch sensor 62 is configured, for example, as a pressure-sensitive sensor, a resistive sensor, or a capacitance sensor. The touch sensor 62 may also be configured to detect operations by performing wireless communication with the indicator 65. The touch sensor 62 may be configured to detect operations at one position on the display panel of the display 61, or may be configured to be able to simultaneously detect operations at multiple positions on the display panel. The touch sensor 62 corresponds to an example of an operation unit.

[0039] The DP communication unit 63 is connected to the communication network 3 and transmits and receives data to and from the projector 1 via the communication network 3. The DP communication unit 63 is, for example, a communication device including a connector for connecting a communication cable and a communication circuit for inputting and outputting signals via the communication cable. The DP communication unit 63 may also be a wireless communication device. In this case, the DP communication unit 63 includes, for example, an antenna, an RF circuit, a baseband circuit, etc.

[0040] The DP control unit 70 includes a processor 71 and a memory 75. The memory 75 is a storage device that stores programs and data executed by the processor 71 in a nonvolatile manner. The memory 75 is configured from a magnetic storage device, a semiconductor storage element such as a flash ROM, or other types of nonvolatile storage device. The memory 75 may include RAM that configures the work area of ​​the processor 71. The memory 75 stores data processed by the processor 71 and a control program 76 executed by the processor 71.

[0041] The processor 71 is configured with a CPU, an MPU, or the like. The processor 71 may be configured with a single processor, or multiple processors may function as the processor 71. The processor 71 may be configured as an SoC integrated with part or all of the memory 75 and / or other circuits. As described above, the processor 71 may also be configured with a combination of a CPU that executes programs and a DSP that executes predetermined arithmetic processing. All of the functions of the processor 71 may be implemented in hardware, or may be configured using a programmable device.

[0042] The processor 71 controls each part of the display device 6 by executing a control program 76 stored in the memory 75. The processor 71 includes a display control unit 72 and a filter 73. The display control unit 72 and the filter 73 are functional units configured through cooperation of hardware and software as a result of the processor 71 executing the control program 76. The memory 75 includes a drawing image memory 77. The drawing image memory 77 is a logical or virtual storage area that uses part of the storage area of ​​the memory 75.

[0043] The display control unit 72 causes the DP communication unit 63 to receive the captured image transmitted by the projector 1. The display control unit 72 causes the display 61 to display the captured image received by the DP communication unit 63.

[0044] The display control unit 72 receives an operation detected by the touch sensor 62. The display control unit 72 generates a drawn image based on the received operation. The display control unit 72 stores the generated drawn image in the drawn image memory 77.

[0045] The display control unit 72 displays the image stored in the drawn image memory 77 on the display 61. The display control unit 72 also generates a transmission image including the image stored in the drawn image memory 77, and transmits the transmission image to the projector 1 via the DP communication unit 63.

[0046] When the processor 71 receives a captured image from the projector 1 while the display control unit 72 is displaying an image stored in the drawn image memory 77 on the display 61, the processor 71 executes processing by the filter 73. The filter 73 filters the image stored in the drawn image memory 77 based on the captured image received from the projector 1. In detail, the filter 73 performs processing to remove, from the images stored in the drawn image memory 77, any image included in the captured image received from the projector 1. The display control unit 72 causes the display 61 to display the image after processing by the filter 73.

[0047] [3. Projection and shooting unit configuration] FIG. 3 is a schematic diagram showing an example of the configuration of the projection unit 10 and the image capture unit 15. As shown in FIG. The projection optical system 12 includes a separation optical element 121 and a projection lens 122. The optical axis of the image light PL projected by the projection lens 122 toward the projection target OB is indicated by the symbol AX. The optical axis AX is the central axis of the image light PL irradiated from the projection lens 122 onto the projection target OB, and is a virtual axis that passes through the optical center of the projection lens 122 and follows the direction in which the image light PL is irradiated from the projection lens 122.

[0048] The image light forming unit 11 includes a light emitting device 111. The light emitting device 111 has light emitting elements arranged in a row on a light emitting surface 112. The light emitting elements arranged on the light emitting surface 112 include a light emitting element that emits red light, a light emitting element that emits blue light, and a light emitting element that emits green light. By arranging these light emitting elements in a matrix, the light emitting device 111 emits image light PL that forms an image from the light emitting surface 112.

[0049] The light-emitting surface 112 faces the separation optical element 121. The image light PL emitted by the light-emitting surface 112 enters the separation optical element 121 along the optical axis AX, passes through the separation optical element 121, and enters the projection lens 122. The projection lens 122 irradiates the image light PL that has passed through the separation optical element 121 onto the projection target OB. The optical axis of the image light PL emitted by the light-emitting surface 112 is defined as the projection optical axis PAX. The projection optical axis PAX is the central axis of the image light PL emitted by the light-emitting device 111, and is a virtual axis that passes through the center of the area in which the light-emitting elements are arranged on the light-emitting surface 112 and is perpendicular to the light-emitting surface 112. In the configuration of FIG. 3 , the projection optical axis PAX coincides with the optical axis AX. In other words, the light-emitting device 111 is arranged on the optical axis AX of the projection optical system 12.

[0050] The photographing unit 15 includes an imaging device 151. The imaging device 151 is disposed opposite the separation optical element 121. In the imaging device 151, imaging elements are disposed side by side on an imaging surface 152 facing the separation optical element 121. The photographing unit 15 photographs an image by each imaging element disposed on the imaging surface 152 receiving light incident from the separation optical element 121. The imaging device 151 faces a surface of the separation optical element 121 different from the surface of the light-emitting device 111. In detail, the light-emitting device 111 is disposed so as to be aligned with the separation optical element 121 in a direction along the optical axis AX. In contrast, the imaging device 151 faces the separation optical element 121 at an angle of 90 degrees with respect to the optical axis AX.

[0051] The separation optical element 121 transmits light emitted by the light emitting device 111 and causes it to enter the projection lens 122, while reflecting light that has entered the separation optical element 121 from the projection lens 122 toward the imaging device 151. The separation optical element 121 may be, for example, a polarization separation element. The separation optical element 121 may also be configured with a dichroic mirror or a dichroic prism.

[0052] The optical axis of light reflected by the separation optical element 121 toward the imaging device 151 is indicated by the symbol IAX. The imaging optical axis IAX is the central axis of light traveling from the separation optical element 121 toward the imaging device 151, and is the axis of light received by the imaging device 151 on the imaging surface 152. The imaging optical axis IAX is a virtual axis perpendicular to the imaging surface 152. In other words, the imaging device 151 is positioned so that the center of the imaging surface 152 coincides with the imaging optical axis IAX.

[0053] The photographing optical axis IAX coincides with the optical axis AX until it is reflected inside the separation optical element 121 . That is, in an area closer to the projection target OB than the projection optical system 12, the projection optical axis PAX of the image light PL emitted by the image light formation unit 11 coincides with the imaging optical axis IAX of the light received by the imaging unit 15. In this way, the projection unit 10 and the imaging unit 15 are optically arranged on the same axis. That is, the projection unit 10 and the imaging unit 15 project and capture the image light PL on the same axis.

[0054] 3 is a schematic diagram, and the projection unit 10, the projection optical system 12, and the imaging unit 15 may include components not shown in Fig. 3. For example, the projection optical system 12 may include optical elements other than the separation optical element 121 and the projection lens 122. For example, the projection optical system 12 may include a light guiding element between the separation optical element 121 and the projection lens 122. A polarization separation element and a polarization conversion element may be provided between the light emitting device 111 and the separation optical element 121 to adjust the polarization of the image light PL incident on the separation optical element 121.

[0055] FIG. 4 is a schematic diagram showing another example of the configuration of the projection unit 10 and the image capture unit 15. In FIG. This configuration example uses a light receiving and emitting device 14 that integrates the projection unit 10 and the image capturing unit 15. In the example of Fig. 4, the projection optical system 12 does not have a separation optical element 121. The projection lens 122 in Fig. 4 may be the same as the projection lens 122 illustrated in Fig. 3, or may have a different configuration.

[0056] The light emitting and receiving device 14 is disposed on the optical axis AX of the projection lens 122. The light emitting and receiving device 14 includes a light emitting element and a light receiving element on a light emitting and receiving surface 14a facing the projection lens 122. That is, as shown enlarged in circle A in Fig. 4, a blue light emitting element 141, a red light emitting element 142, a green light emitting element 143, and an imaging element 145 are disposed on the light emitting and receiving surface 14a.

[0057] The blue light-emitting element 141, the red light-emitting element 142, and the green light-emitting element 143 are configured, for example, by LEDs or OLEDs. The blue light-emitting element 141 is an element that emits light in a blue wavelength region, the red light-emitting element 142 is an element that emits light in a red wavelength region, and the green light-emitting element 143 is an element that emits light in a green wavelength region. In the example of FIG. 4, one pixel region 140 is configured by two blue light-emitting elements 141, one red light-emitting element 142, and one green light-emitting element 143. The pixel region 140 is an area that forms one pixel included in an image formed by the light-receiving and light-emitting device 14. The pixel region 140 forms the color of one pixel by two blue light-emitting elements 141, one red light-emitting element 142, and one green light-emitting element 143.

[0058] The light receiving and emitting surface 14a includes one image sensor 145 per pixel region 140. The image sensor 145 is an element made of a CMOS or a CCD, and receives light incident on the light receiving and emitting surface 14a. The photographing unit 15 photographs an image by receiving light with the image sensor 145.

[0059] In this way, the light receiving and emitting device 14 functions as an image light forming unit 11 that forms image light PL, and as an imaging unit 15 that captures images. The optical axis of the image light PL emitted by the light receiving and emitting device 14 is optical axis AX, and the imaging unit 15 captures images using light that is incident along the optical axis AX. In the configuration of Fig. 4, similar to the configuration shown in Fig. 3, the projection unit 10 and the imaging unit 15 are optically arranged on the same axis, and the projection unit 10 and the imaging unit 15 project and capture images along the same axis.

[0060] [4. Projection System Operation] 5 and 6 are sequence diagrams showing the operation of the projection system 100. Fig. 7 is a schematic diagram showing the operation of the projection system 100. The operation of the projection system 100 will be described with reference to these figures.

[0061] 5 and 6, the processes of steps SA11 to SA14 and SA21 to SA25 are executed by the PJ control unit 20, and the processes of steps SB11 to SB17 and SB21 to SB30 are executed by the DP control unit .

[0062] In step SA11, the projector 1 captures an image of an area including the projection target OB. In step SA12, the projector 1 transmits the captured image to the display device 6.

[0063] In step SB11, the display device 6 receives the captured image transmitted by the projector 1. In step SB12, the display device 6 displays the received captured image on the display 61.

[0064] Here, when the display device 6 detects an operation by the indicator 65, it accepts this operation in step SB13. In step SB14, the display device 6 generates a drawn image based on the operation accepted in step SB13. For example, the display device 6 draws curves and straight lines along the trajectory of the movement of the indicator 65.

[0065] In step SB15, the display device 6 stores the drawn image generated in step SB14 in the drawn image memory 77.

[0066] In step SB16, the display device 6 displays the image stored in the drawn image memory 77 on the display 61. Specifically, the display device 6 generates a composite image by overlaying the image stored in the drawn image memory 77 on the captured image displayed in step SB12, and displays the composite image on the display 61.

[0067] In step SB17, the display device 6 generates a transmission image including the image stored in the drawn image memory 77, transmits the transmission image to the projector 1, and then proceeds to step SB21. In step SA13, the projector 1 receives the image transmitted by the display device 6. In step SA14, the projector 1 projects the received image onto the projection target OB using the projection unit 10, and then proceeds to step SA21.

[0068] 5 show operations from when the projection system 100 starts operating until the display device 6 first transmits a transmission image to the projector 1. In contrast, steps SA21 to SA25 and steps SB21 to SB30 in FIG. 6 show operations of the projection system 100 after the display device 6 first transmits a transmission image to the projector 1.

[0069] In step SA21, the projector 1 captures an image of an area including the projection target OB. In step SA21, the projector 1 executes capture using the capture unit 15 while the projection unit 10 is projecting an image based on the transmitted image. Therefore, the image captured by the capture unit 15 shows the image being projected by the projection unit 10. In step SA22, the projector 1 transmits the captured image to the display device 6.

[0070] In step SB21, the display device 6 receives the captured image transmitted by the projector 1. In step SB22, the display device 6 acquires the image stored in the drawn image memory 77. In detail, the display device 6 acquires the image stored in the drawn image memory 77 as the object to be processed by the filter 73.

[0071] The display device 6 performs filtering on the image acquired in step SB22 based on the captured image received in step SB21 using the filter 73. In this filtering, images that overlap with the captured image are removed from the images stored in the drawn image memory 77.

[0072] The filter 73, for example, superimposes the image stored in the drawn image memory 77 on the photographed image, erases the overlapping portion between the image stored in the drawn image memory 77 and the photographed image, and outputs the result.

[0073] Furthermore, for example, the filter 73 may perform processing to recognize image objects included in the image stored in the drawing image memory 77 and image objects included in the captured image. An image object refers to a group of images such as figures and characters. For example, the filter 73 removes the background from the image stored in the drawing image memory 77 and extracts image objects included in the image after the background has been removed. The filter 73 also removes the background from the captured image and extracts image objects included in the image after the background has been removed. The filter 73 compares the image objects extracted from the image stored in the drawing image memory 77 with the image objects extracted from the captured image and identifies matching image objects. The filter 73 removes the identified image objects from the image stored in the drawing image memory 77 and outputs the image after the removal.

[0074] In this embodiment, the filter 73 performs filtering on the image obtained from the drawing image memory 77 in step SB22. Therefore, even if the filter 73 performs filtering, the image stored in the drawing image memory 77 is not altered. As another example, the image output by the filter 73 after performing filtering may be stored in the drawing image memory 77. In this case, the image stored in the drawing image memory 77 is updated based on the results of the filtering by the filter 73.

[0075] The images stored in the drawn image memory 77 and the images that overlap with the captured image include images that completely match in size, position, and shape. The filter 73 may also remove from the images stored in the drawn image memory 77 any images that differ from the images included in the captured image in any of the size, position, and shape. In this case, the DP control unit 70 may have a threshold that defines an acceptable range for differences in image size, position, and shape. For example, the filter 73 may determine that the images overlap if the amount of difference in image size, position, and shape is smaller than a threshold. The filter 73 may also perform image removal processing based on whether the images' colors match.

[0076] In step SB24, the display device 6 displays the image output by the filter 73 and the captured image received in step SB21 on the display 61. More specifically, in step SB24, the display device 6 updates the image already displayed on the display 61.

[0077] When the display device 6 detects an operation by the indicator 65, it accepts this operation in step SB25. In step SB26, the display device 6 generates a drawn image based on the operation accepted in step SB25. In step SB27, the display device 6 updates the image stored in the drawn image memory 77 based on the drawn image generated in step SB26. For example, the display device 6 overlays the drawn image generated in step SB26 on the image stored in the drawn image memory 77 to composite them, and stores the composite image in the drawn image memory 77.

[0078] In step SB28, the display device 6 updates the display on the display 61 based on the image stored in the updated drawn image memory 77. In step SB28, the display device 6 may perform filtering similar to that in step SB23 using the filter 73, and display the image output by the filter 73 after the filtering process on the display 61. The display device 6 may also add the image object newly stored in the drawn image memory 77 in step SB27 to the display 61 and display it.

[0079] In step SB29, the display device 6 generates a transmission image including the image stored in the drawn image memory 77, and transmits the transmission image to the projector 1.

[0080] In step SA23, the projector 1 receives the image transmitted by the display device 6. In step SA24, the projector 1 projects the received image onto the projection target OB by the projection unit 10.

[0081] In step SA25, the projector 1 determines whether or not to end the operation. If the projector 1 determines to end the operation (step SA25; YES), such as when an operation instructing to end the operation is detected by the input interface 33, the projector 1 ends this processing. If the projector 1 determines not to end the operation (step SA25; NO), the projector 1 returns to step SA21.

[0082] After transmitting the transmission image to the projector 1, the display device 6 determines in step SB30 whether or not to end the operation. If the display device 6 determines to end the operation (step SB30; YES), such as when an operation instructing to end the operation is detected by the touch sensor 62, the display device 6 ends this process. If the display device 6 determines not to end the operation (step SB30; NO), the process returns to step SB21.

[0083] FIG. 7 shows the operating states ST1, ST2, ST3, and ST4 of the projection system 100. An object OB2 is placed on a plane OB1 as a projection target OB of the projector 1. Paper or the like is placed on the plane OB1, and a user of the projector 1 can write characters or figures on the plane OB1 with a writing implement.

[0084] In state ST1, the projector 1 transmits a captured image P1 of a plane OB1 and an object OB2 to the display device 6. The captured image P1 includes an object image OP1, which is a captured image of the object OB2. The display device 6 receives the captured image P1 and displays the captured image P1 on the display 61. The object image OP1 is displayed on the display 61.

[0085] State ST2 is a state in which an operation is performed on the display device 6 using the indicator 65. The display device 6 draws a drawn image DP1 based on the operation of the indicator 65. As described with reference to FIG. 5, the display device 6 stores the drawn image DP1 in the drawn image memory 77. The display device 6 generates a transmitted image P2 including the drawn image DP1 stored in the drawn image memory 77. The display device 6 transmits the transmitted image P2 to the projector 1, and the projector 1 receives the transmitted image P2 and projects the transmitted image P2 onto the projection target OB.

[0086] State ST3 is a state in which a figure is written on flat surface OB1 with a writing implement in projector 1. Projector 1 transmits a captured image P3 of projection target OB to display device 6. Captured image P3 includes object image OP1, which is an image of object OB2, object image OP2, and object image OP3. Object image OP2 is an image captured by capture unit 15 of drawn image DP1 projected by projection unit 10 onto projection target OB. Object image OP3 is an image captured by capture unit 15 of the figure written on flat surface OB1 with a writing implement in projector 1.

[0087] The display device 6 receives the photographed image P3. Here, the display device 6 performs filtering using the filter 73. In state ST3, the drawn image memory 77 stores the drawn image DP1. The filter 73 filters the drawn image DP1 stored in the drawn image memory 77 based on the photographed image P3. That is, from the images stored in the drawn image memory 77, the filter 73 removes images that overlap with the object images OP1, OP2, and OP3 included in the photographed image P3. Through this processing, the filter 73 outputs an image stored in the drawn image memory 77, excluding the drawn image DP1. The display device 6 then displays the image output by the filter 73 and the photographed image P3 on the display 61, superimposed on each other. As a result of these processing steps, the object images OP1, OP2, and OP3 included in the photographed image P3 are displayed on the display 61. The object image OP2 is identical to the drawn image DP1 displayed on the display 61 in state ST2. However, the drawn image DP1 displayed in state ST2 is an image stored in the drawn image memory 77, and the object image OP2 displayed in state ST3 is an image included in the photographed image P3.

[0088] The drawn image DP1 and the object image OP2 are not necessarily completely identical. The object image OP2 is an image projected by the projection unit 10 onto the projection target OB based on the transmitted image P2, and then captured by the capture unit 15. For this reason, there is a possibility that the object image OP2 will not be identical to the drawn image DP1 due to various factors. The factors mentioned above include uneven brightness of the plane OB1 due to ambient light, the shadow of the object OB2 created by ambient light, the color of the plane OB1, and the unevenness of the plane OB1.

[0089] If the drawn image DP1 and the object image OP2 do not perfectly match, and the drawn image DP1 and the object image OP2 are displayed overlapping on the display 61, the display on the display 61 may be distorted. For example, the lines constituting the object image OP2 and the lines constituting the drawn image DP1 may be displayed as double lines. Also, for example, the lines constituting the object image OP2 and the lines constituting the drawn image DP1 may be displayed as interference fringes. These displays result in a decrease in display quality and are a factor in reducing the visibility of the object image OP2 and the drawn image DP1.

[0090] 3 and 4, in particular, when the projection optical axis PAX of the projection unit 10 and the imaging optical axis IAX of the imaging unit 15 are configured to be the same axis, i.e., coaxial, the difference in position and size between the object image OP2 and the drawn image DP1 is very small, which can easily cause a phenomenon in which the lines that make up the object image OP2 and the lines that make up the drawn image DP1 are displayed like interference fringes.

[0091] In the projection system 100 of this embodiment, the display device 6 filters the image stored in the drawn image memory 77 using the filter 73 based on the captured image of the projector 1. For example, if the display device 6 receives a captured image P3 including an object image OP2 corresponding to the drawn image DP1 while displaying the drawn image DP1 on the display 61, the display device 6 displays the captured image P3 and stops displaying the drawn image DP1. This prevents the drawn image DP1 and the object image OP2 from being displayed overlapping each other, preventing a decrease in the display quality on the display 61.

[0092] State ST4 is a state in which an operation is performed on the display device 6 using the indicator 65. The display device 6 draws a drawn image DP2 based on the operation of the indicator 65. The display device 6 updates the image in the drawn image memory 77 based on the drawn image DP2. As a result, a composite image obtained by combining the drawn images DP1 and DP2 is stored in the drawn image memory 77. The display device 6 transmits a transmitted image P4 including the image stored in the updated drawn image memory 77 to the projector 1. The transmitted image P4 includes the drawn image DP1 and the drawn image DP2. The projector 1 receives the transmitted image P4 and projects the transmitted image P4 onto the projection target OB using the projection unit 10.

[0093] [5. Operation of the embodiment] As described above, the projection system 100 described in the embodiment includes the projector 1 and the display device 6. The projector 1 includes a PJ communication unit 42 that communicates with the display device 6, a projection unit 10 that projects an image onto the projection target OB, and a capture unit 15 that captures an image of an area including the projection target OB. The projector 1 projects an image received by the PJ communication unit 42 using the projection unit 10, and transmits an image captured by the capture unit 15 via the PJ communication unit 42. The display device 6 includes a DP communication unit 63 that communicates with the projector 1, a display 61 that displays an image, and a touch sensor 62 that accepts a drawing operation. The display device 6 generates a drawn image based on the drawing operation accepted by the touch sensor 62, and displays the drawn image on the display 61. The display device 6 transmits the drawn image via the DP communication unit 63 and receives the captured image via the DP communication unit 63. The display device 6 performs processing to remove images that overlap with the captured image received from the projector 1 from the drawn image displayed on the display 61, and updates the display on the display 61 based on the processed image and the captured image.

[0094] In the control method of the projection system 100, the projector 1 receives an image from the display device 6, projects the received image onto the projection target OB, captures an image of an area including the projection target OB, and transmits the captured image to the display device 6. In this control method, the display device 6 generates a drawn image based on a drawing operation, displays the drawn image on the display 61, transmits the drawn image to the projector 1, and receives the captured image from the projector 1. In addition, the display device 6 performs processing to remove images that overlap with the captured image received from the projector 1 from the drawn image displayed on the display 61, and updates the display on the display 61 based on the processed image and the captured image.

[0095] This allows the state of the projection target OB of the projector 1 to be displayed by the display device 6, and allows the projector 1 to project an image drawn based on the operation of the display device 6. Therefore, the user using the projector 1 and the user using the display device 6 can share the state of the projection target OB of the projector 1 and the content of the drawing operation on the display device 6.

[0096] The display device 6 performs processing using the filter 73 to remove images that overlap with the captured image received from the projector 1 from the drawn image. This makes it possible to use the filter 73 to eliminate the phenomenon in which the captured image received by the display device 6 from the projector 1 and the drawn image generated by the display device 6 overlap on the display 61. This makes it possible to more reliably prevent degradation of the display quality on the display 61.

[0097] The display device 6 includes a drawn image memory 77 that stores images. When the display device 6 generates a drawn image based on a drawing operation, the display device 6 stores the drawn image in the drawn image memory 77. The display device 6 performs processing to remove, using a filter 73, images that overlap with the captured image received from the projector 1 from the images stored in the drawn image memory 77, and displays the processed image and the captured image on the display 61. This allows a drawn image to be quickly displayed on the display 61 when a drawing operation is performed on the display device 6. Therefore, the filter 73 can be used to eliminate the phenomenon in which the captured image received by the display device 6 from the projector 1 and the drawn image generated by the display device 6 overlap on the display 61, and the operability of the drawing operation can be improved.

[0098] Every time a drawing image is generated based on a drawing operation, the display device 6 synthesizes the generated drawing image with the image stored in the drawing image memory 77, thereby updating the image stored in the drawing image memory 77. This allows the display device 6 to store multiple drawn images in the drawn image memory 77. Therefore, the display device 6 can transmit multiple drawn images generated based on the operation of the indicator 65 at different times to the projector 1. This prevents a decrease in display quality caused by the drawn image and the captured image corresponding to the drawn image overlapping, and allows the drawn images drawn through multiple operations to be shared between the projector 1 and the display device 6.

[0099] In the projector 1, the projection unit 10 includes an image light formation unit 11 that forms image light and a projection optical system 12 that projects the image light toward a projection target OB. The imaging unit 15 includes an imaging device 151 that receives light incident through the projection optical system 12. As a result, in a configuration in which a common projection optical system 12 is used for projection by the projection unit 10 and for capturing images by the capturing unit 15, the captured image by the projector 1 and the drawn image generated by the display device 6 can be displayed with good quality on the display 61. By using the projection optical system 12 that is common to projection and capturing images, the projector 1 can prevent a decrease in display quality on the display 61 due to the difference in images in a configuration in which small differences between the captured image and the drawn image are likely to occur.

[0100] 6. Other Embodiments The above-described embodiment is a preferred embodiment of the present invention, but the present invention is not limited to this and various modifications are possible within the scope of the gist of the present invention.

[0101] 3 and 4 are merely examples, and the configurations of the projection unit 10 and the image capturing unit 15 included in the projector 1 can be changed as desired. For example, the image capturing unit 15 may be configured as a separate entity from the projector 1.

[0102] The configuration of the projector 1 and the display device 6 shown in Fig. 2 shows a functional configuration, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each functional unit individually, and it is of course possible to implement a configuration in which a single processor executes a program to realize the functions of multiple functional units. Furthermore, some of the functions realized by software in the above embodiments and modifications may be realized by hardware, and some of the functions realized by hardware may be realized by software.

[0103] 5 and 6 are divided according to the main processing content in order to facilitate understanding of the operation of each device in projection system 100, and the present invention is not limited by the way the processing units are divided or the names of the processing units. The processing performed by each device can be divided into more processing units according to the processing content, or one processing unit can be divided so that it includes even more processes.

[0104] The control program 26 can be recorded, for example, on a recording medium that is readable by the projector 1. The recording medium can be a magnetic or optical recording medium or a semiconductor memory device. Specific examples include portable or fixed recording media such as flexible disks, optical disc recording media, magneto-optical disc recording media, and semiconductor storage devices. Alternatively, these programs can be stored in a server device or the like, and downloaded from the server device as needed. The same applies to the control program 76. [Explanation of symbols]

[0105] 1...projector, 2...remote control, 3...communication network, 10...projection unit, 11...image light formation unit, 12...projection optical system, 13...drive circuit, 15...photographing unit, 20...projector control unit, 21...processor, 25...memory, 26...control program, 31...operation unit, 32...remote control light receiving unit, 33...input interface, 39...bus, 41...connection unit, 42...projector communication unit, 43...image processing unit, 61...display, 62...touch sensor (operation unit), 63...display communication unit, 65...pointer, 70...display control unit, 71...processor, 72...display control unit, 73...filter, 75...memory, 76...control program, 77...drawing image memory, 100...projection system, OB...projection target

Claims

1. A projector and a display device are provided, The projector includes: a projector communication unit that communicates with the display device; a projection unit that projects an image onto a projection target; an imaging unit that images an area including the projection target, projecting the image received by the projector communication unit by the projection unit; The image captured by the image capturing unit is transmitted by the projector communication unit; The display device includes: a display communication unit that communicates with the projector; a display for displaying an image; an operation unit that accepts drawing operations, generating a drawing image based on the drawing operation received by the operation unit; Displaying the drawn image on the display; transmitting the drawn image by the display communication unit; receiving the captured image by the display communication unit; A projection system that processes the drawn image displayed on the display to remove any images that overlap with the captured image received from the projector, and updates the display based on the processed image and the captured image.

2. The projection system according to claim 1 , wherein the display device performs processing to remove, from the drawn image, an image that overlaps with the captured image received from the projector, using a filter.

3. The display device includes: A drawing image memory for storing an image is provided, When the drawing image is generated based on the drawing operation, the drawing image is stored in the drawing image memory; The projection system of claim 2, wherein the filter removes images stored in the drawing image memory that overlap with the captured image received from the projector, and the processed image and the captured image are displayed on the display.

4. 4. The projection system of claim 3, wherein the display device updates the image stored in the drawing image memory by combining the generated drawing image with the image stored in the drawing image memory each time the display device generates the drawing image based on the drawing operation.

5. the projection unit includes an image light forming unit that forms image light, and a projection optical system that projects the image light toward the projection target; The projection system according to claim 1 , wherein the image capturing section includes an image capturing device that receives light incident through the projection optical system.

6. A control method for a projection system including a projector and a display device, comprising: The projector receiving an image from the display device; The received image is projected onto the projection target, taking an image of an area including the projection target; Transmitting the captured image to the display device; The display device Generate a drawing image based on the drawing operations; Displaying the drawn image on a display; transmitting the drawn image to the projector; receiving the captured image from the projector; A control method for a projection system, which processes the drawn image displayed on the display to remove any images that overlap with the captured image received from the projector, and updates the display based on the processed image and the captured image.

Images