Method for controlling display system, method for controlling display unit, and method for controlling control device

Abstract

To provide a method for controlling a display system, a method for controlling a display unit, and a method for controlling a control device that can execute processing based on operation information on images input from a plurality of sources.SOLUTION: A method for controlling a display system includes: creating, by a control device 300, a first display image 500A in which a capture image 635 and an APP image 615 acquired by a projector 200 are arranged; displaying, by the projector 200, the first display image 500A; performing, by the control device 300, processing on the first display image 500A on the basis of operation information indicating an operation for the first display image 500A, to create a second display image 500B; and displaying, by the projector 200, the second display image 500B.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control method for a display system, a control method for a display device, and a control method for a control device. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known an apparatus that displays a plurality of images on a display surface and executes processing on the displayed plurality of images based on an operation on the display surface.

[0003] For example, Patent Document 1 discloses a display device including a display means, a position detection means, and a processing means. The display means displays input images supplied from multiple image sources in multiple display areas on a display surface. The position detection means detects a pointed position on the display surface. The processing means executes processing across the multiple display areas based on the detected pointed position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-97177 A Summary of the Invention [Problem to be solved by the invention]

[0005] There may be cases where multiple image sources are required to perform a process based on an operation on the display surface. However, not all of the multiple image sources are necessarily capable of performing this process, and there are cases where the process cannot be reflected in the images input from the multiple image sources. [Means for solving the problem]

[0006] The present disclosure is a control method for a display system including: a display device acquiring a first image; a control device generating a first display image in which the first image acquired by the display device and a second image are arranged; the display device displaying the first display image; the control device processing the first display image based on operation information indicating an operation on the first display image to generate a second display image; and the display device displaying the second display image.

[0007] The present disclosure is a method for controlling a display device, including acquiring a first image and transmitting the acquired first image to a control device, receiving from the control device a first display image in which the first image and a second image are arranged, displaying the first display image, receiving from the control device a second display image in which processing has been performed on the first display image based on operation information indicating an operation on the first display image, and displaying the second display image.

[0008] The present disclosure is a control method for a control device, including generating a first display image in which a first image received from a display device and a second image are arranged, transmitting the first display image to the display device, generating a second display image by processing the first display image based on operation information indicating an operation on the first display image, and transmitting the second display image to the display device. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a system configuration of a display system. [Diagram 2] FIG. 1 is a block diagram showing the configuration of a projector. [Diagram 3] FIG. [Figure 4] FIG. 4 shows a first display image. [Diagram 5] FIG. 4 shows a second display image. [Figure 6] FIG. 11 is a diagram showing an operation for changing the display position of the first screen. [Figure 7]4 is a flowchart showing the operation of the control device. [Figure 8] 4 is a flowchart showing the operation of the control device. [Figure 9] 4 is a flowchart showing the operation of the control device. [Figure 10] 4 is a flowchart showing the operation of the projector. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [1. System configuration of the embodiment] FIG. 1 is a diagram showing a system configuration of a display system 1. As shown in FIG. The display system 1 includes an image supply device 100, a projector 200, and a control device 300. The projector 200 is an example of a display device.

[0012] FIG. 1 shows a configuration in which one image supply device 100 is connected to the projector 200, but the number of image supply devices 100 connected to the projector 200 is not limited to one, and a configuration in which two or more image supply devices 100 are connected may also be used.

[0013] The image supply device 100 is a device that supplies image data to the projector 200. For example, the image supply device 100 may be a notebook or desktop personal computer, or a smart device such as a smartphone, a smart stick, or a tablet terminal. The image supply device 100 may also be a recorder such as a DVD (Digital Versatile Disc), a BD (Blu-ray Disc), or a HDD (hard disk drive), or various STBs (Set Top Boxes), etc.

[0014] The projector 200 is a device that displays an image on the projection surface 30. The projector 200 also detects an operation on the projection surface 30 performed using a pointer 50. The projector 200 transmits operation information indicating the detected operation to the control device 300.

[0015] The projector 200 includes a plurality of image interfaces used for communicating image data, and a plurality of data interfaces used for communicating image data and control data. Hereinafter, the interface is also written as I / F. In this embodiment, a case will be described in which the projector 200 includes an HDMI I / F 210 as an image I / F and a USB I / F 220 as a data I / F. The image I / F included in the projector 200 may be, for example, a display port, and the data I / F included in the projector 200 may be, for example, Ethernet. HDMI and Ethernet are registered trademarks. Hereinafter, the image I / F and the data I / F will be collectively referred to as an input I / F.

[0016] FIG. 1 shows a case where the projector 200 includes a first HDMI I / F 211, a second HDMI I / F 213, and a third HDMI I / F 215 as the HDMI I / F 210. FIG. 1 also shows a case in which the projector 200 includes a first USB I / F 221 and a second USB I / F 223 as the USB I / F 220.

[0017] In this embodiment, a case will be described in which the image supply device 100 is connected to the first HDMI I / F 211 and the control device 300 is connected to the second HDMI I / F 213. Also, in this embodiment, a case will be described in which the control device 300 is connected to the first USB I / F 221.

[0018] The control device 300 can be a smart device such as a smartphone, a smart stick, or a tablet terminal. The control device 300 generates a display image 500 to be displayed on the projector 200. The display image 500 includes an image based on image data supplied to the projector 200 by the image supply device 100, and an image generated by an application program executed by the control device 300. The display image 500 includes a first display image 500A shown in Fig. 4 and Fig. 6, and a second display image 500B shown in Fig. 5.

[0019] Furthermore, operation information is input to the control device 300 from the projector 200. The control device 300 generates an object image to be superimposed on the display image 500 based on the operation information input from the projector 200. The control device 300 superimposes the object image on the display image. The control device 300 transmits the display image 500 on which the object image is superimposed to the projector 200.

[0020] In this embodiment, the HDMI I / F 301 is connected to the second HDMI I / F 213 of the projector 200 via a cable 5, and the USB I / F 303 is connected to the first USB I / F 221 of the projector 200 via a cable 7. The first USB I / F 221 corresponds to the first interface.

[0021] When the USB I / F 303 of the control device 300 and the first USB I / F 221 of the projector 200 are connected by the cable 7, the control device 300 causes the projector 200 to be recognized as a USB video class device. As a result, the projector 200 transmits a capture image 635 (described later) in a format defined by the USB video class. The capture image 635 corresponds to the first image.

[0022] In addition, when the HDMI I / F 301 has a male connector and the second HDMI I / F 213 has a female connector, the male connector of the HDMI I / F 301 may be directly connected to the female connector of the second HDMI I / F 213.

[0023] [2. Projector configuration] FIG. 2 is a block diagram showing the configuration of the projector 200. The configuration of the projector 200 will be described with reference to FIG. For simplicity of illustration, FIG. 2 shows the first HDMI I / F 211 and the second HDMI I / F 213 as the image I / F, and shows only the first USB I / F 221 as the data I / F.

[0024] The projector 200 includes an HDMI I / F 210 , a USB I / F 220 , a transmission unit 231 , an imaging unit 233 , an image processing unit 240 , a frame memory 245 , an image projection unit 250 , and a first control unit 270 .

[0025] The HDMI I / F 210 includes a connection port and an interface circuit that are compatible with the HDMI standard. The USBI / F 220 includes a connection port and an interface circuit that are compatible with the USB standard.

[0026] The transmitter 231 includes, for example, a light source such as an LED (Light Emitting Diode) and a device for controlling the turning on and off of the light source. The control device can be configured, for example, with an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0027] The transmission unit 231 outputs a first signal light 235 which is a signal for synchronizing the light emission timing of the indicator 50 with the imaging timing of the imaging unit 233. The first signal light 235 is a near-infrared light signal which can be received by the indicator 50. The transmission unit 231 periodically transmits the first signal light 235 of near-infrared light while the projector 200 is activated.

[0028] The first signal light 235 is a control signal that instructs the indicator 50 on the timing to transmit the second signal light 237. The indicator 50 includes a light source such as an LED and a device that controls turning the light source on and off, and emits the second signal light 237 in a preset emission pattern based on the timing of receiving the first signal light 235. The projector 200 causes the imaging unit 233 to capture an image in accordance with the timing at which the indicator 50 emits the second signal light 237.

[0029] The indicator 50 has a switch at the tip of the indicator 50. This switch is turned on when the tip is in contact with the projection surface 30, and is turned off when the tip is separated from the projection surface 30. The indicator 50 emits the second signal light 237 in a different emission pattern depending on whether the switch is in the on state or the off state.

[0030] The imaging unit 233 is a camera equipped with an imaging element (not shown), such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 233 has an infrared transmission filter that absorbs visible light and transmits infrared light, and captures an image of infrared light emitted from the indicator 50 through the infrared transmission filter. The imaging unit 233 captures an image of the projection surface 30 at a preset imaging timing based on the light emission timing at which the transmission unit 231 emits the first signal light 235. The imaging unit 233 sequentially outputs the captured images to the first control unit 270.

[0031] The first control unit 270 analyzes the captured image input from the imaging unit 233 and detects the emission of the second signal light 237. Moreover, the indicator 50 transmits the second signal light 237 in a different emission pattern depending on whether the switch is on or off, based on the timing of receiving the first signal light 235. For this reason, the first control unit 270 analyzes a plurality of consecutive captured images to determine whether the detected emission of the second signal light 237 is an emission when the indicator 50 is in contact with the projection surface 30 or an emission when the indicator 50 is not in contact with the projection surface 30.

[0032] Furthermore, the first control unit 270 causes the first storage unit 280 to store calibration data. The calibration data is data for converting the emission position of the second signal light 237 in the captured image into a position on the projection surface 30. The first control unit 270 converts the emission position of the second signal light 237 detected from the captured image into a designated position on the projection surface 30 using the calibration data.

[0033] A frame memory 245 is connected to the image processing unit 240. The image processing unit 240 loads an image based on image data input from the second HDMI I / F 213 into the frame memory 245. The image processing unit 240 performs image processing such as resolution conversion processing, resizing processing, distortion correction, shape correction processing, digital zoom processing, and adjustment of color tone and brightness of the image, on the image loaded into the frame memory 245. The image processing unit 240 outputs image information, which is information on the image loaded into the frame memory 245, to a panel driving unit 257 of the image projection unit 250.

[0034] The frame memory 245 and the image processing unit 240 are configured, for example, by an integrated circuit. The integrated circuit includes an LSI (Large Scale Integrated Circuit), an ASIC, a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), and an SoC (System-on-a-chip). The frame memory 245 and the image processing unit 240 may include an analog circuit as part of the integrated circuit configuration, or may be configured by combining the first control unit 270 with an integrated circuit.

[0035] The image projection unit 250 includes a light source 251, a light modulation device, and a panel drive unit 257. The light modulation device includes three liquid crystal panels 253(R), 253(G), and 253(B), and a projection lens 255 which is an optical system unit.

[0036] The light source 251 includes a solid-state light source such as a light-emitting diode or a semiconductor laser. A discharge-type light source lamp such as an extra-high pressure mercury lamp or a metal halide lamp may be used as the light source 251. The light source 251 emits light that is the source of image light under the control of the first control unit 270. The light emitted by the light source 251 is converted into light having a substantially uniform luminance distribution by an integrator optical system (not shown), and is separated into the three primary colors of light, red (R), green (G), and blue (B), by a color separation optical system (not shown). After that, the light separated into the red (R), green (G), and blue (B) color components is incident on the liquid crystal panels 253(R), 253(G), and 253(B), respectively. The light separated into the color components incident on the liquid crystal panels 253(R), 253(G), and 253(B) is called color light.

[0037] The liquid crystal panels 253(R), 253(G), and 253(B) are each composed of a transmissive liquid crystal panel in which liquid crystal is sealed between a pair of transparent substrates, etc. The liquid crystal panels 253(R), 253(G), and 253(B) each have a rectangular image forming area 254(R), 254(G), and 254(B) formed therein and each of which is composed of a plurality of pixels arranged in a matrix, and a driving voltage can be applied to each pixel.

[0038] The panel driving unit 257 forms an image in the image forming areas 254(R), 254(G), and 254(B) of the liquid crystal panels 253(R), 253(G), and 253(B). Specifically, the panel driving unit 257 applies a driving voltage corresponding to the image information input from the image processing unit 240 to each pixel of the image forming areas 254(R), 254(G), and 254(B) in accordance with an instruction from the first control unit 270. This sets each pixel of the image forming areas 254(R), 254(G), and 254(B) to a light transmittance corresponding to the image information. The light emitted from the light source 251 is modulated for each pixel by passing through the image forming areas 254(R), 254(G), and 254(B) of the liquid crystal panels 253(R), 253(G), and 253(B), and an image light corresponding to the image information is formed for each color light. The image lights of each color thus formed are synthesized for each pixel by a color synthesis optical system (not shown) to become image light that represents a color image, and is enlarged and projected onto the projection surface 30 by the projection lens 255 .

[0039] The first control unit 270 includes a first storage unit 280 and a first processor 290.

[0040] The first storage unit 280 includes at least a non-volatile storage device. The first storage unit 280 may also include a volatile storage device. The non-volatile storage device may be, for example, a Read Only Memory (ROM), a flash memory, or an Electrically Erasable Programmable Read-Only Memory (EEPROM).

[0041] The first storage unit 280 is used as a calculation area for the first processor 290. The first storage unit 280 also stores a control program 285 executed by the first processor 290, setting data that is a setting related to the operation of the projector 200, and the like.

[0042] The first processor 290 is an arithmetic processing device including a processor such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The first processor 290 may be configured with a single processor, or may be configured with multiple processors. The first processor 290 may be configured with an SoC integrated with a part or all of the first storage unit 280 or other circuits. The first processor 290 may be configured with a combination of a CPU that executes a program and a DSP (Digital Signal Processor) that executes a predetermined arithmetic processing. Furthermore, all of the functions of the first processor 290 may be implemented in hardware, or may be configured using a programmable device.

[0043] [3. Control device configuration] Returning to FIG. 1, the configuration of the control device 300 will be described. The control device 300 includes an HDMI I / F 301 , a USB I / F 303 , a remote control light receiving unit 310 , and a second control unit 330 .

[0044] The HDMI I / F 301 includes a connection port and an interface circuit that are compliant with the HDMI standard, and corresponds to a second interface. The USB I / F 303 includes a connection port and an interface circuit that are compatible with the USB standard.

[0045] The remote control 320 includes, for example, a screen split button, a cross key corresponding to each of the up, down, left, and right directions, a back button for undoing a received input, a zoom in and zoom out button, a decision button for finalizing a received input, etc. The screen split button is a button for causing the projector 200 to display a display image 500 having multiple screens.

[0046] In this embodiment, an example in which a dedicated remote control 320 for operating the control device 300 is provided will be described, but the remote control 320 may be a mobile terminal such as a smartphone. In this case, operation keys and the like are displayed on a touch panel of the mobile terminal by an application program installed in the mobile terminal.

[0047] The second control unit 330 includes a second storage unit 340 and a second processor 350 .

[0048] The second storage unit 340 includes at least a non-volatile storage device, such as a ROM, a flash memory, an EEPROM, etc. The second storage unit 340 may include a volatile storage device, such as a RAM (Random Access Memory), in addition to the non-volatile storage device.

[0049] The second storage unit 340 is used as a calculation area for the second processor 350. The second storage unit 340 also stores an OS and application programs 343 executed by the second processor 350. Hereinafter, the application programs are referred to as APP.

[0050] The second processor 350 is an arithmetic processing device including a processor such as a CPU or an MPU. The second processor 350 may be configured with a single processor, or may be configured with multiple processors. The second processor 350 may be configured with a SoC integrated with a part or all of the second storage unit 340 or other circuits. The second processor 350 may be configured with a combination of a CPU that executes a program and a DSP that executes a predetermined arithmetic processing. Furthermore, the second processor 350 may be configured with all of its functions implemented in hardware, or may be configured using a programmable device.

[0051] [4. Display system operation] Next, the operation of the display system 1 in which the projector 200 displays the display image 500 having a plurality of screens 600 on the projection surface 30 and processes the display image 500 based on an operation performed using the pointer 50 will be described. On the multiple screens 600, images supplied by the image supply device 100 and the control device 300 are displayed, respectively. In this embodiment, a case will be described where a first screen 610 and a second screen 630 are displayed as the multiple screens 600. Images supplied by the image supply device 100 and the control device 300 include, for example, moving images and still images played by the image supply device 100 and the control device 300, and images generated by executing an application program installed in the image supply device 100 and the control device 300. Hereinafter, the application program will be referred to as APP343. Also, an image generated by executing an application program will be referred to as an APP image.

[0052] FIG. 3 is a diagram showing the source setting screen 400. As shown in FIG. When the user wants to display multiple screens on the projector 200, the user presses a screen division button provided on the remote control 320. When an operation signal indicating that the screen division button has been pressed is input from the remote control light receiving unit 310, the second control unit 330 causes the projector 200 to display a source setting screen 400, which is a screen for setting a source. The second control unit 330 transmits image data of the source setting screen 400 to the projector 200. The projector 200 generates image light based on the received image data, and projects the generated image light onto the projection surface 30. As a result, the source setting screen 400 is displayed on the projection surface 30.

[0053] The source setting screen 400 includes an enter button 401 , a cursor 3 , a source list 410 , and a setting field 430 . The enter button 401 is a button for entering the settings accepted on the source setting screen 400. The display position of the cursor 3 on the source setting screen 400 can be changed by operating the cross key of the remote control 320.

[0054] Sources are displayed in a list in the source list 410. In the source list 410, the input I / F included in the projector 200 and the APP installed in the control device 300 are displayed as sources.

[0055] The setting field 430 is a field for setting a source to be displayed on the projector 200 from among the multiple sources displayed in the source list 410. The setting field 430 includes a first screen setting field 450 and a second screen setting field 470. In the following explanation, a case where two screens, a first screen 610 and a second screen 630, are displayed on the projector 200 will be explained. In the first screen setting field 450, a toggle button 455 is displayed for each source displayed in the source list 410, and in the second screen setting field 470, a toggle button 475 is displayed for each source displayed in the source list 410.

[0056] The user operates the cross key of remote control 320 to move cursor 3 to the position of toggle button 455 corresponding to the source that the user wants to display on first screen 610, and presses the decision button of remote control 320. This changes the selected toggle button 455 from off to on. Similarly, the user operates the cross key of remote control 320 to move cursor 3 to the position of toggle button 475 corresponding to the source that the user wants to display on second screen 630, and presses the decision button of remote control 320. This changes the selected toggle button 475 from off to on.

[0057] Next, the user operates the cross key of the remote control 320 to move the cursor 3 to the position of the decision button 401. After that, the user presses the decision button provided on the remote control 320. When an operation signal indicating that the decision button has been pressed is input from the remote control light receiving unit 310, the second control unit 330 accepts the setting of the source setting screen 400.

[0058] Next, the second control unit 330 transmits a processing request that instructs the projector 200 to execute a process. This processing request includes a first processing request and a second processing request.

[0059] The first processing request includes a request to transmit operation information indicating the position of the projection surface 30 indicated by the indicator 50. The second control unit 330 transmits the first processing request to the projector 200 when the input I / F of the projector 200 is not included in the source set on the source setting screen 400. In other words, the image to be displayed on the projector 200 is only the APP image 615 generated by executing the APP installed in the control device 300, and does not include an image input to the input I / F. For this reason, the second control unit 330 transmits the first processing request, which requests transmission of operation information, to the projector 200.

[0060] The second processing request includes a request to send the operation information included in the first processing request, as well as a request to send the capture image 635 and identification information indicating the input I / F that captures the image. If the input I / F of the projector 200 is included in the source set on the source setting screen 400, the second control unit 330 transmits the second processing request to the projector 200. The second processing request includes, in addition to a request to send the operation information, identification information of the input I / F set on the source setting screen 400, and a request to send the capture image 635 that captures an image input to the input I / F indicated by this identification information.

[0061] When the first control unit 270 receives the first processing request from the control device 300, it causes the transmission unit 231 to start transmitting the first signal light 235. Furthermore, the first control unit 270 causes the imaging unit 233 to perform imaging and generate a captured image, based on the transmission timing of the first signal light 235.

[0062] The first control unit 270 detects the emission of the second signal light 237 from the captured image generated by the imaging unit 233. The first control unit 270 generates operation information indicating a position indicated by the indicator 50, based on the detected emission position of the second signal light 237. The first control unit 270 transmits the generated operation information to the control device 300.

[0063] Furthermore, when the first control unit 270 receives a second processing request from the control device 300, it causes the image processing unit 240 to load the image data input to the input I / F indicated by the identification information in the frame memory 245. Furthermore, the first control unit 270 causes the image processing unit 240 to capture an image based on the image data loaded in the frame memory 245, and generates a captured image 635. The first control unit 270 transmits the generated captured image 635 to the control device 300 via the first USB I / F 221.

[0064] Moreover, the first control unit 270 generates operation information indicating the position indicated by the indicator 50, and transmits the generated operation information to the control device 300, in the same manner as when the first processing request is received.

[0065] FIG. 4 is a diagram showing a first display image 500A displayed on the projection surface 30. As shown in FIG. Next, the second control unit 330 generates a first display image 500A to be displayed by the projector 200. The first display image 500A includes a first screen 610 and a second screen 630.

[0066] When the source set on the source setting screen 400 includes an APP installed in the control device 300, the second control unit 330 starts the APP set on the source setting screen 400. The second control unit 330 executes the set APP and displays an APP image 615 generated by the APP on the first screen 610 or the second screen 630. In the following, it is assumed that the APP image 615 generated by the APP is displayed on the first screen 610. The APP image 615 corresponds to the second image.

[0067] Furthermore, when the input I / F of the projector 200 is included in the sources set on the source setting screen 400, the second control unit 330 receives a capture image 635 transmitted from the projector 200. The second control unit 330 displays the received capture image 635 on the first screen 610 or the second screen 630. In the following, it is assumed that the capture image 635 is displayed on the second screen 630.

[0068] FIG. 5 is a diagram showing a second display image 500B. When the second control unit 330 receives operation information from the projector 200, it generates an object image 550 based on the received operation information and superimposes the generated object image 550 on the position of the first display image 500A indicated by the operation information. The image in which the object image 550 is superimposed on the first display image 500A is called a second display image 500B. The second display image 500B is also an example of the display image 500.

[0069] When the operation information received from the projector 200 includes operation information indicating a designated position on the projection surface 30, the second control unit 330 superimposes the object image 550 at the position on the projection surface 30 indicated by the operation information. The object image 550 generated at this time is, for example, an image indicating a line corresponding to the locus of the designated position indicated by the projection surface 30. Furthermore, when the operation indicated by the operation information is an operation to change the line width or color of the line, the second control unit 330 changes the thickness or color of the line displayed as the object image 550. Furthermore, when an eraser icon is selected as the icon, the second control unit 330 erases the object image 550 displayed at the position on the projection surface 30 indicated by the position information. The process of superimposing the object image 550, the process of changing the thickness or color of the line displayed as the object image 550, or the process of erasing the object image 550 are examples of processes on the first display image. Furthermore, the operation information may be information instructing a change in the display color, brightness, or the like of the first APP image 615 or the captured image 635 displayed in the first display image 500A or the second display image 500B. In this case, the second control unit 330 may control the first APP image 615 to change the display color or brightness of the first APP image 615, or may superimpose an image indicated by the operation information on the captured image 635 to change the display color of the captured image 635.

[0070] When the second control unit 330 generates the second display image 500B, it transmits image data of the generated second display image 500B to the projector 200. The projector 200 generates image light based on the image data received from the control device 300, and displays the generated image light on the projection surface 30 by the projection lens 255. As a result, the second display image 500B generated by the control device 300 is displayed on the projection surface 30.

[0071] FIG. 6 shows an operation for changing the display position of the first screen 610. As shown in FIG. The user can change the display position and display size of first screen 610 and second screen 630 in first display image 500A by operating remote control 320. This operation will be described.

[0072] First, the user operates the cross key of the remote control 320 to move the cursor 3 onto the first screen 610 or the second screen 630 for which the display position or display size is to be changed. After that, the user briefly presses the decision button of the remote control 320. Here, the explanation will be given on the assumption that the first screen 610 has been selected.

[0073] When an operation signal indicating that the decision button has been pressed briefly is input, the second control unit 330 displays arrow images 570 indicating the up, down, left and right directions shown in Fig. 6 on the top side 610A, bottom side 610B, left side 610C and right side 610D of the first screen 610. An arrow image 571 indicating the up direction is displayed on the top side 610A of the first screen 610. An arrow image 573 indicating the down direction is displayed on the bottom side 610B of the first screen 610. An arrow image 575 indicating the left direction is displayed on the left side 610C of the first screen 610. An arrow image 577 indicating the right direction is displayed on the right side 610D of the first screen 610.

[0074] The user operates the cross key of remote control 320 to move cursor 3 to the display position of arrow image 570 corresponding to the direction in which the user wants to move first screen 610. Then, the user presses the enter button on remote control 320. When an operation signal indicating that the enter button has been pressed is input from remote control light receiving unit 310, second control unit 330 moves first screen 610 by a preset distance in the direction corresponding to arrow image 570 overlapping with the display position of cursor 3. Second control unit 330 moves first screen 610 by a preset distance each time the enter button is pressed.

[0075] Furthermore, when the user moves first screen 610 to the position of first display image 500A where the user wants first screen 610 to be displayed, he or she presses the decision button provided on remote control 320. When the decision button on remote control 320 is pressed, second control unit 330 finalizes the display position of first screen 610.

[0076] Also, when enlarging or reducing the display size of the screen 600, the user operates the cross key of the remote control 320 to place the cursor 3 on the screen 600 whose display position or display size is to be changed, and presses the decision button. When changing the display size of the screen 600, the decision button may be pressed twice in succession, or the decision button may be pressed and held down. In this embodiment, it is assumed that when the user performs an operation of pressing and holding down the decision button, the second control unit 330 is in a state in which it is possible to accept the operation of the enlargement button or reduction button provided on the remote control 320. In this state, the user presses the enlargement button or reduction button provided on the remote control 320. The second control unit 330 enlarges or reduces the display size of the screen 600 on which the cursor 3 is placed while the enlargement button or reduction button is being pressed down.

[0077] When the user has finished adjusting the display position and display size of screen 600 in first display image 500A, he or she moves cursor 3 to a position where it does not overlap screen 600, and presses the enter button on remote control 320. When the enter button of the remote control 320 is pressed with the cursor 3 not overlapping the screen 600, the second control unit 330 determines that adjustment of the display position or the display size of the screen 600 is completed. The second control unit 330 transmits image data of the first display image 500A or the second display image 500B to the projector 200 via the HDMI I / F 301.

[0078] The first control unit 270 of the projector 200 causes the image processing unit 240 to load the received image data of the first display image 500A or the second display image 500B in the frame memory 245. The image processing unit 240 performs image processing, such as adjusting the color tone and brightness of the image, on the image loaded in the frame memory 245. The image processing unit 240 then outputs image information, which is information on the image loaded in the frame memory 245, to the image projection unit 250. The image projection unit 250 generates image light corresponding to the image information, and projects the generated image light onto the projection surface 30 by the projection lens 255.

[0079] 7, 8 and 9 are flowcharts showing the operation of the control device 300. The operation of the control device 300 will be described with reference to the flow charts shown in FIGS. First, the second control unit 330 judges whether or not an operation signal indicating that the screen division button of the remote control 320 has been pressed has been input from the remote control light receiving unit 310 (step S1). If an operation signal of the screen division button has not been input (step S1 / NO), the second control unit 330 waits for the start of this processing flow.

[0080] When an operation signal indicating that the screen division button has been pressed is input (step S1 / YES), the second control unit 330 causes the projector 200 to display the source setting screen 400 on the projection surface 30 (step S2). The second control unit 330 generates image data of the source setting screen 400 and transmits it to the projector 200. The projector 200 generates image light based on the received image data and projects the generated image light onto the projection surface 30. As a result, the source setting screen 400 is displayed on the projection surface 30.

[0081] Next, the second control unit 330 accepts the setting of the source setting screen 400 by operating the remote control 320. Then, the second control unit 330 judges whether or not an operation signal indicating that the decision button of the remote control 320 has been pressed has been input from the remote control light receiving unit 310 (step S3). If an operation signal indicating that the decision button has been pressed has not been input (step S3 / NO), the second control unit 330 waits until the decision button is pressed.

[0082] When an operation signal indicating that the decision button on the remote control 320 has been pressed (step S3 / YES), the second control unit 330 determines whether or not the input I / F of the projector 200 is included in the sources selected on the source setting screen 400 (step S4).

[0083] If the source includes the input I / F of the projector 200 (step S4 / YES), the second control unit 330 transmits a second processing request to the projector 200 (step S5). The second processing request includes a request to transmit operation information, a request to transmit the captured image 635, and identification information indicating the input I / F that captures the image.

[0084] Next, the second control unit 330 starts the APP set on the source setting screen 400 (step S6). Here, it is assumed that the first APP is set on the source setting screen 400. The second control unit 330 executes the started first APP and generates image data for the first display image 500A or the second display image 500B (step S7). Alternatively, if the image data for the first display image 500A or the second display image 500B has already been generated, the second control unit 330 updates the image data for the first display image 500A or the second display image 500B (step S7).

[0085] Next, the second control unit 330 determines whether it is time to transmit the image data of the first display image 500A or the second display image 500B to the projector 200 (step S8). If it is not time to transmit (step S8 / NO), the second control unit 330 waits until it is time to transmit.

[0086] When the second control unit 330 determines that it is time to transmit image data of the first display image 500A or the second display image 500B (step S8 / YES), the second control unit 330 transmits this image data to the projector 200 (step S9).

[0087] Next, the second control unit 330 judges whether or not an operation signal instructing to end the display has been input from the remote control light receiving unit 310 (step S10). If an operation signal instructing to end the display has not been input (step S10 / NO), the second control unit 330 returns to the process of step S7. On the other hand, if an operation signal instructing to end the display has been input (step S10 / YES), the second control unit 330 ends this process flow.

[0088] Next, the operation of the control device 300 when it is determined in step S4 that the input I / F of the projector 200 is not included will be described. If the source does not include the input I / F of the projector 200 (step S4 / NO), the second control unit 330 transmits a first processing request to the projector 200 (step S11). The first processing request includes a request to transmit operation information.

[0089] Next, the second control unit 330 starts the APP set on the source setting screen 400 (step S12). Here, it is assumed that two APPs, a first APP and a second APP, are set. The second control unit 330 executes the started first APP and second APP, and generates image data of the first display image 500A or the second display image 500B (step S13). Alternatively, if the image data of the first display image 500A or the second display image 500B has already been generated, the second control unit 330 updates the first display image 500A or the second display image 500B (step S13).

[0090] The operations of steps S14 to S16 following step S13 are the same as steps S8 to S10, and therefore will not be described. The second control unit 330 determines whether or not an operation signal instructing to end the display has been input from the remote control light receiving unit 310, and if an operation signal instructing to end the display has not been input (step S16 / NO), the process returns to step S13.

[0091] Next, step S7 will be described in detail with reference to the flowchart shown in FIG. First, the second control unit 330 executes the first APP to generate a first APP image 615 (step S71), and arranges the generated first APP image 615 on the first screen 610 of the first display image 500A (step S72).

[0092] Next, the second control unit 330 determines whether or not the capture image 635 transmitted by the projector 200 has been received by the USB I / F 303 (step S73). If the second control unit 330 has not received the capture image 635 via the USB I / F 303 (step S73 / NO), the process proceeds to the determination in step S75.

[0093] When the second control unit 330 receives the capture image 635 via the USB I / F 303 (step S73 / YES), the second control unit 330 arranges the received capture image 635 on the second screen 630 of the first display image 500A (step S74).

[0094] Next, the second control unit 330 determines whether or not operation information has been received from the projector 200 via the USB I / F 303 (step S75). If the second control unit 330 has not received the operation information (step S75 / NO), the process proceeds to the determination in step S8.

[0095] When the second control unit 330 receives the operation information (step S75 / YES), it generates an object image 550 based on the operation information (step S76). Next, the second control unit 330 places the generated object image 550 at the position of the first display image 500A indicated by the operation information, and generates a second display image 500B (step S77).

[0096] Next, details of step S13 will be described with reference to the flowchart shown in FIG. First, the second control unit 330 executes a first APP to generate a first APP image, and executes a second APP, which is an APP other than the first APP, to generate a second APP image (step S131).

[0097] Next, second control unit 330 arranges the first APP image on first screen 610 of first display image 500A, and arranges the second APP image on second screen 630 of first display image 500A (step S132).

[0098] Next, the second control unit 330 determines whether or not operation information has been received from the projector 200 via the USB I / F 303 (step S133). If operation information has not been received (step S133 / NO), the second control unit 330 proceeds to the determination in step S14.

[0099] When the second control unit 330 receives the operation information (step S133 / YES), it generates an object image 550 based on the received operation information (step S134). Next, the second control unit 330 places the generated object image 550 at the position of the first display image 500A indicated by the operation information to generate a second display image 500B (step S135).

[0100] FIG. 10 is a flowchart showing the operation of the projector 200. The operation of the projector 200 will be described with reference to the flowchart shown in FIG. First, the first control unit 270 judges whether or not a processing request has been received from the control device 300 (step T1). If a processing request has not been received (step T1 / NO), the first control unit 270 waits until a processing request is received.

[0101] When the first control unit 270 receives a processing request (step T1 / YES), it determines whether the received processing request is a first processing request (step T2).

[0102] When the received processing request is a first processing request (step T2 / YES), the first control unit 270 detects the position pointed to by the indicator 50 and generates operation information (step T3).

[0103] The first control unit 270 analyzes the captured image input from the imaging unit 233 to detect the emission of the second signal light 237 transmitted by the indicator 50. The first control unit 270 also analyzes a plurality of consecutive captured images to determine whether the detected emission of the second signal light 237 is an emission when the indicator 50 is in contact with the projection surface 30 or an emission when the indicator 50 is not in contact with the projection surface 30. The first control unit 270 converts the emission position of the second signal light 237 when the indicator 50 is in contact with the projection surface 30 into operation information indicating the indicated position on the projection surface 30 based on the calibration data. The first control unit 270 transmits the generated operation information indicating the indicated position on the projection surface 30 to the control device 300.

[0104] Next, first control unit 270 causes projection surface 30 to display an image based on the image data received from control device 300 (step T4). The first control unit 270 receives image data of the first display image 500A or the second display image 500B from the control device 300, and generates image light based on the received image data. The first control unit 270 projects the generated image light onto the projection surface 30 by the projection lens 255. As a result, the first display image 500A or the second display image 500B is displayed on the projection surface 30.

[0105] Next, the first control unit 270 judges whether or not an end notification for ending the display of the display image is received from the control device 300 (step T5). If the end notification is not received (step T5 / NO), the first control unit 270 returns to the process of step T3. On the other hand, if the end notification is received (step T5 / YES), the first control unit 270 ends this process flow.

[0106] Furthermore, when the received processing request is not the first processing request (step T2 / NO), the first control unit 270 determines that the received processing request is the second processing request. In this case, similarly to steps T3 and T4, the first control unit 270 detects the position indicated by the indicator 50 and generates operation information (step T6), and then the first control unit 270 executes a capture image generation process (step T7).

[0107] In step T7, the first control unit 270 captures an image based on image data input to the input I / F indicated by the identification information included in the second processing request, and generates a captured image 635. The first control unit 270 transmits the generated captured image 635 to the control device 300 via the first USB I / F 221.

[0108] Next, first control unit 270 causes projection surface 30 to display an image based on the image data received from control device 300 (step T8).

[0109] Next, the first control unit 270 judges whether or not an end notification for ending the display of the display image is received from the control device 300 (step T9). If the end notification is not received (step T9 / NO), the first control unit 270 returns to the process of step T6. On the other hand, if the end notification is received (step T9 / YES), the first control unit 270 ends this process flow.

[0110] 5. Other embodiments The above-described embodiment is a preferred embodiment, however, the present invention is not limited to the above-described embodiment and various modifications are possible without departing from the spirit and scope of the present invention. For example, in the above-described embodiment, the control device 300 has been described as transmitting image data of the first display image 500A or the second display image 500B to the projector 200 via the HDMI I / F 301. Alternatively, the control device 300 may be configured to transmit the image data of the first display image 500A or the second display image 500B to a server device, and have the server device distribute the image data over the Web.

[0111] In addition, in this embodiment, a case will be described in which the projector 200 includes three transmissive liquid crystal panels 253(R), 253(G), and 253(B) as light modulation devices, but the embodiment of the present invention is not limited thereto. For example, the projector 200 may include a reflective liquid crystal panel or an LCOS (Liquid Crystal On Silicon) liquid crystal panel as a light modulation device. The projector 200 may also be configured to include one or more DMDs (Digital Mirror Devices). In these configurations, the projector 200 includes a light modulation device drive unit corresponding to the configuration of the light modulation device instead of the panel drive unit 257. In addition, the present disclosure is applicable to various projectors including the above-mentioned various configurations as a light modulation device, and to display devices other than projectors.

[0112] 1 and 2 show functional configurations, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware corresponding to each functional unit individually, and it is of course possible to implement a configuration in which one processor executes a program to realize the functions of multiple functional units. Also, some of the functions realized by software in the above embodiment may be realized by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of each of the other units of the projector 200 and the imaging unit 233 can also be arbitrarily changed within the scope of the gist of the invention.

[0113] In addition, when the display system control method, the display device control method, and the control device control method are realized using a computer included in the control device 300 and the projector 200, the program executed by the computer can be configured in the form of a recording medium or a transmission medium for transmitting the program. The recording medium can be a magnetic or optical recording medium or a semiconductor memory device. Specifically, portable or fixed recording media such as a flexible disk, HDD (Hard Disk Drive), CD-ROM, DVD, Blu-ray Disc, magneto-optical disk, flash memory, and card-type recording medium can be mentioned. The recording medium may also be a non-volatile storage device such as RAM, ROM, and HDD, which are internal storage devices included in the server device. Blu-ray is a registered trademark.

[0114] 6. Summary of this Disclosure The following is a summary of this disclosure.

[0115] (Appendix 1) A control method for a display system, comprising: a display device acquiring a first image; a control device generating a first display image in which the first image acquired by the display device and a second image are arranged; the display device displaying the first display image; the control device processing the first display image based on operation information indicating an operation on the first display image to generate a second display image; and the display device displaying the second display image.

[0116] According to this, a first display image in which a first image acquired by the display device and a second image are arranged is generated by the control device, and the generated first display image is displayed on the display device. Also, a second display image is generated by the control device based on operation information for the first display image, and the generated second display image is displayed on the display device. Therefore, when the device supplying the first image to the display device cannot execute processing based on the operation information, the control device generates a second display image based on the operation information for the first display image. Then, the second display image is displayed by the display device. Therefore, processing can be executed for the first image and the second image based on the operation information and reflected in the first image.

[0117] (Appendix 2) The control method for a display system described in Appendix 1, wherein the control device generates the first display image includes generating the first display image that includes the second image and does not include the first image when the display device is unable to acquire the first image.

[0118] According to this, when the display device cannot acquire the first image, a first display image including the second image and not including the first image is generated, so that the second image can be processed by operation information for the first display image including only the second image.

[0119] (Appendix 3) A method for controlling a display system described in Appendix 1 or 2, further comprising the display device transmitting the acquired first image to the control device via a first interface, and the control device transmitting the first display image and the second display image to the display device via a second interface different from the first interface.

[0120] According to this, the first image acquired by the display device and the first and second display images generated by the control device are transmitted through different interfaces, thereby reducing delays in displaying the first and second display images transmitted by the control device to the display device.

[0121] (Appendix 4) A method for controlling a display system described in any one of Appendix 1 to 3, further comprising the display device transmitting the first image to the control device, wherein the display device transmitting the first image to the control device comprises the display device being recognized as a USB Video Class device by the control device when the display device and the control device are connected, and the display device transmitting the first image to the control device in a format defined in the USB Video Class.

[0122] According to this, the first image transmitted from the display device to the control device is transmitted in a format defined by the USB Video Class, which eliminates the need for the control device to perform processing such as converting the format of the first image, thereby reducing the processing load on the control device.

[0123] (Appendix 5) A control method for a display system described in any one of Appendix 1 to 4, further comprising: the display device acquiring the operation information including a position of the first display image indicated by an indicator; and the display device transmitting the operation information to the control device, and the control device processing the first display image to generate a second display image includes the control device superimposing an object image based on the operation information on the first display image to generate the second display image.

[0124] According to this, the second display image is generated by performing processing on the first display image based on the position information indicating the pointing position of the pointer, and therefore, the object image can be displayed at the position pointed to by the pointer.

[0125] (Appendix 6) A method for controlling a display device, comprising: acquiring a first image and transmitting the acquired first image to a control device; receiving from the control device a first display image in which the first image and a second image are arranged; displaying the first display image; receiving from the control device a second display image in which the first display image has been processed based on operation information indicating an operation on the first display image; and displaying the second display image.

[0126] According to this, the display device can display a first display image in which the first image acquired by the display device and the second image are arranged. Also, the display device can display a second display image processed based on operation information on the first display image. Therefore, it is possible to display the second display image that reflects the processing on the first image and the second image based on the operation information.

[0127] (Appendix 7) A control method for a control device, comprising: generating a first display image in which a first image received from a display device and a second image are arranged; transmitting the first display image to the display device; generating a second display image by processing the first display image based on operation information indicating an operation on the first display image; and transmitting the second display image to the display device.

[0128] According to this, a first display image in which the first image acquired by the display device and the second image are arranged can be generated, and the generated first display image is transmitted to the display device and can be displayed on the display device. Also, a second display image can be generated based on operation information for the first display image, and the generated second display image is displayed on the display device and can be displayed on the display device. Therefore, when the device supplying the first image to the display device cannot execute processing based on the operation information, the control device generates a second display image based on the operation information for the first display image. Then, the control device causes the display device to display the second display image. Therefore, processing for the first image and the second image can be executed based on the operation information. [Explanation of symbols]

[0129] 1...display system, 3...cursor, 30...projection surface, 50...pointer, 100...image supply device, 200...projector, 210...HDMII / F, 211...first HDMII / F, 213...second HDMII / F, 215...third HDMII / F, 220...USBI / F, 221...first USBI / F, 223...second USBI / F, 231...transmitting unit, 233...imaging unit, 23 5...first signal light, 237...second signal light, 240...image processing unit, 245...frame memory, 250...image projection unit, 251...light source, 253...liquid crystal panel, 254...image forming area, 255...projection lens, 257...panel drive unit, 270...first control unit, 280...first storage unit, 285...control program, 290...first processor, 300...control device, 301...HDMI I / F, 303...USB I / F, 310...remote control receiver, 320...remote control unit, 330...second control unit, 340...second storage unit, 350...second processor, 400...source setting screen, 401...decision button, 410...source list, 430...setting field, 450...first screen setting field, 455...toggle button, 470...second screen setting field, 475...toggle button, 500...display image , 500A...first display image, 500B...second display image, 550...object image, 570...arrow image, 571...arrow image, 573...arrow image, 575...arrow image, 577...arrow image, 600...screen, 610...first screen, 610A...top edge, 610B...bottom edge, 610C...left edge, 610D...right edge, 635...APP image, 630...second screen, 635...capture image.

Claims

1. acquiring a first image with a display device; A control device generates a first display image by arranging the first image and a second image acquired by the display device; the display device displays the first display image; and the control device performs processing on the first display image based on operation information indicating an operation on the first display image to generate a second display image; the display device displays the second display image; and A method for controlling a display system comprising:

2. The control device generating the first display image includes: generating the first display image including the second image and not including the first image when the display device is unable to acquire the first image. A method for controlling the display system according to claim 1.

3. The display device further includes transmitting the acquired first image to the control device via a first interface; the control device further includes transmitting the first display image and the second display image to the display device via a second interface different from the first interface. A method for controlling the display system according to claim 1.

4. The display device further includes transmitting the first image to the control device; The display device transmits the first image to the control device, When the display device and the control device are connected to each other, the display device is recognized by the control device as a USB video class device; The display device transmits the first image to the control device in a format defined by the USB Video Class; 2. The method of claim 1, further comprising:

5. The display device acquires the operation information including a position of the first display image pointed to by a pointer; The display device transmits the operation information to the control device; Further comprising: The control device performing processing on the first display image to generate a second display image is the control device superimposing an object image based on the operation information on the first display image to generate the second display image. A method for controlling the display system according to claim 1.

6. capturing a first image and transmitting the captured first image to a controller; receiving a first display image in which the first image and the second image are arranged from the control device; displaying the first display image; receiving, from the control device, a second display image obtained by performing processing on the first display image based on operation information indicating an operation on the first display image; displaying the second display image; and A method for controlling a display device comprising:

7. generating a first display image in which the first image received from the display device and the second image are arranged; transmitting the first display image to the display device; generating a second display image by processing the first display image based on operation information indicating an operation on the first display image; transmitting the second display image to the display device; A control method for a control device comprising:

Images