Information processing method of information processing apparatus

Abstract

Information processing method of information processing apparatus. Processing of display screen at the time when mirror is stopped by user is not considered. Information processing method of information processing apparatus includes: displaying application window; displaying data transmission window based on action instruction; transmitting display data that causes display screen to be displayed to projection apparatus via web browser, executing projection processing that causes projection apparatus to project display screen; displaying execution information in application window; receiving browser window stop operation; based on browser window stop operation, stopping projection processing, closing data transmission window; and executing display change processing that changes execution information to stop information.

Description

Technical Field

[0001] This disclosure relates to an information processing method for an information processing apparatus. Background Technology

[0002] Image display devices that connect to control devices such as computers are known. The projector described in Patent Document 1 is an example of an image display device, displaying an input image based on image information supplied via a connection terminal. The projector is capable of displaying using Miracast. Miracast is a registered trademark. Miracast is a display transmission technology based on wireless communication. The projector can use Miracast to mirror the display screen of the control device. Miracast is an example of a mirroring technique.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-64131 Summary of the Invention

[0006] However, the handling of the display screen when the user stops the mirroring was not considered.

[0007] The information processing method of the information processing apparatus disclosed herein includes: launching an application that controls a projection device and displaying an application window that operates the application; establishing a connection with the projection device across a network for data transmission; displaying a data transmission window containing a data transmission page related to projection in a web browser based on an action instruction from the application; sending display data that causes a display screen to be displayed to the projection device via the web browser, and performing projection processing to cause the projection device to project the display screen; displaying execution information indicating that the projection processing is being performed in the application window; accepting a browser window stop operation that closes the data transmission window; stopping the projection processing and closing the data transmission window based on the browser window stop operation; and performing a display change process that changes the execution information to stop information indicating that the projection processing has not been performed.

[0008] The information processing method of the information processing apparatus disclosed herein includes: launching an application for controlling a projection device and displaying an application window for operating the application; establishing a connection with the projection device across a network for transmitting data; displaying a data transmission window containing a data transmission page related to projection in a web browser based on an action instruction from the application; sending display data that causes a display screen to be displayed to the projection device via the web browser, and performing projection processing to cause the projection device to project the display screen; displaying execution information indicating that the projection processing is being performed in the application window; accepting a stop operation in the application window to stop the projection processing; and stopping the projection processing and closing the data transmission window based on the stop operation. Attached Figure Description

[0009] Figure 1 It is a diagram showing the general structure of the projection system.

[0010] Figure 2 This is a diagram showing the modular structure of the projection system.

[0011] Figure 3 It is a diagram showing the general structure of the projection unit.

[0012] Figure 4 It is a diagram showing the system structure of the projection system.

[0013] Figure 5 This is an example of a displayed screen.

[0014] Figure 6 It is a diagram that shows the control flow executed by the control device.

[0015] Figure 7 This is an example of a displayed screen.

[0016] Figure 8 This is an example of a displayed screen.

[0017] Figure 9 This is an example of a displayed screen.

[0018] Figure 10 It is a diagram representing a projected image projected onto a projection surface.

[0019] Figure 11 This is a diagram illustrating an example of a control flow executed by a control device.

[0020] Figure 12 This is an example of a displayed screen.

[0021] Figure 13 This is a diagram illustrating an example of a control flow executed by a control device.

[0022] Figure 14 This is a diagram illustrating an example of a control flow executed by a control device.

[0023] Figure 15 This is an example of a displayed screen.

[0024] Figure 16 This is a diagram illustrating an example of a control flow executed by a control device.

[0025] Figure 17 This is an example of a displayed screen.

[0026] Explanation of reference numerals in the attached figures

[0027] 1…Projection system, 10…Control device, 15…Display, 17…Input unit, 20…Output unit, 21…Display data output unit, 23…Sound output unit, 30…Control unit, 31…Browser control unit, 33…Application control unit, 40…Storage unit, 50…Communication unit, 70…Dedicated operating system, 80…Web browser, 81…Projection setting page, 81a…Mirror start icon, 81w…Projection setting page window, 83…Data acquisition page, 83a…Close button, 83w…Data acquisition page window, 8 4…Capture application, 90…Virtual device, 91…Projection application, 100…Projector, 110…Projection unit, 111…Light source, 111a…Light source unit, 111b…Reflector, 113…Liquid crystal light valve, 113a…Pixel area, 113B…Liquid crystal light valve for blue light, 113G…Liquid crystal light valve for green light, 113R…Liquid crystal light valve for red light, 115…Light valve drive unit, 117…Projection lens, 120…Projector built-in speaker, 130…Projector control unit, 131…Projection control unit, 140…Projector storage Storage unit, 150… Projector communication unit, 151… Streaming communication unit, 153… Signal communication unit, 170… UI screen, 171… Projector name display unit, 172… Mirroring acceptance icon, 172a… Switch icon, 173… Photo display acceptance icon, 174… Document display acceptance icon, 175… Camera image display acceptance icon, 176… Remote control function acceptance icon, 177… Multi-projection acceptance icon, 178… Cut-off acceptance icon, 179… End acceptance icon, 180… Selection screen, 181… Select image, 1 81a…First selected image, 183…Share icon, 185…Share cancel icon, 200…Router, 300…Cloud server, 310…Network server, 320…WebSocket server, 330…Server storage device, 330a…Server storage unit, 340…Server communication device, BW…Browser window, DS…Display screen, NW…Communication network, PG…Projected image, PL…Pixel, SC…Projection surface, SP…Streaming path, TP…TCP / IP communication path, TW…Label window. Detailed Implementation

[0028] Figure 1 The schematic structure of projection system 1 is shown. Projection system 1 includes a control device 10, a projector 100, a router 200, and a cloud server 300. The projector 100 projects a projected image PG onto the projection surface SC. The cloud server 300 is communicatively connected to the control device 10 via a communication network NW. Figure 1 The projection system 1 shown has a control device 10 and a projector 100, but is not limited thereto. The projection system 1 may also have multiple control devices 10 and multiple projectors 100.

[0029] The control device 10 is communicatively connected to the projector 100. The control device 10 sends projected image data to the projector 100. The control device 10 executes actions to cause the projector 100 to project a display screen DS onto the monitor 15 as a mirror image of the projected image PG. The control device 10 may have functions such as adjusting the shape of the projected image PG projected by the projector 100 onto the projection surface SC. The control device 10 corresponds to an example of an information processing device. The control device 10 is a tablet terminal, mobile computer, desktop computer, etc. Figure 1 The mobile computer equipped with a display 15 and an input unit 17 is referred to as a control device 10.

[0030] The control device 10 executes various programs, such as a web browser 80, on a dedicated operating system 70. The dedicated operating system 70 is, for example, ChromeOS. Chrome is a registered trademark. The dedicated operating system 70 is an example of an operating system. The dedicated operating system 70 can construct a virtual environment. In the virtual environment, programs that run on an operating system different from the dedicated operating system 70 can run. An operating system different from the dedicated operating system 70 is, for example, AndroidOS. Android is a registered trademark. Programs that run on an operating system different from the dedicated operating system 70 are, for example, Android applications.

[0031] The display 15 displays images and videos. The display 15 is composed of a liquid crystal panel, an organic EL (electro-luminescence) panel, etc. The display 15 may have a touch input function to accept user input operations.

[0032] Input unit 17 accepts user input operations. Figure 1 The input unit 17 shown represents a keyboard, but is not limited to it. The input unit 17 may consist of a mouse, touchpad, etc. When the display 15 has touch input functionality, the display 15 functions as the input unit 17.

[0033] The projector 100 is positioned opposite the projection surface SC. The projector 100 corresponds to an example of a projection device. The projector 100 is communicatively connected to the control device 10. Figure 1The projector 100 shown communicates with the control device 10 via router 200, but the projector 100 can also communicate directly with the control device 10. The projector 100 can also communicate with external devices other than the control device 10. The projector 100 receives projected image data from the control device 10. The projector 100 projects a projected image PG onto the projection surface SC based on the projected image data. The projector 100 can also mirror the display screen DS shown on the monitor 15 of the control device 10 based on the projected image data. The projector 100 can also project the projected image PG onto the projection surface SC based on pre-stored OSD data. OSD is an abbreviation for Screen Display.

[0034] The projection surface SC displays the projected image PG projected from the projector 100. Figure 1 The projection surface SC shown is formed by a screen, but is not limited to this. The projection surface SC can also be an indoor wall, ceiling, building exterior wall, etc. The shape of the projection surface SC is not limited to a plane, but can also be a curved surface, a surface with concave and convex features, a sphere, or other three-dimensional shapes.

[0035] Router 200 is a communication device that connects control device 10, projector 100, and cloud server 300 to each other. Router 200 connects control device 10 and cloud server 300 via communication network NW. Router 200 connects control device 10 and projector 100 according to a defined communication protocol. Router 200 can also connect projector 100 and cloud server 300.

[0036] The communication network NW connects the cloud server 300 to the control device 10 and other devices. As an example, the communication network NW is a WAN (Wide Area Network).

[0037] A cloud server 300 is a virtual server that operates in a cloud computing environment. A cloud server 300 is a service that utilizes one or more virtualized servers via the Internet. Figure 1 The cloud server 300 shown includes a network server 310 and a WebSocket server 320.

[0038] The network server 310 provides HTML and object display to the web browser 80 of the client software. Figure 1 A virtual network server 310 is displayed. Based on instructions from the control device 10, the network server 310 provides web applications to the web browser 80 of the control device 10. When the control device 10 or the like pre-stores web applications, the projection system 1 may not need to have a network server 310.

[0039] The WebSocket server 320 is a server that enables data communication between web applications on a web browser 80 according to the WebSocket standard. Figure 1 The WebSocket server 320 is virtually shown. The WebSocket server 320 is used as a signaling server for SDP exchange when streaming shared data using WebRTC (Web Real-Time Communication) technology. SDP is an abbreviation for Session Description Protocol.

[0040] Figure 2 The module structure of projection system 1 is shown. Figure 2 The module structure of the projection system 1, including the control device 10, the projector 100, and the cloud server 300, is shown. Figure 2 The projection system 1 is shown with router 200 omitted.

[0041] The control device 10 includes a display 15, an input unit 17, an output unit 20, a control unit 30, a storage unit 40, and a communication unit 50. The control device 10 also includes a built-in speaker (not shown).

[0042] The display 15 displays the display screen DS based on the control of the control unit 30. The display 15 is connected to the display data output unit 21 of the output unit 20. The display data output unit 21 will be described later. Figure 2 The display 15 shown is disposed on the control device 10, but is not limited thereto. The display 15 may also be an external display panel disposed on the control device 10.

[0043] The input unit 17 accepts various input operations from the user. The input unit 17 generates an input signal corresponding to the input operation and outputs the input signal to the control unit 30. Figure 2 The input unit 17 shown is provided in the control device 10, but is not limited thereto. The input unit 17 may also be an input device external to the control device 10.

[0044] The output unit 20 outputs various generated data generated by the control unit 30, etc., to the display 15, etc. The output unit 20 is composed of various circuits, output terminals, etc. The output unit 20 corresponds to an example of an output section. The output unit 20 has a display data output section 21 and a sound output section 23.

[0045] The display data output unit 21 outputs image data of the displayed image to the display 15. The image data includes image data of the displayed still image. The image data is included in the generated data. The display data output unit 21 can also output image data to a display panel externally mounted on the control device 10.

[0046] The sound output unit 23 outputs sound data related to sound. The sound data is included in the generated data. The sound output unit 23 is connected to a built-in speaker (not shown) or the like. The speaker or the like can be built into the control device 10 or externally mounted on the control device 10. The sound output unit 23 outputs sound data via wired or wireless means.

[0047] The control unit 30 is a device control controller that controls the various parts of the control device 10. For example, the control unit 30 is a processor with a CPU (Central Processing Unit). The control unit 30 is composed of one or more processors. A dedicated operating system 70 runs in the control unit 30. The control unit 30 executes various programs on the dedicated operating system 70. The control unit 30 functions as various functional units by executing these programs. For example, the control unit 30 functions as a browser control unit 31 and an application control unit 33. The control unit 30 can also function as a functional unit other than the browser control unit 31 and the application control unit 33.

[0048] The browser control unit 31 enables the web browser 80 to run on the dedicated operating system 70 and display various web pages. The browser control unit 31 also enables web applications to run on the web browser 80 and perform their functions. For example, the browser control unit 31 enables a capture application 84, which is a web application, to run on the web browser 80 and acquire at least any one of image data and sound data. Details of the capture application 84 will be described later.

[0049] Application control unit 33 enables applications to run on dedicated operating system 70 or in a virtual environment. Application control unit 33 enables applications with specifications corresponding to dedicated operating system 70 to run on dedicated operating system 70. Application control unit 33 also enables applications with specifications different from those corresponding to dedicated operating system 70 to run in a virtual environment. The virtual environment is built on virtual device 90 running on dedicated operating system 70. Virtual device 90 will be described later.

[0050] Storage unit 40 stores various programs and data. Storage unit 40 stores device control programs, various network applications, and various other applications. Storage unit 40 stores various data generated by control unit 30. Storage unit 40 is composed of volatile semiconductor memory such as RAM (Random Access Memory), non-volatile memory such as ROM (Read Only Memory), and flash memory. Storage unit 40 can also function as the working area of ​​control unit 30. Storage unit 40 corresponds to one example of a storage section.

[0051] The communication unit 50 is a communication interface circuit that communicates with the projector 100, cloud server 300, etc. The communication unit 50 wirelessly connects to the projector 100, etc., according to a specified communication protocol. The communication unit 50 includes a wireless communication port, such as a wireless LAN (Local Area Network) communication port. The communication unit 50 can also be wired to the projector 100, etc. The communication unit 50 has a wired connector such as a USB (Universal Serial Bus). The communication unit 50 sends various control signals and projected image data to the projector 100. The projected image data includes video data and audio data. The communication unit 50 receives various data and response signals from the cloud server 300, etc.

[0052] The projector 100 includes a projection unit 110, a built-in speaker 120, a control unit 130, a storage unit 140, and a communication unit 150.

[0053] The projection unit 110 amplifies and projects an image based on the projected image data onto the projection surface SC. The projection unit 110 projects a projected image PG onto the projection surface SC under the control of the projector control unit 130. The projection unit 110 projects dynamic images, static images, OSD images, etc., as the projected image PG based on image data contained in the projected image data. OSD images are images representing various settings, operating information, etc., related to the projector 100.

[0054] Figure 3 This shows a schematic structure of the projection unit 110. Figure 3 An example of a projection unit 110 is shown. The projection unit 110 includes a light source 111, three liquid crystal light valves 113, a light valve driving unit 115, and a projection lens 117.

[0055] Light source 111 emits light into liquid crystal light valve 113. Light source 111 includes a light source unit 111a, a reflector 111b, an integrating optical system (not shown), and a dichroic optical system (not shown). Light source unit 111a emits light. Light source unit 111a is composed of a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, etc. Light source unit 111a emits light under the control of projector control unit 130. Reflector 111b reduces the deviation in the emission direction of the light emitted from light source unit 111a. The integrating optical system reduces the brightness distribution deviation of the light emitted from light source unit 111a. Light passing through reflector 111b is incident on the dichroic optical system. The dichroic optical system separates the incident light into red, green, and blue light components.

[0056] The liquid crystal light valve 113 modulates the light emitted from the light source 111. The liquid crystal light valve 113 generates a projected image PG, etc., by modulating the light. The liquid crystal light valve 113 is constructed from a transmissive liquid crystal panel, etc., in which liquid crystal is sealed between a pair of transparent substrates. The liquid crystal light valve 113 has a rectangular pixel region 113a containing a plurality of pixels PL arranged in a matrix. In the liquid crystal light valve 113, a driving voltage is applied to the liquid crystal according to each pixel PL. Figure 3 The projection unit 110 shown has three liquid crystal light valves 113. The projector 100 is configured to use a transmissive liquid crystal panel with liquid crystal light valves 113, but is not limited thereto. The projector 100 may have a reflective liquid crystal panel or an LCOS (Liquid Crystal On Silicon) liquid crystal panel. The projector 100 may also have a structure with more than one DMD (Digital Mirror Device).

[0057] The three liquid crystal light valves 113 are a red light liquid crystal light valve 113R, a green light liquid crystal light valve 113G, and a blue light liquid crystal light valve 113B. The red light component separated by the color separation optical system is incident on the red light liquid crystal light valve 113R. The green light component separated by the color separation optical system is incident on the green light liquid crystal light valve 113G. The blue light component separated by the color separation optical system is incident on the blue light liquid crystal light valve 113B.

[0058] The light valve driving unit 115 applies a driving voltage to each pixel PL based on the projected image data received from the projector control unit 130. The light valve driving unit 115 is, for example, a control circuit. The driving voltage is supplied by a driving source (not shown). The light valve driving unit 115 may also apply a driving voltage to each pixel PL based on the projected image data corrected by the projection control unit 131 described later. When the light valve driving unit 115 applies a driving voltage to each pixel PL, each pixel PL is set to a light transmittance based on the projected image data. Light emitted from the light source 111 is modulated through the pixel region 113a. The three liquid crystal light valves 113 form a color composition image of each color light.

[0059] The projection lens 117 synthesizes and magnifies the images of each color component formed by the liquid crystal light valve 113 and projects them. The projection lens 117 projects the projected image PG onto the projection surface SC. The projected image PG is a multicolor image synthesized from the images of each color component.

[0060] Figure 2 The projector's built-in speaker 120, as shown, outputs sound based on sound data contained in the projected image data. The projector's built-in speaker 120 receives sound data from the projector control unit 130 and outputs sound corresponding to the sound data. Figure 2 The projector 100 shown has a built-in speaker 120, but is not limited thereto. The projector 100 may also have an external speaker. The external speaker outputs sound based on sound data.

[0061] The projector control unit 130 is a projector control controller that controls the various units of the projector 100. As an example, the projector control unit 130 is a processor with a CPU. The projector control unit 130 is composed of one or more processors. The projector control unit 130 functions as a projection control unit 131 by executing firmware. The projector control unit 130 can also function as a functional unit other than the projection control unit 131.

[0062] The projection control unit 131 controls the projection of the projection unit 110. Based on the image data contained in the projection image data sent from the control device 10, the projection control unit 131 causes the projection unit 110 to project a projection image PG. Based on the sound data contained in the projection image data, the projection control unit 131 causes the projector's built-in speaker 120 to output sound. The projection control unit 131 synchronizes the projection image PG projected from the projection unit 110 with the sound output from the projector's built-in speaker 120. The projection control unit 131 performs various correction processes on the image data, such as keystone correction.

[0063] The projector storage unit 140 stores firmware and various data. It stores projected image data sent from the control device 10, as well as correction data for correcting image data contained within the projected image data. The projector storage unit 140 is composed of volatile semiconductor memory such as RAM, and non-volatile memory such as ROM and flash memory.

[0064] The projector communication unit 150 is a communication interface circuit for communicating with the control device 10, external devices, etc. The projector communication unit 150 wirelessly communicates with the control device 10 and other devices according to a specified communication protocol. The projector communication unit 150 includes a wireless communication port, such as a wireless LAN communication port. The projector communication unit 150 can also be wired to communicate with the control device 10 and other devices. The projector communication unit 150 has a wired connector such as a USB connector. The projector communication unit 150 receives projected image data, control signals, etc., from the control device 10. The projector communication unit 150 sends various data, such as projector setting information, to the control device 10 and other devices.

[0065] The cloud server 300 includes a network server 310, a WebSocket server 320, a server storage device 330, and a server communication device 340. The network server 310, the WebSocket server 320, the server storage device 330, and the server communication device 340 are configured in one or more devices.

[0066] Server storage device 330 stores various programs and data. Server storage device 330 stores web pages and web applications provided by various network servers 310. Server storage device 330 is composed of magnetic recording media such as HDD (Hard Disk Drive). Server storage device 330 may also be composed of volatile semiconductor memory such as RAM, or non-volatile memory such as ROM and flash memory.

[0067] Server communication device 340 is a communication interface circuit that communicates with control device 10, etc. Server communication device 340 communicates with control device 10, etc., via wired or wireless means according to a specified communication protocol. Server communication device 340 includes a wired connector, a wireless communication port, etc. The wired connector is a USB connector, a LAN connector, etc. The wireless communication port is a Wi-Fi communication port, a Bluetooth communication port, etc. Wi-Fi and Bluetooth are registered trademarks. Server communication device 340 sends network applications, etc., to control device 10. Server communication device 340 receives various signals sent from control device 10.

[0068] Figure 4 The system structure of projection system 1 is shown. Figure 4 The control device 10, projector 100, and cloud server 300 are shown.

[0069] A dedicated operating system 70 runs in the control device 10. The dedicated operating system 70 is, for example, ChromeOS. The control unit 30 of the control device 10 enables the dedicated operating system 70 to run. Applications corresponding to the dedicated operating system 70 can run on the dedicated operating system 70. Figure 4 On the dedicated operating system 70 shown, the web browser 80 and the virtual device 90 run. Applications other than the web browser 80 and the virtual device 90 can also run on the dedicated operating system 70.

[0070] Web browser 80 is software that connects to web server 310. Web browser 80 runs on dedicated operating system 70. Web browser 80 is, for example, Chrome. Web browser 80 displays various web pages in browser window BW, which includes tab window TW. Web browser 80 retrieves web pages from web server 310. Web pages may include web applications running within the web pages. Web browser 80 displays projection settings page 81 and data retrieval page 83 in browser window BW or tab window TW. Capture application 84 is launched within data retrieval page 83.

[0071] The projection settings page 81 accepts a mirror start instruction operation. When the user performs a mirror start instruction operation on the projection settings page 81, the projection settings page 81 configures the projector 100 to perform mirroring. The projection settings page 81 is an example of a webpage. The web browser 80 obtains projection settings page information related to the projection settings page 81 from the web server 310. Based on the projection settings page information, the web browser 80 displays the projection settings page 81 in the browser window (BW) or tab window (TW). The projection settings page 81 corresponds to an example of a settings page.

[0072] Data acquisition page 83 causes capture application 84 to run and acquire shared data. Data acquisition page 83 is an example of a webpage. Web browser 80 obtains data acquisition page information related to data acquisition page 83 from web server 310. Based on the data acquisition page information, web browser 80 displays data acquisition page 83 in browser window (BW) or tab window (TW). Capture application 84 is launched on data acquisition page 83. Data acquisition page 83 causes capture application 84 to start and acquire shared data.

[0073] The capture application 84 is launched on the data acquisition page 83 displayed on the web browser 80. The web browser 80 obtains the data acquisition page 83 containing the capture application 84 from the web server 310. The capture application 84 acquires generated data output from the output unit 20 of the control device 10 as shared data. The generated data includes image data output from the display data output unit 21. The image data is the data that causes the display screen DS to be displayed on the monitor 15. The image data corresponds to an example of display data. The shared data may also include sound data output from the sound output unit 23. The shared data is sent to the projector 100 as projected image data via the streaming path SP. The capture application 84 may also send the shared data to the projection application 91. The capture application 84 corresponds to an example of a web application.

[0074] Virtual device 90 is a virtual device running on dedicated operating system 70. Virtual device 90 is an example of an emulator running on dedicated operating system 70. Virtual device 90 functions as a device that enables the running of an operating system different from dedicated operating system 70 on dedicated operating system 70. Virtual device 90 provides a virtual environment on dedicated operating system 70. Virtual device 90 is, for example, a device running Android OS. Applications running on a different operating system than dedicated operating system 70 can be launched on virtual device 90. Applications running on a different operating system than dedicated operating system 70 are Android applications. Figure 4 The projection application 91 shown is an example of an Android application.

[0075] Projection application 91 is software that controls projector 100. Projection application 91 corresponds to an example of an application. Projection application 91 is pre-stored in storage unit 40. Projection application 91 causes projector 100 to project various projected images PG. Projection application 91 sends various control signals to projector 100, cloud server 300, etc. Projection application 91 receives response signals from projector 100, cloud server 300, etc. Projection application 91 causes capture application 84 to acquire shared data. Projection application 91 causes capture application 84 to send the shared data as projected image data to projector 100. Projection application 91 can also cause capture application 84 to send shared data to projection application 91.

[0076] The projection application 91 controls the display of the web browser 80. The projection application 91 controls the display mode of the browser window (BW) or tab window (TW) that displays the projection settings page 81 and the data acquisition page 83. By controlling the display mode of the browser window (BW) or tab window (TW), the projection application 91 allows the user to check the status of the image, etc. The projection application 91 controls the display mode by sending setting information such as the control settings for the display mode of the browser window (BW) or tab window (TW) to the web browser 80.

[0077] The projection application 91 controls the display mode of its UI screen 170 based on various signals sent from the web browser 80. The UI screen 170 will be described later. By controlling the display mode of the UI screen 170, the projection application 91 enables the user to check the status of the image, etc.

[0078] The projection application 91, running on the virtual device 90, sends and receives various control signals to and from the dedicated operating system 70 via various APIs (Application Programming Interfaces). The APIs vary depending on the version of the operating system running on the virtual device 90. Depending on the version of the operating system running on the virtual device 90, the projection application 91 may sometimes have difficulty controlling the projector 100. The projection application 91 uses the functions of the web browser 80 and the cloud server 300 to send shared data to the projector 100 as projected image data.

[0079] Figure 5 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 5 The display screen DS shown illustrates the screen displayed when the control device 10 starts the projection application 91. The display screen DS includes the UI screen 170 of the projection application 91. UI is short for User Interface. Figure 5 The UI screen 170 shown is displayed in the browser window BW of web browser 80.

[0080] UI screen 170 displays multiple icons, etc. UI screen 170 includes projector name display 171, mirror acceptance icon 172, photo display acceptance icon 173, document display acceptance icon 174, camera image display acceptance icon 175, remote control function acceptance icon 176, multi-projection acceptance icon 177, cut-off acceptance icon 178, and end acceptance icon 179.

[0081] The projector name display unit 171 displays the projector name of the projector 100 connected to the control device 10. The projector name display unit 171 displays the projector name based on the model information of the projector 100 obtained from the projector 100.

[0082] The mirror acceptance icon 172 accepts the user's mirror instruction operation. When the user performs a specified operation on the mirror acceptance icon 172, the mirror acceptance icon 172 accepts the mirror instruction operation. When accepting the mirror instruction operation, the projection application 91 generates a mirror control signal, etc., corresponding to the mirror instruction operation. The mirror acceptance icon 172 includes a switch icon 172a.

[0083] The switch icon 172a indicates whether the projector 100 is performing mirroring. By checking the display of the switch icon 172a, the user can determine whether the projector 100 is performing mirroring. Figure 5 The switch icon 172a shown indicates that mirroring has not been performed. Switch icon 172a can accept mirroring instructions from the user.

[0084] The photo display acceptance icon 173 accepts the user's photo display instruction. When the user performs a specified operation on the photo display acceptance icon 173, the photo display acceptance icon 173 accepts the photo display instruction. When the photo display instruction is accepted, the projection application 91 performs photo display processing on the photo image stored in the projection control device 10 of the projector 100. The projection application 91 sends the photo image data for displaying the photo image to the projector 100, causing the projector 100 to project the photo image as the projected image PG.

[0085] The document display acceptance icon 174 accepts the user's document display instruction. When the user performs a specified operation on the document display acceptance icon 174, the document display acceptance icon 174 accepts the document display instruction. When the document display instruction is accepted, the projection application 91 performs document display processing on PDF files, etc., stored in the projection control device 10 of the projector 100. As an example, the projection application 91 sends PDF data for displaying a PDF image to the projector 100, causing the projector 100 to project the PDF image as the projected image PG.

[0086] The camera image display acceptance icon 175 accepts the user's camera image display instruction. When the control device 10 has a camera or other imaging device, the camera image display acceptance icon 175 can be operated. When the user performs a specified operation on the camera image display acceptance icon 175, the camera image display acceptance icon 175 accepts the camera image display instruction. When the camera image display instruction is accepted, the projection application 91 performs camera image display processing to cause the projector 100 to project the captured image from the imaging device. The projection application 91 sends the captured data for displaying the captured image to the projector 100, causing the projector 100 to project the captured image as the projected image PG.

[0087] The remote control function acceptance icon 176 accepts the user's remote control function action instructions. When the user performs a specified operation on the remote control function acceptance icon 176, the remote control function acceptance icon 176 accepts the remote control function action instruction. When the remote control function action instruction is accepted, the projection application 91 displays the remote control image representing the remote control operating the projector 100 on the projected image PG. When the user inputs an operation on the remote control image, the projector 100 can be controlled according to the input operation.

[0088] The multi-projection acceptance icon 177 accepts multiple projection instructions from the user. When multiple projectors 100 are connected to the control device 10, the multi-projection acceptance icon 177 can be operated. When the user performs a specified operation on the multi-projection acceptance icon 177, the multi-projection acceptance icon 177 accepts the multiple projection instructions. The projection application 91 performs multiple projections, causing the multiple projectors 100 connected to the control device 10 to project the projected image PG onto the projection surface SC.

[0089] The cut-off acceptance icon 178 accepts the user's cut-off instruction. When the user performs a prescribed operation on the cut-off acceptance icon 178, the cut-off acceptance icon 178 accepts the cut-off instruction. When the cut-off instruction is accepted, the projection application 91 cuts off the communication connection between the control device 10 and the projector 100.

[0090] The End Acceptance icon 179 accepts the user's application termination request. When the user performs a specified operation on the End Acceptance icon 179, the End Acceptance icon 179 accepts the application termination request. When the application termination request is accepted, the projected application 91 stops running. The UI screen 170 becomes non-displayable.

[0091] Figure 4 The streaming path SP shown is the path for real-time communication between the web browser 80 and the projector 100. As an example, the streaming path SP uses WebRTC technology for communication. The web browser 80 can send shared data to the projector 100 via the streaming path SP. Because the web browser 80 sends shared data to the projector 100 via the streaming path SP, the control device 10 can send the shared data to the projector 100 without degrading its quality. The streaming path SP corresponds to an example of a browser communication path.

[0092] The TCP / IP communication path TP is a communication path that uses TCP / IP (Transmission Control Protocol / Internet Protocol). The TCP / IP communication path TP and the streaming path SP are different communication paths. Projection application 91 sends various control signals to projector 100 via the TCP / IP communication path TP. Projection application 91 also receives control signals from projector 100 via the TCP / IP communication path TP. Projection application 91 can also send shared data to projector 100 via the TCP / IP communication path TP. When sending shared data to projector 100 via the TCP / IP communication path TP, quality degradation may occur in the shared data depending on the condition of the virtual device 90. It is preferable to send shared data to projector 100 via the streaming path SP. The TCP / IP communication path TP corresponds to an example of a signal communication path.

[0093] The projector communication unit 150 of the projector 100 has a streaming communication unit 151 and a signal communication unit 153.

[0094] The streaming communication unit 151 is connected to the capture application 84 via the streaming path SP. The streaming communication unit 151 corresponds to WebRTC technology. The streaming communication unit 151 receives shared data sent from the capture application 84 through real-time communication.

[0095] The signal communication unit 153 is communicatively connected to the projection application 91 via the TCP / IP communication path TP. The signal communication unit 153 receives control signals from the projection application 91. The signal communication unit 153 sends control signals generated by the projector 100 to the projection application 91. By transmitting and receiving control signals between the projection application 91 and the projector 100 through the signal communication unit 153, the projection application 91 can control the projector 100.

[0096] The cloud server 300 includes a network server 310 with a server storage unit 330a. The server storage unit 330a is part of the server storage device 330. The server storage unit 330a stores web page information related to various web pages. This web page information includes network application information related to various web applications. The network server 310 provides the web page information stored in the server storage unit 330a to the web browser 80.

[0097] WebSocket server 320 receives various control signals sent from projection application 91 via TCP / IP communication path TP. These control signals include webpage requests. For example, a webpage request is a signal instructing web server 310 to send a webpage containing a webpage from web browser 80. Upon receiving a webpage request, WebSocket server 320 sends the webpage request to web server 310. Based on the webpage request, web server 310 sends webpage information to web browser 80.

[0098] exist Figure 4In the projection system 1 shown, the cloud server 300 includes a network server 310 that provides various web pages, but is not limited to these. Projection setting page information related to the projection setting page 81 and data acquisition page information related to the data acquisition page 83 can be pre-stored in the storage unit 40 of the control device 10 or the projector storage unit 140 of the projector 100. Projection setting page information and data acquisition page information are examples of web page information. When the projection setting page information, etc., is stored in the storage unit 40, the web browser 80 retrieves the projection setting page information, etc., from the storage unit 40. In this case, the control unit 30 has virtual server functionality equivalent to the cloud server 300. When the projection setting page information, etc., is stored in the projector storage unit 140, the web browser 80 retrieves the projection setting page information, etc., from the projector 100. In this case, the projector control unit 130 has virtual server functionality equivalent to the cloud server 300. When the projection setting page information, etc., is pre-stored in the storage unit 40 or the projector storage unit 140, the projection system 1 may not have a cloud server 300.

[0099] Figure 6 The control flow executed by projection system 1 is shown. Figure 6 The control flow executed by the control device 10, the projector 100, and the cloud server 300 is shown. The control flow executed by the control device 10 corresponds to an example of an information processing method. The control device 10 executes the control flow to cause the projector 100 to project a mirror image of the display screen DS displayed on the monitor 15.

[0100] Projection application 91 is launched in step S101. When the input unit 17 of the control device 10 receives a specified operation from the user, the control device 10 launches projection application 91. Projection application 91 is launched on virtual device 90. Projection application 91 executes in the virtual environment. When projection application 91 is launched, UI screen 170 for operating projection application 91 is displayed on display screen DS. UI screen 170 corresponds to an example of an application window.

[0101] Before or after the projection application 91 is launched, the web browser 80 is launched in step S121. Figure 6 This illustrates the case where the web browser 80 is launched before the projection application 91 is launched. When the user performs a specified input operation using the input unit 17, the browser control unit 31 receives a browser launch instruction and executes browser launch processing to launch the web browser 80. The web browser 80 is launched before the projection application 91 receives a mirroring instruction operation. When both the projection application 91 and the web browser 80 are launched... Figure 5 The UI screen 170 and the browser window BW shown are displayed on the display screen DS.

[0102] After startup, the projection application 91 establishes a communication connection with the projector 100 in step S102. The projection application 91 sends control signals such as connection requests to the projector 100 via the TCP / IP communication path TP to establish a communication connection with the projector 100.

[0103] After the projection application 91 is started, the projector 100 establishes a communication connection with the control device 10 in step S141. The projector 100 receives control signals such as connection requests from the projection application 91 via the TCP / IP communication path TP. The projector 100 sends various data such as model information to establish a communication connection with the control device 10.

[0104] After establishing a communication connection with the projector 100, the projection application 91 handles the mirroring instruction operation in step S103. When the projection application 91 starts, as... Figure 5 As shown, a UI screen 170 is displayed on the monitor 15. When the user performs a mirror instruction operation on the mirror acceptance icon 172 included in the UI screen 170 displayed on the monitor 15, the mirror acceptance icon 172 accepts the mirror instruction operation.

[0105] Figure 7 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 7 The displayed screen DS includes the browser window BW of the web browser 80 and the UI screen 170 of the projection application 91. Figure 7 The displayed screen DS shows the screen when the user performs a mirror instruction operation on the mirror acceptance icon 172.

[0106] When a user performs a mirror instruction operation on the mirror acceptance icon 172, such as Figure 7 As shown, the display mode of switch icon 172a is changed. The application control unit 33 performs a display mode change process. Switch icon 172a is slid to the right, switching to an execution display indicating that mirroring is being performed. The execution display corresponds to an example of execution information.

[0107] like Figure 6 As shown, when a mirroring instruction operation is received, the projection application 91 sends a projection settings page request to the cloud server 300 in step S104. Sending the projection settings page request through the projection application 91 causes the projection settings page 81 to be displayed in the tab window TW of the web browser 80. The projection settings page window 81w containing the projection settings page 81 is displayed as the tab window TW. The projection application 91 displays the projection settings page window 81w as the tab window TW based on the mirroring instruction operation. The projection application 91 can send the projection settings page request directly to the cloud server 300, or it can send it to the cloud server 300 via the web browser 80.

[0108] When the projection application 91 sends a projection settings page request, the cloud server 300 receives the projection settings page request in step S161. The network server 310 included in the cloud server 300 receives the projection settings page request. Based on the projection settings page address contained in the projection settings page request, the network server 310 obtains the projection settings page information.

[0109] After receiving the projection settings page request, cloud server 300 sends the projection settings page information to web browser 80 in step S162. Web server 310, included in cloud server 300, sends the projection settings page information to web browser 80.

[0110] When the cloud server 300 sends the projection settings page information, the web browser 80 receives the projection settings page information in step S122.

[0111] When the projection settings page information is received, the web browser 80 displays the projection settings page 81 in step S123. Based on the projection settings page information, the browser control unit 31 performs projection settings page display processing, which displays the projection settings page window 81w containing the projection settings page 81 as a tab window TW. The web browser 80 displays the projection settings page window 81w of the first display size as a tab window TW.

[0112] Figure 8 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 8 The displayed screen DS shows the screen that the browser control unit 31 displays when it performs projection setting page display processing based on projection setting page information.

[0113] Figure 8 The web browser 80 shown displays the projection settings page window 81w as a tab window TW. The projection settings page 81w displayed in the projection settings page window 81w accepts input operations related to mirroring the display screen DS. The projection settings page window 81w contains a mirroring start icon 81a.

[0114] The projection settings page window 81w, displayed as the tab window TW, is shown at the first display size. For example, the first display size is the full screen size of the monitor 15. The first display size can also be the size of the browser window BW when the web browser 80 is launched in step S121. The first display size is not limited to any size that can display the mirrored start icon 81a.

[0115] like Figure 6As shown, after displaying the projection settings page 81, the web browser 80 accepts the mirror start indication operation in step S124. When the user performs a mirror start indication operation on the mirror start icon 81a, the projection settings page 81 accepts the mirror start indication operation.

[0116] When the image start instruction is received, the web browser 80 sends the image start instruction to the cloud server 300 in step S125.

[0117] When the web browser 80 sends a mirror start instruction, the cloud server 300 receives the mirror start instruction in step S163.

[0118] When a mirroring start instruction is received, the cloud server 300 sends a mirroring start instruction to the projection application 91 in step S164.

[0119] When the cloud server 300 sends a mirror start instruction, the projection application 91 receives the mirror start instruction in step S105.

[0120] Upon receiving the mirror start instruction, the projection application 91 sends a data retrieval page request and data retrieval page settings to the cloud server 300 in step S106. By sending the data retrieval page request and data retrieval page settings, the projection application 91 displays the data retrieval page window 83w, which contains the data retrieval page 83, as a tab window TW. The web browser 80, based on the mirror start instruction operation, causes the data retrieval page window 83w to be displayed as a tab window TW.

[0121] The data retrieval page settings are the display settings when the web browser 80 displays the data retrieval page window 83w as a tab window TW. The projection application 91 specifies the display size of the tab window TW when the web browser 80 displays the data retrieval page window 83w as a tab window TW by sending the data retrieval page settings to the cloud server 300. The projection application 91 also sends the data retrieval page settings to the cloud server 300, causing the data retrieval page window 83w, displayed as a tab window TW, to be displayed at a second display size smaller than the first display size.

[0122] When the projection application 91 sends a data retrieval page request and data retrieval page settings, the cloud server 300 receives the data retrieval page request and data retrieval page settings in step S165. The network server 310 included in the cloud server 300 receives the data retrieval page request and data retrieval page settings. The network server 310 retrieves the data retrieval page information based on the data retrieval page address contained in the data retrieval page request.

[0123] After receiving the data retrieval page request and the data retrieval page settings, the cloud server 300 sends the data retrieval page information to the web browser 80 in step S166. The web server 310 included in the cloud server 300 also sends the data retrieval page information to the web browser 80.

[0124] When the cloud server 300 sends data to obtain page information, the web browser 80 receives data to obtain page information in step S126.

[0125] When the data retrieval page information is received, the web browser 80 displays the data retrieval page 83 as a tab window TW in step S127. Based on the data retrieval page information, the browser control unit 31 performs retrieval page display processing to display the data retrieval page window 83w containing the data retrieval page 83. Through the retrieval page display processing, the web browser 80 displays the data retrieval page window 83w as a tab window TW at a second display size smaller than the first display size.

[0126] When the data acquisition page window 83w is displayed as a tab window TW on the display screen DS, it is preferable to keep the projection settings page window 81w hidden. When the receiving mirror starts indicating an operation, the web browser 80 instructs the browser control unit 31 to perform a settings window tab-hiding process that closes the projection settings page window 81w, which will be displayed as a tab window TW. The settings window tab-hiding process corresponds to an example of the tab window-hiding process. The browser control unit 31 performs the settings window tab-hiding process and closes the projection settings page window 81w. By closing the projection settings page window 81w, multiple operations and accidental operations by the user on the projection settings page window 81w are prevented.

[0127] After displaying the data acquisition page 83, the web browser 80 displays a selection screen 180 on the monitor 15 in step S128. The selection screen 180 allows the user to select the display screen DS projected by the projector 100. The web browser 80 may also change the display size of the projection setting page window 81w, which is displayed as the tab window TW, while displaying the selection screen 180.

[0128] Figure 9 This represents an example of a display screen DS. The display screen DS is shown on monitor 15. Figure 9 The displayed screen DS shows the screen DS when the web browser 80 displays the selection screen 180. Figure 9 The displayed screen DS shows selection screen 180 and browser window BW. Selection screen 180 is displayed on web browser 80.

[0129] Selection screen 180 includes a selection image 181, a share icon 183, and a share cancel icon 185. Selection screen 180 displays more than one selection image 181. Figure 9 The selection screen 180 shown includes a first selection image 181a as the selection image 181. When multiple selection images 181 are displayed in the selection screen 180, the multiple selection images 181 are arranged and displayed within the selection screen 180.

[0130] Image selection 181 accepts user input. Image selection 181 is a captured image of the display screen DS projected by projector 100. The first image selection 181a is... Figure 9 The image shown is a scaled-down version of the display screen DS. When the user performs a specified selection operation on the selection image 181, the selected image 181 is selected as the display screen DS projected by the projector 100.

[0131] The shared icon 183 accepts user input. When the user performs a specified operation on the shared icon 183, the web browser 80 accepts the selection instruction. The selected image 181, after the selection operation, is selected as the display screen DS projected by the projector 100.

[0132] The share cancel icon 185 accepts user input. When the user performs a specified operation on the share cancel icon 185, the selection screen 180 becomes inactive, and the mirroring is canceled. The control device 10 does not send sharing data to the projector 100.

[0133] When the selection screen 180 is displayed on the display screen DS, it is preferable to minimize the display of the projection settings page window 81w, which is displayed as a tab window TW. When the web browser 80 displays the selection screen 180 on the display screen DS, the web browser 80 is controlled to minimize the display of the projection settings page window 81w. By minimizing the display of the projection settings page window 81w, the visibility of the selection screen 180 and the browser window BW is improved.

[0134] like Figure 6 As shown, after displaying the selection screen 180, the web browser 80 accepts the selection instruction operation in step S129. When the user performs a specified operation on the share icon 183 within the selection screen 180, the web browser 80 accepts the selection instruction operation. When the selection instruction operation is accepted, the web browser 80 sets the selection screen 180 to be hidden.

[0135] When the selection instruction is received, the web browser 80 acquires shared data in step S130. The web browser 80 acquires the generated data output from the output unit 20 as shared data through the function of the capture application 84 running within the data acquisition page 83. The shared data includes image data output from the display data output unit 21. The shared data may also include audio data output from the audio output unit 23.

[0136] When retrieving shared data, the web browser 80 performs page display processing, causing the data retrieval page window 83w, which is currently displayed as a tab window TW, to be displayed at a second display size. By displaying the data retrieval page window 83w at a second display size, the proportion of the data retrieval page window 83w occupied by the web browser 80 within the display screen DS can be reduced. This allows the user to see other parts of the screen without being obstructed by the data retrieval page window 83w.

[0137] When shared data is acquired, the web browser 80 sends the shared data to the projector 100 in step S131. The data acquisition page 83w, displayed in the data acquisition page window 83, performs a mirror-based start instruction operation, executes the shared data sending process, and sends the shared data to the projector 100. The shared data includes image data. The web browser 80 sends the shared data to the projector 100 via the streaming path SP. The data acquisition page 83 corresponds to an example of the data sending page. The data acquisition page window 83w corresponds to an example of the data sending window. The control device 10 executes a mirroring process to cause the projector 100 to project the display screen DS. The mirroring process corresponds to an example of the projection process. When sending shared data, the web browser 80 enters step B.

[0138] When the web browser 80 sends shared data to the projector 100, the projector 100 receives the shared data in step S142. The projector communication unit 150 of the projector 100 receives the shared data sent via the streaming path SP. The projector control unit 130 obtains the shared data via the projector communication unit 150.

[0139] When shared data is received, projector 100 performs mirroring in step S143. Projector 100 acquires the image data contained in the shared data. Projection control unit 131 of projector control unit 130 sends the image data to projection unit 110. Projection control unit 131 causes projection unit 110 to project the image data. Projection unit 110 projects a projection image PG based on the image data onto projection surface SC. When the shared data contains sound data, projector 100 acquires the sound data contained in the shared data. Projection control unit 131 sends the sound data to projector built-in speaker 120. Projector built-in speaker 120 outputs sound based on the sound data. Based on the shared data containing image data and sound data, projector 100 displays the display screen DS shown on display 15 of control device 10 and outputs sound based on the sound data. Projector 100 performs mirroring of the display screen DS based on the shared data containing image data.

[0140] Figure 10 The projected image PG is shown projected onto the projection surface SC. Figure 10 This shows the projected image PG projected onto the projection surface SC when the projector 100 performs mirroring. The projected image PG is the same image as the display screen DS displayed on the monitor 15. The display screen DS, which is the same as the projected image PG, is displayed on the monitor 15.

[0141] The projected image PG includes a browser window BW and a tab window TW. The tab window TW is a data acquisition page window 83w containing data acquisition page 83. The data acquisition page window 83w, displayed as tab window TW, is shown at a second display size smaller than the first display size. By displaying the data acquisition page window 83w at the second display size, the user can easily see the browser window BW. This is displayed on the monitor 15. Figure 10 The projected image PG shown is the same as the displayed screen DS. The user can easily see the browser window BW.

[0142] The data acquisition page window 83w has a close button 83a. When the user performs a specified input operation on the close button 83a, the data acquisition page window 83w is closed. When the data acquisition page window 83w is closed, the acquisition of shared data and the transmission of shared data to the projector 100 are stopped.

[0143] While the control device 10 is transmitting shared data, the user can input data onto the UI screen 170 or the data acquisition page window 83w. The user can stop the mirroring process by performing specified input operations on the UI screen 170 or the data acquisition page window 83w.

[0144] Figure 11 This represents an example of a control flow executed by the control device 10. Figure 11 This illustrates an example of the control flow when a web browser 80 sends shared data to a projector 100. Figure 11 Show Figure 6 The control flow of the projection application 91 in process A and the control flow of the web browser 80 in process B are shown as an example.

[0145] After sending a data retrieval page request or setting a data retrieval page, the projection application 91 sends an action confirmation signal to the web browser 80 in step S201. The projection application 91 sends the action confirmation signal to the web browser 80 at a predetermined time. The action confirmation signal corresponds to an example of a confirmation signal.

[0146] When the projection application 91 sends an action confirmation signal, the web browser 80 receives the action confirmation signal in step S221. The data acquisition page window 83w of the web browser 80 receives the action confirmation signal.

[0147] Upon receiving an action confirmation signal, the web browser 80 sends a confirmation response signal to the projection application 91 in step S222. The data acquisition page window 83w sends a confirmation response signal to the projection application 91 within a specified time after receiving the action confirmation signal. The confirmation response signal corresponds to an example of a response signal.

[0148] When the web browser 80 sends an acknowledgment signal, the projection application 91 receives the acknowledgment signal in step S202. The projection application 91 determines that the data acquisition page window 83w is running by receiving the acknowledgment signal.

[0149] When the projector 100 performs mirroring, the projection application 91 sends an action confirmation signal at a predetermined time. The data acquisition page window 83w returns an acknowledgment response signal in response to the action confirmation signal at a predetermined time. The projection application 91 and the data acquisition page window 83w perform polling when sending shared data to the projector 100.

[0150] During polling, the web browser 80 sometimes accepts a close operation in step S223. When the user performs a specified input operation on the close button 83a of the data retrieval page window 83w, a close operation is accepted to close the data retrieval page window 83w. The close operation corresponds to an example of the browser window stopping operation.

[0151] When the closing operation is accepted, the web browser 80 will close the data acquisition page window 83w, which will be displayed as tab window TW, in step S224. When the data acquisition page window 83w is closed, the capture application 84 stops functioning. Acquisition and transmission of shared data cease. Shared data is not sent to the projector 100, and mirroring processing stops.

[0152] After the web browser 80 closes the data acquisition page window 83w, the projection application 91 sends an action confirmation signal in step S203. Figure 11 In the control flow shown, the projection application 91 cannot recognize that the user has closed the data acquisition page window 83w. Independent of the state of the data acquisition page window 83w, the projection application 91 sends an action confirmation signal to the web browser 80 at a predetermined time.

[0153] After sending the action confirmation signal, the projection application 91 does not receive a confirmation response signal within a specified time in step S204. During operation, the data acquisition page window 83w sends a confirmation response signal to the projection application 91 in response to the action confirmation signal within a specified time. When the data acquisition page window 83w is closed, it does not send a confirmation response signal to the projection application 91. If no confirmation response signal is received within the specified time, the projection application 91 determines that the data acquisition page window 83w has been closed. The projection application 91 then determines that mirroring processing has stopped.

[0154] If no confirmation response signal is received, the projection application 91 switches the display mode of the UI screen 170 in step S205. When it is determined that the mirroring process has stopped, the application control unit 33 performs a display mode change process.

[0155] Figure 12 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 12 The displayed screen DS shows the screen when the data acquisition page window 83w, which was originally displayed as the tab window TW, is closed. Figure 12 The displayed screen DS shows the UI screen 170 and the browser window BW. The UI screen 170 is displayed on the web browser 80.

[0156] When the data acquisition page window 83w is closed, the application control unit 33 slides the switch icon 172a to the left and changes it to a stop display indicating that mirroring processing has not been performed. Stop display corresponds to an example of a stop message. Display mode change processing corresponds to an example of a display change process. The projection application 91 changes to a stop display in response to the user's closing operation of the data acquisition page window 83w. By linking the state of the data acquisition page window 83w with the display mode of the projection application 91, the user can reliably identify when mirroring processing has stopped.

[0157] The control flow of the control device 10 includes: starting the projection application 91 that controls the projector 100 and displaying the UI screen 170 for operating the projection application 91; establishing a communication connection with the projector 100; displaying a data acquisition page window 83w related to projection, including a data acquisition page 83, on the web browser 80 based on a data acquisition page request from the projection application 91; sending the image data displayed on the display screen DS to the projector 100 via the web browser 80 and performing mirroring processing to project the display screen DS onto the projector 100; displaying an execution display on the UI screen 170 indicating that mirroring processing is in progress; accepting a close operation to close the data acquisition page window 83w; stopping the mirroring processing and closing the data acquisition page window 83w based on the close operation; and performing a display mode change process to change the execution display to a stop display indicating that mirroring processing has not been performed.

[0158] The status of the data acquisition page window 83w and the display method of the projection application 91 are consistent in the display without mirroring. The user can reliably see the situation where the control device 10 has not performed mirroring.

[0159] The control flow of the control device 10 preferably includes: when performing mirroring, causing the projection application 91 to send an action confirmation signal to the data acquisition page window 83w at a predetermined time; and when the projection application 91 does not receive a confirmation response signal for the action confirmation signal from the data acquisition page window 83w, performing a display mode change process.

[0160] The projection application 91 sends an action confirmation signal, and the data acquisition page window 83w processes the confirmation response signal in response to the action confirmation signal. Thus, the projection application 91 can determine that the data acquisition page window 83w has been closed.

[0161] Figure 13 An example of a control flow executed by control device 10 is shown. Figure 13 This illustrates an example of the control flow when a web browser 80 sends shared data to a projector 100. Figure 13 Show Figure 6 The control flow of the projection application 91 in process A and the control flow of the web browser 80 in process B are shown as an example. Figure 13 In the control flow shown, the projection application 91 does not send an action confirmation signal to the web browser 80. Figure 13 Showing with Figure 11 Different control processes.

[0162] When sending shared data to the projector 100, the web browser 80 sometimes accepts a close operation in step S321. When the user performs a specified input operation on the close button 83a of the data acquisition page window 83w, a close operation is accepted to close the data acquisition page window 83w.

[0163] When a close operation is accepted, the web browser 80 sends a projection stop signal to the projection application 91 in step S322. If the capture application 84 running on the data acquisition page 83 has the function of sending a projection stop signal, the capture application 84 determines that a close operation has been accepted. When a close operation is determined to be accepted, the capture application 84 sends a projection stop signal to the projection application 91. The projection stop signal is a signal indicating that the data acquisition page window 83w is closed and the mirroring process has ended. The projection stop signal corresponds to an example of a stop signal.

[0164] When the web browser 80 sends a projection stop signal, the projection application 91 receives the projection stop signal in step S301. Upon receiving the projection stop signal, the projection application 91 closes the data acquisition page window 83w, indicating that the mirroring process has ended.

[0165] When the closing operation is accepted, the web browser 80 will close the data acquisition page window 83w, which will be displayed as tab window TW, in step S323. When the data acquisition page window 83w is closed, the capture application 84 stops functioning. Acquisition and transmission of shared data cease. Shared data is not sent to the projector 100, and mirroring processing stops.

[0166] When a projection stop signal is received, the projection application 91 switches the display mode of the UI screen 170 in step S302. When it is determined that the mirroring process has stopped, the application control unit 33 performs a display mode change process. The application control unit 33 slides the switch icon 172a to the left, changing it to a stop display indicating that mirroring processing has not been performed. The projection application 91 changes to a stop display corresponding to the user's closing operation of the data acquisition page window 83w. By linking the state of the data acquisition page window 83w with the display mode of the projection application 91, the user can reliably identify when mirroring processing has stopped.

[0167] The control flow of the control device 10 preferably includes: when a closing operation is accepted, the capture application 84 running in the data acquisition page window 83w sends a projection stop signal indicating the end of mirroring to the projection application 91; and causes the projection application 91 to perform a display mode change process.

[0168] By capturing the application 84 and sending a projection stop signal to the projection application 91, the projection application 91 can determine that the mirroring process has ended.

[0169] Figure 14 An example of a control flow executed by control device 10 is shown. Figure 14 This illustrates an example of the control flow when a web browser 80 sends shared data to a projector 100. Figure 14 Show Figure 6 The control flow of the projection application 91 in process A and the control flow of the web browser 80 in process B are shown as an example.

[0170] Figure 14 This illustrates the control flow when a user inputs an action on the switch icon 172a of the UI screen 170 during the process of sending shared data in the web browser 80. Figure 14 This illustrates the control flow that stops the mirroring process based on user input.

[0171] In step S401, the projection application 91 accepts a projection stop operation. The user performs a specified projection stop operation on the mirror acceptance icon 172 or the switch icon 172a on the UI screen 170. The projection stop operation is an input operation that stops the mirroring process. The projection stop operation corresponds to an example of a stop operation. For example, the user performs a... Figure 10 The switch icon 172a shown is a display change operation that slides from right to left. This display change operation corresponds to one example of a change operation. The user can also perform a click operation by tapping the mirror acceptance icon 172, which includes the execution display, as a projection stop operation. When the user performs a projection stop operation, the projection application 91 accepts the projection stop operation.

[0172] When a projection stop operation is accepted, the projection application 91 switches the display mode in step S402. Based on the projection stop operation, the projection application 91 switches the switch icon 172a from an active display to a stopped display. By switching the display of the switch icon 172a, the user can confirm that a projection stop operation has been accepted.

[0173] When a projection stop operation is received, projection application 91 sends a projection stop indication signal to web browser 80 in step S403. Projection application 91 generates a projection stop indication signal. The projection stop indication signal is a signal that causes web browser 80 to close the data acquisition page window 83w displayed as a tab window TW and to stop the transmission of shared data. The projection stop indication signal corresponds to an example of a browser stop signal. Projection application 91 sends the generated projection stop indication signal to web browser 80.

[0174] When the projection application 91 sends a projection stop indication signal, the web browser 80 receives the projection stop indication signal in step S421. The data acquisition page window 83w of the web browser 80 receives the projection stop indication signal.

[0175] Upon receiving a projection stop indication signal, the web browser 80 closes the data acquisition page window 83w in step S422. The web browser 80 closes the data acquisition page window 83w based on the projection stop indication signal. When the data acquisition page window 83w is closed, sending shared data to the projector 100 stops, and mirroring processing ceases.

[0176] Figure 15 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 15 The displayed screen DS shows the screen when the user stops the projection of UI screen 170.

[0177] When the user stops the projection of the UI screen (170°), such as Figure 15 As shown, sliding the switch icon 172a to the left switches it to off display. Additionally, as... Figure 10 The data retrieval page window 83w displayed as the tab window TW is closed and becomes non-displayed. The user can confirm that mirroring processing has stopped by seeing the display screen DS. While the data retrieval page window 83w, displayed as the tab window TW, is closed, the web browser 80 maintains the display of the browser window BW, which does not contribute to the transmission of shared data.

[0178] The control flow of the control device 10 includes: starting the projection application 91 that controls the projector 100 and displaying the UI screen 170 for operating the projection application 91; establishing a communication connection with the projector 100; displaying a data acquisition page window 83w related to projection, including a data acquisition page 83, in the web browser 80 based on a data acquisition page request from the projection application 91; sending the image data displayed on the display screen DS to the projector 100 via the web browser 80 and performing mirroring processing to project the display screen DS onto the projector 100; displaying an execution display indicating that mirroring processing is being performed in the UI screen 170; accepting a projection stop operation to stop the mirroring processing in the UI screen 170; and stopping the mirroring processing and closing the data acquisition page window 83w based on the projection stop operation.

[0179] The projection application 91 closes the data acquisition page window 83w based on the projection stop operation, making it easy for users to determine that the mirroring process has stopped.

[0180] The projection stop operation is a display change operation that changes the display from performing a mirroring operation to indicating that the mirroring process has not been performed. The control flow of the control device 10 preferably includes: based on the display change operation, causing the projection application 91 to generate a projection stop indication signal that closes the data acquisition page window 83w; causing the projection application 91 to send the projection stop indication signal to the web browser 80; and based on the projection stop indication signal, causing the web browser 80 to end the mirroring process.

[0181] By sending a projection stop indication signal to the web browser 80 through the projection application 91, the web browser 80 can close the data acquisition page window 83w.

[0182] Figure 16 An example of a control flow executed by control device 10 is shown. Figure 16 This illustrates an example of the control flow when a web browser 80 sends shared data to a projector 100. Figure 16 Show Figure 6 The control flow of the projection application 91 in process A and the control flow of the web browser 80 in process B are shown as an example.

[0183] Figure 16 This illustrates the control flow when a user inputs an end-acceptance icon 179 on the UI screen 170 during the process of sending shared data in the web browser 80. Figure 16 This illustrates the control flow that stops the mirroring process based on user input.

[0184] In step S501, the projection application 91 accepts the application termination operation. The user performs a prescribed projection stop operation by clicking the cut-off acceptance icon 178 or the end acceptance icon 179 on the UI screen 170. As an example, the user clicks the cut-off acceptance icon 178 or the end acceptance icon 179 on the UI screen. Figure 10 The "End Acceptance" icon 179 shown indicates the application is terminated. The "End Acceptance" icon 179 corresponds to an example of an application termination icon. The application termination operation corresponds to an example of an termination instruction operation. When the user terminates the application, the projected application 91 accepts the application termination operation.

[0185] When the application finishes its operation, the projection application 91 sends a projection application stop signal to the web browser 80 in step S502. The projection application 91 generates a projection application stop signal. The projection application stop signal indicates that the projection application 91 has ended. The projection application stop signal corresponds to an example of an application stop signal. The projection application 91 sends the generated projection application stop signal to the web browser 80.

[0186] When the application ends, the projection application 91 closes the UI screen 170 in step S503. The projection application 91 closes the UI screen 170 according to the application end operation, making the UI screen 170 non-displayable. By closing the UI screen 170, the user can confirm that the application end operation has been accepted.

[0187] When the projection application 91 sends a projection application stop signal, the web browser 80 receives the projection application stop signal in step S521. The data acquisition page window 83w of the web browser 80 receives the projection application stop signal.

[0188] When a projection application stop signal is received, the web browser 80 closes the data acquisition page window 83w in step S522. The web browser 80 closes the data acquisition page window 83w based on the projection application stop signal. When the data acquisition page window 83w is closed, sending shared data to the projector 100 stops, and mirroring processing stops.

[0189] Figure 17 An example of a display screen DS is shown. The display screen DS is displayed on monitor 15. Figure 17 The displayed screen DS shows the screen when the user has performed an application termination operation on UI screen 170.

[0190] When the user performs an application termination operation on UI screen 170, such as Figure 17 As shown, UI screen 170 is turned off and becomes non-displayable. Additionally, as... Figure 10 The data retrieval page window 83w displayed as the tab window TW is closed and becomes non-displayed. The user can confirm that the mirroring has stopped by seeing the display screen DS. While the data retrieval page window 83w, displayed as the tab window TW, is closed, the web browser 80 maintains the display of the browser window BW, which does not contribute to the transmission of shared data.

[0191] The projection stop operation is an application termination operation for the end-of-service icon 179 included in the UI screen 170. The control flow of the control device 10 preferably includes: setting the UI screen 170 to not display based on the application termination operation; causing the projection application 91 to generate a projection application stop signal indicating that the projection application 91 has ended based on the application termination operation; causing the projection application 91 to send the projection application stop signal to the web browser 80; and causing the web browser 80 to close the data acquisition page window 83w based on the projection application stop signal.

[0192] By sending a stop signal to the web browser 80 via the projection application 91, the web browser 80 can close the data acquisition page window 83w.

[0193] The following is a summary published in this note.

[0194] Postscript 1

[0195] An information processing method for an information processing device includes: launching an application that controls a projection device; displaying an application window that operates the application; establishing a connection across a network with the projection device for data transmission; displaying a data transmission window containing a data transmission page related to projection in a web browser based on an action instruction from the application; sending display data that causes a display screen to be displayed to the projection device via the web browser; performing projection processing to cause the projection device to project the display screen; displaying execution information in the application window indicating that the projection processing is being performed; accepting a browser window stop operation that closes the data transmission window; stopping the projection processing and closing the data transmission window based on the browser window stop operation; and performing a display change process that changes the execution information to stop information indicating that the projection processing has not been performed.

[0196] The state of the data transmission window and the display method of the application window are consistent with those in a display where mirroring is not performed. The user can reliably see that the projection device is not performing projection processing.

[0197] Appendix 2

[0198] The information processing method of the information processing device described in Appendix 1 includes: when performing the projection processing, causing the application to send a confirmation signal to the data sending window at a predetermined time interval; and when the application does not receive a response signal for the confirmation signal from the data sending window, performing the display change processing.

[0199] The application sends an acknowledgment signal, and the data sending window processes the response signal to the acknowledgment signal, thus enabling the application to determine that the data sending window has been closed.

[0200] Appendix 3

[0201] The information processing method of the information processing device described in Appendix 1 includes: when the browser window stops operating, the web application running in the data sending window sends a stop signal indicating the end of the projection process to the application; and causing the application to perform the display change process.

[0202] A stop signal is sent to the application via a web application, which then determines that the projection process has ended.

[0203] Appendix 4

[0204] An information processing method for an information processing device includes: launching an application for controlling a projection device; displaying an application window for operating the application; establishing a connection across a network with the projection device for data transmission; displaying a data transmission window containing a data transmission page related to projection in a web browser based on an action instruction from the application; sending display data that causes a display screen to be displayed to the projection device via the web browser; performing projection processing to cause the projection device to project the display screen; displaying execution information indicating that the projection processing is being performed in the application window; accepting a stop operation in the application window to stop the projection processing; and stopping the projection processing and closing the data transmission window based on the stop operation.

[0205] The application allows users to easily determine when projection processing has stopped by closing the data sending window based on a stop operation.

[0206] Appendix 5

[0207] In the information processing method of the information processing device described in Appendix 4, the stop operation is a change operation that changes the execution information to stop information indicating that the projection process has not been executed. The information processing method includes: based on the change operation, causing the application to generate a browser stop signal that closes the data sending window; causing the application to send the browser stop signal to the web browser; and based on the browser stop signal, causing the web browser to terminate the projection process.

[0208] Sending a stop signal to the web browser via an application can cause the web browser to close the data sending window.

[0209] Appendix 6

[0210] In the information processing method of the information processing device described in Appendix 4, the stop operation is an end indication operation for an application end icon contained in the application window. The information processing method includes: making the application window not displayed based on the end indication operation; making the application generate an application stop signal indicating that the application has ended based on the end indication operation; making the application send the application stop signal to the web browser; and making the web browser close the data sending window based on the application stop signal.

[0211] Sending an application stop signal to a web browser via the application itself can cause the web browser to close the data sending window.

Claims

1. An information processing method for an information processing device, wherein, The information processing method includes: Launch the application that controls the projection device, and display the application window that operates the application. In order to transmit data, a connection is established with the projection device across the network; Based on the action instructions from the application, a data sending window containing a data sending page related to the projection is displayed in the web browser; The display data for displaying the screen will be sent to the projection device via the web browser, and the projection device will be used to project the screen. The application window displays execution information indicating that the projection process is being performed; The browser window that closes the data sending window will cease operation. Based on the browser window's operation being stopped, the projection process is halted, and the data sending window is closed; as well as Perform a display change process that changes the execution information to indicate that the projection process was not executed.

2. The information processing method of the information processing device according to claim 1, wherein, The information processing method includes: During the projection process, the application sends an acknowledgment signal to the data transmission window at predetermined time intervals; and The display change process is executed when the application does not receive a response signal for the confirmation signal from the data sending window.

3. The information processing method of the information processing device according to claim 1, wherein, The information processing method includes: Upon receiving the browser window stop operation, the web application running in the data sending window will send a stop signal indicating the end of the projection process to the application; and The application is then instructed to perform the display change processing.

4. An information processing method of an information processing device, wherein, The information processing method includes: Launch the application that controls the projection device, and display the application window that operates the application. In order to transmit data, a connection is established with the projection device across the network; Based on the action instructions from the application, a data sending window containing a data sending page related to the projection is displayed in the web browser; The display data for displaying the image will be sent to the projection device via the web browser, and the projection device will be used to project the image. The application window displays execution information indicating that the projection process is being performed; The application window accepts a stop operation that halts the projection process; and Based on the stop operation, the projection process is stopped and the data sending window is closed.

5. The information processing method of the information processing device according to claim 4, wherein, The stop operation is a modification operation that changes the execution information to stop information indicating that the projection process was not executed. The information processing method includes: Based on the change operation, the application generates a browser stop signal that closes the data sending window; The application causes the browser to send a stop signal to the web browser; as well as Based on the browser stop signal, the web browser terminates the projection process.

6. The information processing method of the information processing device according to claim 4, wherein, The stop operation is an end indication operation targeting the application end icon contained in the application window. The information processing method includes: Based on the aforementioned end instruction operation, the application window is made non-displayed; Based on the termination instruction operation, the application generates an application stop signal indicating that the application has terminated; The application sends a stop signal to the web browser; and Based on the application stop signal, the web browser closes the data sending window.

Images