Communication systems, communication devices, and communication methods

Abstract

By displaying information based on the shared recipient's settings, communication is facilitated. [Solution] The first device executes the processing of an application program on its own OS, obtains drawing command information indicating the elements of a window represented based on the first OS setting which is the setting of the OS, configures a first display screen based on the first OS setting and the drawing command information, and transmits the drawing command information to the second device. The second device receives the drawing command information from the first device and configures a second display screen based on the second OS setting which is the setting of its own OS and the drawing command information.

Description

Technical Field

[0001] This application relates to a communication system, a communication device, and a communication method, for example, a communication system for sharing screen information between multiple locations.

Background Art

[0002] In recent years, communication systems for sharing screens acquired between multiple locations have become widespread. Some such communication systems can provide a function of sharing a screen separate from the video representing the participants' appearances in a communication conference using terminal devices such as personal computers (PCs) between multiple locations. For example, the information processing system described in Patent Document 1 identifies a first screen selected on the screen managed by the first device, determines the position of the first screen in the entire screen managed by the first device to a position where the first screen fits within the entire screen, and transmits the information of the first screen with the determined position to the second device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, operating system (OS) settings may differ between multiple terminal devices. Each terminal device displays the screen by arranging the display information that constitutes the screen elements in a window based on its own settings. If the screen of the sharing terminal device is sent as is to other terminal devices that are sharing the screen, the sharing device may display the received screen in a manner instructed by OS settings (e.g., language, color) that differ from the sharing terminal device's OS settings. Because the received screen is displayed based on OS settings that are unfamiliar to the sharing user, the sharing user may not be able to immediately identify the elements that the sharing user is pointing to, which can make communication difficult. [Means for solving the problem]

[0005] This invention was made to solve the above problems, and a communication system according to one embodiment comprises at least a first device and a second device, wherein the first device executes the processing of an application program on its own operating system (OS), obtains drawing command information indicating elements of a window represented based on a first OS setting which is the setting of the OS, configures a first display screen based on the first OS setting and the drawing command information, transmits the drawing command information to the second device, and the second device receives the drawing command information from the first device and configures a second display screen based on a second OS setting which is the setting of its own OS and the drawing command information.

[0006] In the above communication system, the first device further transmits the first OS settings to the second device, and the second device may configure a second display screen using the same type of window elements as those shown in the drawing command information, based on the second setting item, when the first setting item, which is a setting item of the first OS settings related to the drawing command information, and the second setting item, which is a setting item of the second OS settings related to the drawing command information, are different.

[0007] In the above communication system, the first OS setting and the second OS setting are, respectively It may include at least one item related to language and color.

[0008] In the above communication system, the first device transmits the first OS settings to the second device in response to the start of sharing of the display screen based on the drawing command information.

[0009] In the above communication system, the second device starts an application program of the same type as the application program in response to the start of sharing.

[0010] A communication device according to a second aspect of the present invention may receive drawing command information from another device that indicates elements of a window represented based on the first OS settings, which are the settings of the first OS, by executing the processing of an application program on the first OS, which is the operating system (OS) of the other device, and configure a second display screen based on the second OS settings, which are the OS settings of the device itself, and the drawing command information.

[0011] A third aspect of the present invention relates to a communication method in a communication system comprising at least a first device and a second device, wherein the first device executes processing of an application program on its operating system (OS) to obtain drawing command information indicating elements of a window represented based on a first OS setting which is a setting of the OS, configures a first display screen based on the first OS setting and the drawing command information, transmits the drawing command information to the second device, and the second device receives the drawing command information from the first device and configures a second display screen based on a second OS setting which is a setting of its own OS and the drawing command information. [Effects of the Invention]

[0012] According to the embodiment of the present invention, communication can be facilitated by presenting display information based on the settings of the sharing recipient. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic block diagram showing an example configuration of the communication system according to this embodiment. [Figure 2] This is a schematic block diagram showing an example configuration of a terminal device according to this embodiment. [Figure 3] This is a schematic block diagram showing an example of the functional configuration of the terminal device according to this embodiment. [Figure 4] This flowchart shows an example of the application screen sharing process according to this embodiment. [Figure 5] This is a data flow diagram of an example of the application screen sharing process according to this embodiment. [Figure 6] This figure shows a first example of the display screen according to this embodiment. [Figure 7] This figure shows a second example of the display screen according to this embodiment. [Figure 8] This figure shows a first example of the application screen sharing process according to this embodiment. [Figure 9] This figure shows a second example of the application screen sharing process according to this embodiment. [Modes for carrying out the invention]

[0014] Hereinafter, embodiments of the present application will be described with reference to the drawings. An example of the configuration of the communication system S1 according to this embodiment will be described. Figure 1 is a schematic block diagram showing an example configuration of the communication system S1 according to this embodiment. The communication system S1 is comprised of multiple terminal devices 1. The multiple terminal devices 1 are connected to each other via a network NW, enabling them to send and receive various types of data. The network NW may be the Internet, a public communication network, a local area network (LAN), a virtual private network (VPN), a dedicated line, or a combination of any of these. In the example of FIG. 1, two terminal devices 1 are described, and each terminal device 1 is distinguished by attaching sub-numbers such as A, etc., like terminal device 1A, etc. The number of terminal devices 1 connected to the network NW may be an unspecified number of three or more, or may be fixed to a specific number.

[0015] The communication system S1 is configured as, for example, a screen sharing system. When any one of a plurality of terminal devices (for example, terminal device 1A) executes an application program (which may be referred to as an "application" or "app" in this application) and displays a screen (which may be referred to as an "app screen" in this application), other terminal devices (for example, terminal device 1B) can also display the app screen (app screen sharing). App screen sharing is a form of information sharing among multiple terminals.

[0016] In the following description, the terminal device 1 that is the source of the app screen sharing is referred to as the "first device", and the app screen displayed on the first device is referred to as the "first display screen". The terminal device 1 that is the destination of the app screen sharing is referred to as the "second device", and the app screen displayed on the second device may be referred to as the "second display screen". Also, executing a process instructed by a command described in an application or other program may be referred to as "executing a program" or "execution of a program".

[0017] Each terminal device 1 executes the installed operating system, and on each OS, executes an application based on the settings of each OS (which may be referred to as "OS settings" in this application). The first device acquires instruction information for the OS to execute an application and draw a window (hereinafter abbreviated as "drawing instruction information"), and constructs a first display screen based on the first OS settings, which are the OS settings of the device itself, and the drawing instruction information. The first device displays the constructed first display screen. The first device transmits the first OS settings and the drawing instruction information used when constructing the first display screen to the second device. The drawing instruction information can also be regarded as information indicating the elements of the window that constitutes the application screen. That is, the drawing instruction information can be, for example, drawing instructions for screen components (UI components) such as buttons and tabs, or a set thereof (drawing instruction set). Also, among the drawing instruction information, menu strings and the like may be separated as resource files and become a set of instructions for indicating which resource to use for drawing. Note that such screen components may be operated by the user of the first device. The second device receives the first OS settings and the drawing instruction information from the first device, and reconstructs a second display screen based on the second OS settings, which are the OS settings of the device itself.

[0018] Here, the first device refers to the terminal device 1 that is the source of sharing the application screen, and the second device refers to the terminal device 1 that is the destination of sharing the application screen, and does not necessarily refer to the hardware configuration and other functional configurations of each terminal device 1. A single terminal device (for example, terminal device 1A) may, in relation to another terminal device (for example, terminal device 1B), (1) perform the functions of the first device and not perform the functions of the second device, (2) perform the functions of the second device and not perform the functions of the first device, or (3) perform both the functions of the first device and the functions of the second device. In the description of the functional configuration of the terminal device 1 described later, the case where it has both the functions of the first device and the functions of the second device will be taken as an example.

[0019] Furthermore, the communication system S1 may transmit audio or video acquired by individual terminal devices 1 to other terminal devices 1, and the receiving terminal device 1 may display the audio or video received from the transmitting terminal device 1 (audio-video communication). Audio-video communication may be performed in parallel with application screen sharing. The first and second devices may be the source, destination, or both of the audio and video.

[0020] Next, an example of the configuration of the terminal device 1 according to this embodiment will be described. Figure 2 is a schematic block diagram showing an example of the configuration of the terminal device 1 according to this embodiment. The terminal device 1 may be a general-purpose information terminal device such as a personal computer, a tablet terminal device, or a multi-functional mobile phone (including so-called smartphones). In the following description, the case where the terminal device 1 is a PC will be used as an example.

[0021] The terminal device 1 includes a host system 10, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a display 24, a camera 25, an audio system 26, a communication module 27, an input / output interface 28, an embedded controller (EC) 31, an input device 32, a power supply circuit 33, and a power switch 36.

[0022] The host system 10 is the core computer system of the terminal device 1. The host system 10 comprises a processor, main memory, and a chipset. In this application, the hardware constituting the host system 10 may be referred to as the "host device." The processor is the core processing unit that controls the operation of the entire terminal device 1. The processor is, for example, a CPU (Central Processing Unit). The processor is the core processing unit that executes arithmetic operations instructed by various commands written in the software (program).

[0023] The processor may include a GPU (Graphics Processing Unit) in addition to the CPU. The GPU is a processing unit primarily used to implement functions related to image display. The GPU processes drawing instructions issued by the CPU (image processing) and outputs display data showing the obtained display information to the display 24. The GPU may be integrated with the CPU and formed on the same core, or it may be formed on a separate core from the CPU.

[0024] Main memory is writable memory used as a reading area for the processor's executable program or as a working area for writing processing data for the executable program. Main memory is composed of, for example, multiple DRAM (Dynamic Random Access Memory) chips. The processor and main memory constitute the minimum hardware required for the host system 10.

[0025] The chipset includes multiple controllers, enabling connection to multiple devices and various data input / output. The controllers included in the chipset may be, for example, USB (Universal Serial Bus), SPI (Serial Peripheral Interface) bus, PCI-Express bus, etc.

[0026] ROM22 primarily stores firmware. The firmware stored in ROM22 includes the BIOS (Basic Input-Output System) and other firmware specific to individual devices. ROM22 is composed of rewritable non-volatile memory such as EEPROM (Electrically Erasable Programmable Read Only Memory) and flash ROM.

[0027] The auxiliary storage device 23 stores various data used in the processing of the host system 10, various data acquired through such processing, or various programs. The auxiliary storage device 23 may be, for example, an SSD (Solid State Drive) or an HDD (Hard-disk Drive).

[0028] The display 24 displays a screen based on display data input from the host system 10. The display 24 may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0029] Camera 25, following the control of the host system 10, captures images representing subjects appearing within its field of view. Camera 25 is a video camera capable of capturing video, i.e., moving images. Camera 25 outputs video data showing the captured video to the host system 10.

[0030] The audio system 26 performs input and output of audio data. Following the control of the host system 10, the audio system 26 converts the analog input audio signal received from the microphone into digital input audio data and outputs the resulting input audio data to the host system 10. The microphone detects the sound propagating within itself and outputs an input audio signal indicating the detected sound to the audio system 26. The audio system 26 converts the digital output audio data output from the host system 10 into an analog output audio signal and outputs the resulting output audio signal to the speaker. The speaker presents sound based on the output audio signal received from the audio controller. The camera 25, microphone, and speaker may be provided on the terminal device 1, some or all of them may be detachably connected to the terminal device 1, or they may be separate from the terminal device 1.

[0031] The communication module 27 connects to a communication network, enabling it to send and receive various types of data wirelessly or via a wired connection. The communication module 27 communicates various types of data with other devices connected to the communication network. The communication module 27 is, for example, a wireless LAN module that connects to a wireless LAN.

[0032] The I / F28 connects to various devices for data input and output via wired or wireless connections. For example, the I / F28 includes a USB connector for wired data input and output in accordance with USB specifications.

[0033] EC31 is a controller that monitors and controls the operation of various devices connected to it, regardless of the operating status of the host system 10. EC31 has a separate CPU, ROM, RAM, timer, and input / output interface from the host system 10. Devices with slower data transfer speeds than those connected to the host system 10 can be connected to EC31. In the example shown in Figure 2, an input device 32, a power supply circuit 33, and a power switch 36 are connected to EC31.

[0034] The EC31 reads predetermined firmware from its own ROM, executes the read firmware, and provides its function. Alternatively, the firmware may be pre-stored in ROM22 instead of its own ROM, and the EC31 may execute the read firmware.

[0035] The input device 32 detects user operations, generates an operation signal according to the detected operation, and outputs it to EC31. The input device 32 may be, for example, a keyboard, a touchpad, or a combination of either.

[0036] The power supply circuit 33 includes a voltage converter and a charger. The voltage converter converts the voltage of DC power supplied from an external power source or battery (not shown) to the voltage required for the operation of each device constituting the terminal device 1, and supplies power with the converted voltage to the receiving device. The power supply circuit 33 performs power supply to the device according to the control of EC31. The charger charges the battery with any remaining power from the external power source that is not consumed by each device. If no power is supplied from the external power source, or if the power supplied from the external power source does not meet the demand, the charger supplies power discharged from the battery to each device. The battery charges with power supplied from the power supply circuit 33, or discharges power stored in itself to the power supply circuit 33. The battery may be, for example, a lithium-ion battery, a sodium-ion battery, or any other type.

[0037] Each time a press operation is received, the power switch 36 controls the power supply state to the host system 10 to either power ON or power OFF. When a press operation is received, the power switch 36 outputs a press signal to EC31 indicating that a press has been made. When the terminal device 1 is powered off and a press signal is input from the power switch 36, EC31 instructs the power supply circuit 33 to start supplying power to each device of the terminal device 1 (power on). When power is supplied to the terminal device 1 and a press signal is input from the power switch 36, EC31 causes the host system 10 to perform a shutdown process.

[0038] Next, an example of the functional configuration of terminal device 1 will be described. The functions of terminal device 1 are realized by the CPU of the host system 10 executing various programs in cooperation with the main memory and other hardware. Figure 3 is a schematic block diagram showing an example of the functional configuration of terminal device 1 according to this embodiment. Terminal device 1 includes an application processing unit 12, an information sharing processing unit 14, and a setting information management unit 16.

[0039] The application processing unit 12 executes applications on the OS and manages the execution status of those applications. The application processing unit 12 starts the execution of applications that are instructed to run and terminates the execution of applications that are instructed to stop. The application to be operated on, and whether to run or terminate it, may be instructed by operation signals input from the input device 32.

[0040] The application processing unit 12 includes an output processing unit 122. The output processing unit 122 executes the application and outputs the application screen. In the first device that serves as the source for sharing the application screen, the output processing unit 122 identifies drawing command information instructed by some commands of the running application as first drawing command information. The output processing unit 122 configures the first window based on the identified first drawing command information, according to the first OS settings set in the setting information management unit 16. The drawing command information consists of elements of the window, including screen components and element information indicating their display mode. Drawing command information to be displayed is instructed by the commands of individual applications. For example, pre-configured screen components such as buttons and tabs may be instructed as drawing command information. Screen components may be pre-configured in the OS. For example, position, size, and color may be instructed as display mode. Even screen components of the same type may be represented using different drawing command information if their display mode is different. Screen components may be used to instruct the function or execution state of the application in response to user operations.

[0041] The output processing unit 122 sets the application screen, including the first window, as the first application screen. Depending on the processing instructed by other commands of the running application, content to be displayed may be acquired by executing that processing. For example, in a document editing application, elements of content may include strings generated in response to user operations, selected or generated shapes or symbols. In a presentation application, slides or videos instructed by user operations may become content. In that case, the output processing unit 122 places the acquired content into the first window and configures it as the first application screen. The output processing unit 122 outputs display data to the display 24 indicating the configured application screen as the first application screen. The display 24 displays the first application screen indicated by the display data input from the output processing unit 122.

[0042] In the second device, which is the destination for sharing the application screen, the output processing unit 122 receives second drawing command information from the first device via the information sharing processing unit 14. Based on the received second drawing command information, the output processing unit 122 configures the second window. The output processing unit 122 sets the application screen, including the second window, as the second application screen. If content is received from the second device via the information sharing processing unit 14, the received content is placed in the second window to configure the second application screen. The output processing unit 122 outputs display data indicating the configured second application screen to the display 24. The display 24 displays the second application screen indicated by the display data input from the output processing unit 122.

[0043] The information sharing processing unit 14 executes processing related to sharing the application screen with other terminal devices 1. The information sharing processing unit 14 includes an execution control unit 140, an OS setting acquisition unit 142, a screen information acquisition unit 144, an information sharing unit 146, and a drawing command information verification unit 148. The execution control unit 140, the OS setting acquisition unit 142, the screen information acquisition unit 144, and the information sharing unit 146 are used to implement the functions of the first device, which is the source of application screen sharing. The execution control unit 140, the OS setting acquisition unit 142, the information sharing unit 146, and the drawing command information verification unit 148 are used to implement the functions of the second device.

[0044] The execution control unit 140 controls whether or not to perform application screen sharing. For example, when an operation signal indicating application screen sharing is input from the input device 32, the execution control unit 140 starts processing to start the function of the first device which is the source of the application screen sharing. Here, the execution control unit 140 may display an operation screen on the display 24 that presents screen components (e.g., a screen sharing button) for instructing the start of application screen sharing. The operation screen may be dedicated to application screen sharing, but it may also be shared with other functions, such as audio and video communication.

[0045] The execution control unit 140 may, for example, display an application list on the display 24 when it detects that the screen sharing button has been pressed. The application list will show one or more applications installed on the terminal device 1. When the application processing unit 12 receives an operation signal indicating any one of the applications, it may select the application indicated by the operation signal as the shared application and start the application screen sharing process along with the execution of the selected shared application. The execution control unit 140 sends an application screen sharing notification to the other terminal device 1 indicating that the application screen sharing is being performed using the shared application.

[0046] When the execution control unit 140 receives an application screen sharing notification from another terminal device 1, it starts processing to start the functions of the second device that will be the recipient of the application screen sharing. Here, the execution control unit 140 causes the application processing unit 12 to start executing the shared application instructed by the application screen sharing notification. Subsequently, the execution control unit 140 sends an application screen sharing confirmation to the other terminal device 1 as a response to the application screen sharing notification. The execution control unit 140 then starts the functions of the second device. When the execution control unit 140 receives confirmation of application screen sharing from another terminal device 1, it starts the functions of the first device.

[0047] When the function of the first device is started, the OS setting acquisition unit 142 reads a portion of the OS setting information set in the setting information management unit 16 as the first OS setting. The first OS setting includes setting parameters related to the display of drawing command information that constitutes the components of the application screen. Setting parameters related to the display of drawing command information include, for example, language and color mode. The OS setting acquisition unit 142 may also read the first OS setting each time the first OS setting is updated. The OS setting acquisition unit 142 outputs the read first OS setting to the information sharing unit 146.

[0048] Similarly, whenever the second device starts functioning, or whenever the second OS settings are updated, the OS setting acquisition unit 142 reads a portion of the OS setting information set in the setting information management unit 16 as the second OS settings. The OS setting acquisition unit 142 outputs the read second OS settings to the drawing command information verification unit 148.

[0049] The screen information acquisition unit 144 monitors the execution status of the application in the output processing unit 122 and acquires the first drawing command information instructed by the running application. The screen information acquisition unit 144 sequentially acquires new first drawing command information as the application processing progresses. Depending on the application's function or execution status, the screen information acquisition unit 144 may also acquire new content from the output processing unit 122. The screen information acquisition unit 144 outputs the acquired content to the information sharing unit 146.

[0050] The information sharing unit 146 shares OS settings and drawing command information between the first and second devices. Depending on the function or execution state of the application, the information sharing unit 146 may also share content. In the first device, the information sharing unit 146 transmits the first OS settings input from the OS setting acquisition unit 142 and the first drawing command information input from the screen information acquisition unit 144 to the second device. If content is input from the screen information acquisition unit 144, the information sharing unit 146 transmits the input content to the second device. In the second device, the information sharing unit 146 outputs the first OS settings and first drawing command information received from the first device to the drawing command information verification unit 148. When content is received from the first device, the information sharing unit 146 outputs the received content to the application processing unit 12.

[0051] The drawing command information verification unit 148 verifies whether the first drawing command information input from the information sharing unit 146 conforms to the second OS setting input from the information sharing unit 146. If it is determined to be conforming, the drawing command information verification unit 148 adopts the input first drawing command information as the second drawing command information. If it is determined to be inconclusive, the drawing command information verification unit 148 replaces the first drawing command information with the second drawing command information which conforms to the second OS setting. The drawing command information verification unit 148 outputs the adopted second drawing command information to the application processing unit 12. The configuration information management unit 16 pre-configures window elements by associating them with each individual drawing command information. However, for some window elements, multiple screen components of the same type but with different display modes may be configured as window elements. Multiple screen components are classified by type, and each display mode is associated with different configuration parameters.

[0052] More specifically, the drawing command information verification unit 148 determines whether the second setting parameter relating to the screen component shown in the first drawing command information, which constitutes the second OS setting, is equal to the first setting parameter applied to the screen component shown in the first drawing command information, which constitutes the first OS setting. If the second setting parameter and the first setting parameter are equal, the drawing command information verification unit 148 adopts the input first drawing command information as the second drawing command information. For example, suppose the first setting parameter is set to a color mode setting indicating light mode, the second setting parameter is set to a language setting indicating light mode, and the first drawing command information instructs a tab with a black string and a white background. In this case, the first setting parameter applied to the first drawing command information, the color mode setting, is light mode, and the second setting parameter relating to the tab, the color mode setting, is light mode, which are the same. Therefore, the drawing command information verification unit 148 adopts the first drawing command information with a black string and a white background from the setting information management unit 16 as the second drawing command information.

[0053] If the second setting parameter and the first setting parameter related to the second drawing command information are different, the drawing command information verification unit 148 refers to the setting information management unit 16 and adopts as the second drawing command information a screen component of the same type as the screen component shown in the input second drawing command information and represented by the display mode instructed by the second setting parameter. For example, suppose the language setting indicating Japanese is set as the first setting parameter, the language setting indicating Chinese is set as the second setting parameter, and the first drawing command information instructs a button with a Japanese string attached. In this case, the language setting, which is the first setting parameter applied to the first drawing command information, is Japanese, while the language setting, which is the second setting parameter related to the button, is Chinese. Therefore, the drawing command information verification unit 148 identifies the drawing command information indicating a button with a Chinese string attached from the setting information management unit 16 and adopts the identified drawing command information as the second drawing command information.

[0054] The configuration information management unit 16 stores various configuration information. This configuration information includes various OS configuration information, configuration information for individual applications, and the drawing command information mentioned above. The drawing command information may be managed in association with the OS configuration information, or it may be included as part of the OS configuration information. The configuration information management unit 16 may display a predetermined configuration screen on the display 24 and change some configuration parameters according to operation signals from the input device 32.

[0055] Next, an example of the application screen sharing process according to this embodiment will be described using Figures 4 and 5. Figure 4 is a flowchart of an example of the application screen sharing process according to this embodiment. Figure 5 is a data flow diagram related to an example of the application screen sharing process according to this embodiment. In the examples of Figures 4 and 5, terminal device 1A is the first device and terminal device 1B is the second device. It is assumed that the execution of the shared application has already started on terminal device 1A.

[0056] (Step S102) The execution control unit 140 of terminal device 1A starts sharing the application screen in response to user operation. The execution control unit 140 notifies terminal device 1B of the application screen sharing notification. The execution control unit 140 of terminal device 1B instructs its own application processing unit 12 to start executing the shared application in response to the application screen sharing notification from terminal device 1A. (Step S104) The OS setting acquisition unit 142 of the terminal device 1A acquires the OS settings set in its unit from the setting information management unit 16 as the first OS settings. (Step S106) The screen information acquisition unit 144 of the terminal device 1A acquires the first drawing command information instructed by the execution of the shared application. (Step S108) The information sharing unit 146 of terminal device 1A transmits the acquired first OS settings and first drawing command information to terminal device 1B.

[0057] (Step S112) The information sharing unit 146 of terminal device 1B receives the first OS settings and the first drawing command information from terminal device 1A. (Step S114) The OS setting acquisition unit 142 of the terminal device 1B acquires the OS settings set in its unit from the setting information management unit 16 as the second OS settings. (Step S116) The drawing command information verification unit 148 of the terminal device 1B determines whether the received first drawing command information conforms to the second OS setting. If it conforms, the drawing command information verification unit 148 adopts the first drawing command information as the second drawing command information. If it does not conform, it replaces it with second drawing command information that indicates a window element of the same type as the window element indicated by the first drawing command information and that conforms to the second OS setting. (Step S118) The output processing unit 122 of the terminal device 1B reconfigures the window based on the second window and configures an application screen having the reconfigured window. The output processing unit 122 displays the display data indicating the application screen on the display 24. After that, the process shown in Figure 4 is terminated.

[0058] Next, we will describe examples of screen components instructed by drawing command information. Figure 6 shows a first example of a display screen according to this embodiment. Figure 6 illustrates a menu bar. The menu bar has multiple buttons as screen components, and each button is arranged at a different position along the top edge of the window. Different strings are superimposed on each button. The strings are variable according to the language setting, which is an example of a setting parameter. In the example in Figure 6(a), the strings displayed on the buttons are in Japanese. In the example in Figure 6(b), the strings displayed on the buttons are in English. The language of the strings displayed on each button is indicated by the language setting. That is, Figure 6(a) illustrates a menu bar when the language setting is Japanese, and Figure 6(b) illustrates a menu bar when the language setting is English.

[0059] Figure 7 shows a second example of the display screen according to this embodiment. Figure 7 illustrates a settings screen. The settings screen has multiple buttons and tabs as screen components, with each button and tab arranged in different positions within the window. Text and images are superimposed on each button and tab. The background color of the buttons and tabs, and the color of the text are variable according to the color mode, which is a setting parameter. Figure 7(a) illustrates the settings screen when the color mode is light mode. In Figure 7(a), the background of each button and tab is shown in white or grayish-white, and the respective text is shown in black. The settings screen as a whole is displayed brightly. Figure 7(b) illustrates the settings screen when the color mode is dark mode. In Figure 7(b), the background of each button and tab is shown in black, and the respective text is shown in white. The settings screen as a whole is displayed darker than the settings screen illustrated in Figure 7(a). However, the images at the top and bottom center of the settings screen show the display screen and color bar of the display 24 at that time in the content. These images fall under the category of content that is not subject to brightness or color tone settings based on color mode.

[0060] Next, an example of executing the application screen sharing process according to this embodiment will be described. Figure 8 is a diagram showing a first example of executing the application screen sharing process according to this embodiment. Figure 8 shows the first application screen displayed on the display 24 of terminal device 1A as an example of the first device on the left, and the second application screen displayed on the display 24 of terminal device 1B as an example of the second device on the right. The first and second application screens are display screens for a common document editing application. Each display screen contains common content in its respective window. The content represents a master slide for one page and thumbnails for multiple pages. A long, narrow menu bar is arranged horizontally at the top edge of each window according to the application settings. Screen components such as buttons and tabs included in the menu bar are indicated by drawing command information, and the display manner of the strings represented in each screen component is indicated by the OS settings.

[0061] In the example shown in Figure 8, Chinese is specified as the language setting in the first OS configuration of terminal device 1A. Chinese characters are displayed on individual buttons or tabs on the display 24. Drawing command information indicating the menu bar and Chinese as the language setting for the first OS configuration are transmitted from terminal device 1A to terminal device 1B. In the second OS settings of terminal device 1B, Japanese is selected as the language setting, and Japanese text is displayed instead of Chinese on individual buttons or tabs that make up the menu bar.

[0062] Therefore, when the user of terminal device 1A, the source of the sharing, operates the menu bar, the Chinese text displayed on that menu bar is replaced with Japanese text on terminal device 1B, the recipient of the sharing. As a result, the user of terminal device 1B is presented with the text on the menu bar they are operating in familiar Japanese. Therefore, the user of terminal device 1B can easily understand the operations performed on the menu bar by the user of terminal device 1A, the source of the sharing. This embodiment is useful, for example, when the user of terminal device 1A instructs the user of terminal device 1B on how to operate the application via voice communication.

[0063] Figure 9 shows a second execution example of the application screen sharing process according to this embodiment. Figure 9 shows the first application screen on the left and the second application screen on the right. The first and second application screens are setting screens for a common communication application. Each setting screen contains multiple buttons and tabs that indicate common setting items, each housed in its own window. The content of each setting screen represents a master slide on one page and thumbnails on multiple pages. Each button and tab is indicated by drawing command information, and the display mode of each button and tab is indicated by the OS settings.

[0064] In the example shown in Figure 9, the first OS settings of terminal device 1A specify Chinese as the language setting and light mode as the color mode. On display 24, Chinese characters are displayed on individual buttons or tabs, and the background is filled with white or grayish-white. When the settings screen is shown from terminal device 1A to terminal device 1B, the language setting of the first OS settings, Chinese, and light mode are transmitted. In the second OS settings of terminal device 1B, Japanese is specified as the language setting, and dark mode is specified as the color mode. Instead of Chinese text, Japanese characters are displayed in white on individual buttons or tabs in the settings screen, with a black background. Buttons and tabs are identified by their respective button IDs and tab IDs in the application settings.

[0065] Therefore, when the user of terminal device 1A, the source of the sharing, operates the settings screen, the Chinese characters displayed on the settings screen in white on a black background are replaced with Japanese characters displayed in black on a white background on terminal device 1B, the recipient of the sharing. The user of terminal device 1B is presented with familiar Japanese characters displayed in black on a white background, instead of the characters attached to the tabs of the settings screen they are operating. Thus, the user of terminal device 1B can easily understand the operations performed on the settings screen by the user of terminal device 1A, the source of the sharing. This embodiment is useful, for example, when the user of terminal device 1A instructs the user of terminal device 1B on how to configure the application via voice communication.

[0066] The above explanation primarily uses the example of sharing an application screen between two terminal devices 1A and 1B, with the communication system S1 having two connections, but it is not limited to this. The communication system S1 may have three or more connections and be applied to communication between three or more terminal devices 1. In that case, at least one of the three or more terminal devices 1 is designated as the first device, and the other multiple terminal devices are designated as second devices. The first device transmits the first OS settings and first drawing command information acquired by its own device to each of the second devices. Each second device acquires second drawing command information based on the first drawing command information received from the first device, the first OS settings received, and its own second OS settings. The second device then reconstructs the application screen based on the acquired second drawing command information.

[0067] Furthermore, the execution control unit 140 of a terminal device 1 may select another terminal device 1 as the second device to be shared for application sharing during voice-video communication. If there are multiple other terminal devices 1 that are the recipients during voice-video communication, the execution control unit 140 may select any of the multiple terminal devices 1 as the second device, or it may select the terminal device 1 indicated by the operation signal input from the input device 32 in response to the operation as the second device.

[0068] Furthermore, while the above explanation primarily focused on cases where the sharing application involved in application screen sharing is a document editing application or a presentation application, it is not limited to these. The sharing application involved in application screen sharing may be other types of applications, such as email software, web browsers, file management applications, spreadsheet applications, the conferencing application itself used to perform application sharing, or web applications accessible from a browser.

[0069] As described above, the communication system S1 according to this embodiment comprises at least a first device (e.g., terminal device 1A) and a second device (e.g., terminal device 1B). The first device executes the processing of an application program on its own OS and obtains drawing command information that indicates window elements (e.g., buttons, tabs) represented based on the first OS settings, which are the settings of the device's OS. The first device configures a first display screen (e.g., a first application screen) based on the first OS settings and the drawing command information and transmits the drawing command information to the second device. The second device receives the drawing command information from the first device and configures a second display screen (e.g., a second application screen) based on the second OS settings, which are the settings of the device's OS, and the received drawing command information.

[0070] In this configuration, the second device configures a second display screen in accordance with its own second OS settings, using the window elements indicated in the drawing command information received from the first device. Therefore, the user of the second device, which is the recipient of the drawing command information, can understand the window elements instructed by the source by viewing the second display screen, which is configured with window elements according to the familiar second OS settings. Consequently, communication with the user of the first device, which is the source of the drawing command information, becomes smoother.

[0071] This embodiment may be implemented as a communication device (for example, terminal devices 1A, 1B) that executes the processing of an application program on a first OS, which is the OS of the other device, from another device, receives drawing command information that indicates the elements of a window represented based on the first OS settings, which are the settings of the first OS, and configures a second display screen based on the second OS settings, which are the OS settings of its own device, and the drawing command information.

[0072] The first device may further transmit the first OS settings to the second device. The second device has pre-configured window elements, which are elements of a window corresponding to drawing command information, for each setting item of the OS settings. When the first setting item, which is a setting item of the first OS settings related to the drawing command information received from the first device (e.g., language setting, color mode setting), and the second setting item, which is a setting item of the second OS settings related to the drawing command information, are different, the second display screen may be configured using the same type of window elements as those indicated in the drawing command information, based on the second setting item.

[0073] The first OS setting and the second OS setting may each include at least one item related to language and color. The first device may send the first OS settings to the second device in response to the start of sharing of the display screen based on drawing command information. The second device may launch the same type of application program as the application program executed by the first device in response to the start of sharing.

[0074] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configurations are not limited to the embodiments described above, and include designs and the like that do not depart from the spirit of this invention. The configurations described in the embodiments described above can be combined in any way. [Explanation of Symbols]

[0075] S1...Communication system, 1 (1A, 1B)...Terminal device, 10...Host system, 12...Application processing unit, 14...Information sharing processing unit, 16...Setting information management unit, 22...ROM, 23...Auxiliary storage device, 24...Display, 25...Camera, 26...Audio system, 27...Communication module, 28...Input / output I / F, 31...EC, 32...Input device, 33...Power supply circuit, 36...Power switch, 122...Output processing unit, 140...Execution control unit, 142...OS setting acquisition unit, 144...Screen information acquisition unit, 146...Information sharing unit, 148...Drawing command information verification unit

Claims

1. A communication system comprising at least a first device and a second device, The first apparatus is The application program is executed on the operating system (OS) of the device, and drawing command information indicating the elements of a window, which is represented based on the first OS setting, which is the setting of the OS, is obtained. Based on the first OS settings and the drawing command information, a first display screen is configured. The drawing command information is transmitted to the second device, The second device is The drawing command information is received from the first device. The second display screen is configured based on the second OS settings, which are the OS settings of the device itself, and the drawing command information. Communication system.

2. The first apparatus is The aforementioned first OS settings are further transmitted to the second device. The second device is For each setting item in the OS settings, a window element, which is an element of the window corresponding to the drawing command information, is pre-configured. When the first setting item, which is a setting item of the first OS setting related to the drawing command information, and the second setting item, which is a setting item of the second OS setting related to the drawing command information, are different, the second display screen is configured using window elements of the same type as the window elements shown in the drawing command information, based on the second setting item. The communication system according to claim 1.

3. The first OS setting and the second OS setting are, respectively Includes at least one item of language and color. The communication system according to claim 1.

4. The first apparatus is The first OS settings are transmitted to the second device in response to the start of sharing of the display screen based on the drawing command information. The communication system according to claim 1.

5. The second device is In response to the start of the sharing, launch an application program of the same type as the aforementioned application program. The communication system according to claim 4.

6. The system receives drawing command information from another device, which is the operating system (OS) of the other device, that executes the processing of an application program on the first OS and represents the elements of a window based on the settings of the first OS. The second display screen is configured based on the second OS settings, which are the OS settings of the device itself, and the drawing command information. Communication device.

7. A communication method in a communication system comprising at least a first device and a second device, The first apparatus is The application program is executed on the operating system (OS) of the device, and drawing command information indicating the elements of a window, which is represented based on the first OS setting, which is the setting of the OS, is obtained. Based on the first OS settings and the drawing command information, a first display screen is configured. The drawing command information is transmitted to the second device, The second device is The drawing command information is received from the first device. The second display screen is configured based on the second OS settings, which are the OS settings of the device itself, and the drawing command information. Communication method.

Images