Environment configuration device, environment configuration method, and program

Abstract

To make operability and convenience compatible in a case where a situation relating to a first terminal device is changed in a remote control system including the first terminal device, which outputs first information for output from a user interface to the user interface, and a second terminal device which is remotely controlled by the first terminal device and transmits second information for obtaining the first information to the first terminal device.SOLUTION: An environment configuration of a remote control system is changed in accordance with a change of situation information representing at least any one of a utilization situation, an apparatus situation and a line situation of a first terminal device, and at least a part of first information and second information is changed in accordance with the change of the environment configuration.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for controlling an environment configuration based on remote operation.

Background Art

[0002] There is known a technique of remotely operating a terminal device located at a remote location by operating a terminal device at hand, acquiring information for output (for example, video output) from a user interface at the terminal device at hand or its peripheral devices, and performing output from the user interface based on the acquired information (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the prior art, since the environmental configuration (e.g., screen layout) of the remote operation system is fixed, when the situation regarding the terminal device at hand changes, the operation may become difficult. Further, in such a case, it is inconvenient to manually reconfigure the environmental configuration. Such a problem is not limited to the case where the situation regarding the terminal device at hand changes, but is a common problem in a remote operation system including a first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device, when the situation regarding the first terminal device changes.

[0005] The present invention provides a technology that achieves both operability and convenience when the situation regarding a first terminal device changes in a remote operation system including a first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device.

Means for Solving the Problems

[0006] The environmental configuration device changes the environmental configuration of the remote operation system in response to a change in situation information representing at least any one of the usage status, device status, and line status of the first terminal device, and changes at least a part of the first information and the second information in response to the change in the environmental configuration.

Effects of the Invention

[0007] Thereby, in a remote operation system including a first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device, it is possible to achieve both operability and convenience when the situation regarding the first terminal device changes.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [First Embodiment] First, the first embodiment will be described. <Overall Configuration> As illustrated in FIG. 1, the remote operation system 1 of this embodiment includes a local terminal device 11 (first terminal device) and a connection destination terminal device 12 (second terminal device). The local terminal device 11 and the connection destination terminal device 12 are configured to be communicable through a network 10. The local terminal device 11 is configured to output output information UI1 (first information) for the output from the user interface 111 to the user interface 111. The connection destination terminal device 12 is remotely operated by the local terminal device 11 and is configured to transmit remote information UI2 (second information) for obtaining the output information UI1 (first information) to the local terminal device 11.

[0010] <Local Terminal Device 11 (First Terminal Device)> The hand-held terminal device 11 is a device directly operated by the user 100 and is arranged near the user 100 (for example, at hand). The user 100 inputs to the user interface 111 or perceives the output from the user interface 111. The input to the user interface 111 may be an operation input (for example, mouse input or keyboard input), a contact input (for example, touch panel input), a non-contact input (for example, touchless input or gesture input), an image input (for example, image reading), a voice input, or other inputs. The output from the user interface 111 is some kind of sensory information, and for example, it may be an image output, a video output, a virtual space output, a text output, or a voice output. As illustrated in FIG. 2, the hand-held terminal device 11 of this embodiment includes a user interface 111, a situation information acquisition unit 112, a remote operation unit 113, a remote information control unit 114, a communication interface 115, and a memory 116. The information input to the hand-held terminal device 11 and the information output from each part of the hand-held terminal device 11 are stored in the memory 116 one by one and read out as necessary. The hand-held terminal device 11 is, for example, a general-purpose or dedicated computer equipped with a processor (hardware processor) such as a CPU (central processing unit), a memory such as a RAM (random-access memory) and a ROM (read-only memory), a communication device, and input / output devices such as a display and a speaker, and is configured by executing a predetermined program. This computer may be equipped with one processor and memory, or may be equipped with a plurality of processors and memory. This program may be installed in the computer or may be recorded in a ROM or the like in advance. Also, instead of an electronic circuit (circuitry) that realizes a functional configuration by loading a program like a CPU, some or all of the processing units may be configured using an electronic circuit that realizes a processing function without using a program. Also, the electronic circuit constituting the hand-held terminal device 11 may include a plurality of CPUs.Specific examples of the local terminal device 11 include a personal computer terminal device, a tablet terminal device, a smartphone terminal device, a smartwatch terminal device, etc. The user interface 111 is, for example, a touch panel, a microphone, a speaker, a mouse, a keyboard, a virtual touch pad, a headset, a virtual controller, etc.

[0011] <Connected destination terminal device 12 (second terminal device)> The connected destination terminal device 12 is a device remotely operated by the local terminal device 11. That is, the user 100 remotely operates the connected destination terminal device 12 using the local terminal device 11. As illustrated in FIG. 3, the connected destination terminal device 12 includes an environment configuration device 121, storage units 122 and 123, a remote information providing unit 124, a communication interface 125, and a memory 126. The environment configuration device 121 includes an environment configuration changing unit 121a and an environment configuration applying unit 121b. The connected destination terminal device 12 is, for example, a device configured by a general-purpose or dedicated computer including a processor, a memory, a communication device, an input / output device, etc. executing a predetermined program. This computer may be provided with one processor and memory, or may be provided with a plurality of processors and memory. This program may be installed in the computer, or may be recorded in a ROM or the like in advance. Also, instead of an electronic circuit that realizes a functional configuration by loading a program like a CPU, some or all of the processing units may be configured using an electronic circuit that realizes a processing function without using a program. Also, the electronic circuit constituting the connected destination terminal device 12 may include a plurality of CPUs. Specific examples of the connected destination terminal device 12 include a personal computer terminal device, a server device, a tablet terminal device, a smartphone terminal device, a smartwatch terminal device, DaaS (Desktop as a Service), a virtual desktop infrastructure, a virtual machine, and a virtual terminal device operating on a physical device or a virtual device such as a container or container-type virtualization.

[0012] <Pretreatment> As a pre - process, a plurality of environment configuration patterns 105 are stored in the storage unit 122 of the connected terminal device 12. The environment configuration pattern 105 is a pattern representing the environment configuration of the remote operation system 1. The environment configuration (for example, environment, system configuration, operating environment) means the environment and management settings for the system or software to operate. The environment configuration pattern 105 may represent the environment configuration of the local terminal device 11, or the environment configuration of the connected terminal device 12, or the environment configuration of both the local terminal device 11 and the connected terminal device 12, or the environment configuration between the local terminal device 11 and the connected terminal device 12. The environment configuration may include hardware settings, software settings, network settings, power supply settings, security settings, or settings for the correspondence relationship with other devices and peripheral devices. For example, the environment configuration may be the setting of the input (for example, keyboard, mouse, etc.) of the local terminal device 11 or the connected terminal device 12, or the setting of the output (for example, screen configuration, screen layout, relationship of icon arrangement, resolution, size of task bar icon, character, color scheme, arrangement, etc.), or other settings of the user interface 111, or the power supply setting of the local terminal device 11 or the connected terminal device 12, or the setting of the correspondence relationship between the input / output of the local terminal device 11 and the input / output of the connected terminal device 12, or the setting of the network 10. FIG. 5 exemplifies the environment configuration pattern 105. The environment configuration pattern 105 exemplified in FIG. 5 has a usage situation column 1051 representing the usage situation (usage scene) (for example, "during business negotiation"), an environment configuration pattern name column 1052 representing the environment configuration pattern name (for example, "projector presentation"), an environment configuration item column 1053 representing the items of the environment configuration, and an environment configuration value column 1054 representing the values of the environment configuration items.In the environmental configuration item column 1053 of this example, there are set items such as "Mouse cursor size" representing the percentage (%) of the mouse cursor size relative to the standard, "Character size" representing the character size (%) relative to the standard, "Image magnification ratio" representing the image magnification ratio (%) relative to the standard, "IME input mode" representing the input mode (Japanese / English / disabled) of the IME (Input Method Editor), "Resolution" representing the resolution (dpi), "Number of colors" representing the color depth (bit), "Desktop display mode" representing whether to display on the desktop (ON / OFF), "Background display / non-display mode" representing whether to display the background (ON / OFF), "Animation mode" representing whether to display animation (ON / OFF), "Icon spacing" representing the icon spacing (pixels), "Volume level" representing the volume level (%) relative to the standard, and "Notification mode" representing whether to perform notifications (ON / OFF). In the environmental configuration value column 1054 of this example, the values "150", "150", "200", "Japanese", "264", "64", "OFF", "OFF", "OFF", "64", "20", and "OFF" corresponding to the environmental configuration item column 1053 are set respectively.

[0013] Also, as preprocessing, in the storage unit 123 of the connection destination terminal device 12, remote information representing the provided information provided to the local terminal device 11 by remote operation of the local terminal device 11 is stored. The provided information is some kind of five-sense information, and for example, it may be text information, image information, video information, virtual space information, acoustic information, or voice information.

[0014] <Remote operation process> Next, the remote operation process of this embodiment will be described. The local terminal device 11 (FIGS. 1 and 2: the first terminal device) remotely operates the connection destination terminal device 12 (FIGS. 1 and 3: the second terminal device) to obtain the remote information UI2 (the second information), and outputs the output information UI1 (the first information) obtained from the remote information UI2 to the user interface 111, and the user interface 111 performs an output based on the output information UI1. Details will be described below.

[0015] <Step S101> User 100 performs an operation input for remote control on the user interface 111 of the local terminal device 11 (Fig. 2). The operation input may be an operation or input to a mouse, keyboard, touch panel, etc., an image input to a camera, etc., a voice input to a microphone, etc., or other input. This operation input is for remotely operating the connected terminal device 12 and obtaining remote information corresponding to the operation input from the connected terminal device 12. The remote information corresponding to the operation input may be, for example, text information corresponding to the operation input, image information corresponding to the operation input, video information corresponding to the operation input, virtual space information corresponding to the operation input, voice information corresponding to the operation input, or acoustic information corresponding to the operation input. The text information corresponding to the operation input is, for example, the input characters or the text file specified by the input. The image information corresponding to the operation input is, for example, the image drawn or changed by the input or the specified image file. The video information corresponding to the operation input is, for example, the video drawn or changed by the input or the specified video file. The virtual space information corresponding to the operation input is the video of the virtual space drawn or changed by the input. The acoustic information corresponding to the operation input is, for example, the music played by the input or the specified acoustic file. The voice information corresponding to the operation input is, for example, the voice information responding to the input or the voice file specified by the input.

[0016] <Step S102> The user interface 111 (Figure 2) sends the content of the operation input to the remote operation unit 113. The remote operation unit 113 generates remote operation information 103 for specifying remote information corresponding to the operation input and sends it to the communication interface 115. The remote operation information 103 may be information that directly specifies remote information corresponding to the operation input (for example, information representing the input characters or a text file specified by the input), or may be information that indirectly specifies it (for example, information representing the operation content for drawing or changing an image, etc.). The communication interface 115 transmits the remote operation information 103 to the destination terminal device 12 via the network 10.

[0017] <Step S103> The communication interface 125 of the destination terminal device 12 (Figure 3) receives the remote operation information 103 and sends it to the remote information providing unit 124. The remote information providing unit 124 uses the remote operation information 103 to extract the remote information specified by the remote operation information 103 from the storage unit 123. The extracted remote information is sent to the environment configuration application unit 121b of the environment configuration device 121.

[0018] <Step S104> The environment configuration application unit 121b changes or maintains the transmitted remote information according to the current environment configuration, and generates and outputs remote information 104 (remote information UI2: second information) according to the current environment configuration of the remote operation system 1. The environment configuration is as described above. The current environment configuration may be represented by any one of the environment configuration patterns 105, may not be represented by the environment configuration pattern 105, or may be a modification of the environment configuration represented by the environment configuration pattern 105. The environment configuration is changed by the environment configuration change unit 121a according to a change in the status information representing at least any one of the usage status, device status, and line status of the local terminal device 11 (first terminal device). The initial state (default state) of the environment configuration may be set in advance or may be set based on an input. The remote information 104 corresponding to a specific environment configuration and the change process of the environment configuration will be described later. The remote information 104 according to the current environment configuration is sent to the communication interface 125. The communication interface 125 transmits the remote information 104 to the local terminal device 11 via the network 10.

[0019] <Step S105> The communication interface 115 of the local terminal device 11 (Figure 2) receives the remote information 104 and sends it to the remote information control unit 114. The remote information control unit 114 obtains and outputs output information (UI1: the first information) for output from the user interface 111 based on the remote information 104 (remote information UI2: the second information). For example, if the remote information 104 is text information, the output information is information for displaying text on the user interface 111. For example, if the remote information 104 is image information or video information, the output information is information for displaying an image or video on the user interface 111. For example, if the remote information 104 is virtual space information, the output information is information for displaying an image or video of the virtual space from the user interface 111. For example, if the remote information 104 is acoustic information or voice information, the output information is information for outputting sound or voice from the user interface 111. The output information (UI1: the first information) is output to the user interface 111. The user interface 111 performs an output (for example, display of text, image, or video, output of sound or voice, etc.) based on the output information. The user 100 perceives the output from the user interface 111.

[0020] The processes of steps S101 to S105 described above may be executed each time an operation input is performed on the user interface 111 of the local terminal device 11 (Figure 2), or may be executed each time the operation input to the user interface 111 satisfies a specific condition. Examples of the specific condition include that the operation amount of the operation input exceeds a predetermined amount, and that the time interval of the operation input has elapsed a predetermined time, etc.

[0021] <Remote information 104 according to a specific environmental configuration> Next, the remote information 104 according to a specific environmental configuration will be described. The remote information 104 according to a specific environmental configuration is remote information that conforms to this environmental configuration and / or remote information that is suitable for this environmental configuration.

[0022] For example, if a specific environmental configuration includes hardware settings, the remote information 104 corresponding to the specific environmental configuration may include remote information that conforms to this hardware setting, or may include remote information suitable for this hardware setting. For example, if the user interface 111 of the local terminal device 11 includes a plurality of displays (e.g., an in-vehicle display and an external display, etc.), and a specific display is set according to a specific environmental configuration, the remote information 104 corresponding to the specific environmental configuration may be remote information for performing an output suitable for the display set in this environmental configuration (e.g., a screen configuration suitable for the set display, a screen layout, a relationship of icon arrangements, a resolution, a size of a task bar icon, characters, a color scheme, an arrangement, etc.). For example, the same applies when a specific projector is set instead of a display according to a specific environmental configuration. For example, if the user interface 111 of the local terminal device 11 includes a headset and is set to use the headset according to a specific environmental configuration, the remote information 104 corresponding to the specific environmental configuration may be remote information for performing an output suitable for the headset set in this environmental configuration (e.g., a display such as a screen configuration suitable for the headset, an audio output with a volume suitable for the headset, etc.).

[0023] For example, when a specific environmental configuration includes software settings, the remote information 104 corresponding to the specific environmental configuration may include remote information that conforms to the software settings or remote information that is suitable for the software settings. For example, if the user interface 111 of the local terminal device 11 includes a display and the output of this display (screen configuration, screen layout, relationship of icon arrangement, resolution, size of task bar icon, character, color scheme, arrangement, etc.) is set by a specific environmental configuration, the remote information 104 corresponding to the specific environmental configuration may be remote information for performing the output set in this environmental configuration. For example, when it is set by a specific environmental configuration to display a presentation screen full screen and enlarge the mouse cursor for use as a pointer, the remote information 104 corresponding to the specific environmental configuration may be remote information for performing the output set in this environmental configuration. The same applies when it is possible to set the use of a specific projector, headset, etc. instead of the display.

[0024] For example, when a specific environmental configuration includes the settings of the network 10 used for communication between the local terminal device 11 and the destination terminal device 12, the remote information 104 corresponding to the specific environmental configuration may include remote information that conforms to the network settings or remote information that is suitable for the network settings. For example, if the network 10 includes a plurality of networks (for example, a mobile line and a Wi-Fi (registered trademark) line) and a specific network is set by a specific environmental configuration, the remote information 104 corresponding to the specific environmental configuration may be remote information with an information amount suitable for the network set in this environmental configuration. For example, the information amount of the remote information 104 when using a network with a slow communication speed (for example, a mobile line) may be smaller than the information amount of the remote information 104 when using a network with a faster communication speed (for example, a Wi-Fi line). The information amount of the remote information 104 can be changed, for example, by the resolution, number of colors, update frequency, compression rate of the image output from the user interface 111, whether to display the background, the bandwidth, update frequency, compression rate, etc. of the audio information output from the speaker.

[0025] For example, when a specific environmental configuration includes the power supply setting of the local terminal device 11 or the connected terminal device 12, the remote information 104 corresponding to the specific environmental configuration may include remote information adapted to this power supply setting or remote information suitable for this power supply setting. For example, when it is possible to set whether to perform the power supply means of the local terminal device 11 by external power supply or by an internal battery according to the environmental configuration, the remote information 104 corresponding to the specific environmental configuration may include remote information on power consumption suitable for the power supply means set in the specific environmental configuration. For example, the resolution, update frequency, etc. of the screen represented by the remote information 104 when power is supplied by the internal battery are made lower than those of the screen represented by the remote information 104 when power is supplied by external power supply, and the power consumption when power is supplied by the internal battery may be suppressed compared to the power consumption when power is supplied by external power supply.

[0026] For example, when a specific environment configuration includes security settings, the remote information 104 corresponding to the specific environment configuration may include remote information that conforms to this security setting, or may include remote information suitable for this security setting. For example, when it is possible to set the local terminal device 11 to a high security level or a lower security level than that, the remote information 104 corresponding to the specific environment configuration may be remote information that conforms to the security level set in this environment configuration. For example, the remote information that conforms to a high security level is information for displaying a privacy filter together with confidential information so that the confidential information is difficult to see on the user interface 111, and the remote information that conforms to a low security level may be information for displaying the confidential information on the user interface 111 without displaying the privacy filter. For example, the remote information that conforms to a high security level is information set so that a click operation is required to display the confidential information on the user interface 111, and the remote information that conforms to a low security level may be information set so that a click operation is not required to display the confidential information on the user interface 111.

[0027] <Environmental configuration change process> Next, the environmental configuration change process will be described. The environmental configuration change unit 121a of the environmental configuration device 121 of the connection destination terminal device 12 (FIG. 3) changes the environmental configuration of the remote operation system according to a change in the status information (for example, at least one of the usage record 101 and the local terminal record 102) representing at least one of the usage status, device status, and line status of the local terminal device 11 (the first terminal device). The details of the environmental configuration change process will be illustrated with reference to FIG. 8.

[0028] <Step S11> The status information acquisition unit 112 of the local terminal device 11 (FIG. 2) acquires (updates) the status information representing at least one of the usage status, device status, and line status of the local terminal device 11 (the first terminal device).

[0029] The usage situation (usage scenario) of the hand-held terminal device 11 means the situation and environment in which the hand-held terminal device 11 is used. Examples of the usage situation of the hand-held terminal device 11 include the physical environment in which the hand-held terminal device 11 is used (e.g., during movement, stopped, indoors, outdoors, away from home, at the company, daytime, nighttime, season, time, location, inclination, moving speed, ambient noise, ambient brightness, etc.), the social environment in which the hand-held terminal device 11 is used (e.g., during a meeting, business negotiation, presentation, commuting, drafting, creating a form, etc.), the purpose for which the hand-held terminal device 11 is used and the functions used on the hand-held terminal device 11 (e.g., business use, industrial use, design use, accounting use, household use, educational use, charging, not charging, display brightness, with or without face authentication, etc.), the type of user of the hand-held terminal device 11 (e.g., experts, students, general consumers, etc.), and specific situations or scenarios in which the hand-held terminal device 11 is used (e.g., emergencies, daily work, entertainment activities, etc.). In this embodiment, the information representing the usage situation of the hand-held terminal device 11 will be referred to as the usage situation record 101. Fig. 4A illustrates the configuration of the usage situation record 101 of this embodiment. The usage situation record 101 illustrated in Fig. 4A has a creation date and time column 1011 representing the creation date and time (YYYY / MM / DD HH:MM:SS) (e.g., 2023 / 11 / 20 14:20), a usage situation item column 1012 representing usage situation items, and a usage situation value column 1013 representing the values of the usage situation items. In the usage situation item column 1012 of this example, items such as "User Setting" representing the usage situation set by the user, "Display Brightness" representing the percentage (%) of the brightness with respect to the reference of the display constituting the user interface 111, "Location Information" representing the location (GPS coordinates) of the hand-held terminal device 11, "Inclination" representing the inclination (Degree), "Moving Speed" representing the moving speed (Km / h), "Charging" representing whether the hand-held terminal device 11 is charging or using the battery, "Ambient Noise" representing the noise level (dB) around the hand-held terminal device 11, and "Face Recognition" representing whether the hand-held terminal device 11 uses face authentication are set. In the usage situation value column 1013 of this example, values corresponding to the usage situation item column 1012, namely, "During business negotiation", "80", "41.40338, 2.17403", "40", "5", "Charging", "40", and "Yes", are set.

[0030] The device status of the hand-held terminal device 11 means the specifications (specs), current state, and operating status of the hand-held terminal device 11. Examples of the device status of the hand-held terminal device 11 include the model name of the hand-held terminal device 11, the OS version, the specifications, state, and status of the display, mouse, speaker, etc. that make up the user interface 111. The line status of the hand-held terminal device 11 means the static and / or dynamic status of the line (network) used for communication between the hand-held terminal device 11 and the destination terminal device 12. Examples of the line status of the hand-held terminal device 11 include the type of network, communication speed, etc. In this embodiment, information representing at least a part of the device status and line status of the hand-held terminal device 11 will be referred to as the hand-held terminal record 102. Fig. 4B illustrates the configuration of the hand-held terminal record 102 of this embodiment. The hand-held terminal record 102 illustrated in Fig. 4B has a creation date and time column 1021 representing the creation date and time (YYYY / MM / DD HH:MM:SS) (for example, 2023 / 11 / 20 14:20), a device / line item column 1022 representing the items of the device status and line status of the hand-held terminal device 11, and a device / line item value column 1023 representing the values of the items of the device status and line status. In the device / line item column 1022 of this example, there are set items such as "Model" representing the model of the hardware constituting the hand-held terminal device 11 (for example, model name·OS version), "Display Type" representing the type (internal / external) of the display that makes up the user interface 111, "Resolution" representing the resolution (number of horizontal pixels × number of vertical pixels) of the display that makes up the user interface 111, "Keyboard Type" representing the type of keyboard (virtual / external) of the hardware constituting the hand-held terminal device 11, "Key Mapping" representing the key array name, "Mouse Type" representing the mouse type (virtual / external), "Button Layout" representing the number of displayed buttons, and "Speaker Type" representing the type of speaker (speaker / earphone). In the device / line item value column 1023 of this example, the values "iPhone (registered trademark) 13 Pro", "Internal Display", "1080×1920", "Virtual Keyboard", "Japanese", "Virtual Mouse", "2 Buttons", "Speaker", "WIFI (registered trademark)", and "600" corresponding to the device / line item column 1022 are set respectively.

[0031] Such situation information may be obtained based on information (e.g., profiles) obtained from the hardware (including, for example, a CPU, sensors, etc.) and software (including, for example, an OS, applications, etc.) that constitute the local terminal device 11, or may be obtained based on information (e.g., tasks performed by the user 100) input by the user 100 to the user interface 111, or may be obtained by applying any of this information to a model, determination conditions, etc. Further, the process of the first step S11 may be executed, for example, when the local terminal device 11 connects to the destination terminal device 12 (e.g., remote desktop connection), or may be executed when the user 100 inputs information to the user interface 111, or may be executed in conjunction with other processes.

[0032] <Step S12> The situation information acquisition unit 112 compares the situation information obtained in the immediately preceding step S11 with the situation information one step further before that (for example, the initial value of the situation information, or the situation information obtained in step S11 one step further before the immediately preceding step S11), and determines whether the change in the situation information exceeds a criterion. That the change in the situation information exceeds the criterion may mean that the magnitude of the change in the situation information (for example, the magnitude of the change in resolution or communication speed) exceeds a first criterion, or that the number of items of the changed situation information (for example, the number of changed usage status value columns 1013 or device / line item value columns 1023) exceeds a second criterion, or that the number of items of the situation information whose magnitude of value change exceeds the first criterion. The first criterion is, for example, a value larger than the minimum value of the magnitude of the change in the situation information. The second criterion is, for example, a value larger than 1. The magnitude of the change in the situation information may be the difference between each value included in the situation information (for example, at least any one value of the usage status value column 1013 and the device / line item value column 1023), or may be the difference in the norm (magnitude) of the vector having each value included in the situation information as an element. Also, the situation information acquisition unit 112 may determine only whether the change in the usage status record 101 exceeds the criterion, or may determine only whether the change in the local terminal record 102 exceeds the criterion, or may determine only whether the changes in the usage status record 101 and the local terminal record 102 exceed the criterion, or may determine only whether the change in the usage status record 101 or the local terminal record 102 exceeds the criterion. When there is no situation information before the situation information obtained in the immediately preceding step S11, it may be determined whether the change in the situation information obtained in the immediately preceding step S11 with respect to the predetermined situation information exceeds the criterion, or it may be determined that the change in the situation information exceeds the criterion. Here, if the change in the situation information does not exceed the criterion, the process returns to the process of step S11. On the other hand, if the change in the situation information exceeds the criterion, the process proceeds to step S13.

[0033] <Step S13> The situation information acquisition unit 112 sends the situation information (for example, the usage record 101 and the local terminal record 102) obtained in the immediately preceding step S11 to the communication interface 115. The communication interface 115 transmits the situation information to the destination terminal device 12 via the network 10.

[0034] <Step S14> The communication interface 125 of the destination terminal device 12 (Fig. 3) receives the situation information (for example, the usage record 101 and the local terminal record 102) and sends it to the environment configuration change unit 121a of the environment configuration device 121. The environment configuration change unit 121a changes the environment configuration of the remote operation system 1 according to the change in the situation information. For example, the environment configuration change unit 121a of this embodiment selects the environment configuration pattern 105 corresponding to the situation information from among the plurality of environment configuration patterns 105 stored in the storage unit 122, and changes the environment configuration of the remote operation system 1 based on the selected environment configuration pattern 105. For example, the situation information is a combination of the usage record 101 (information representing the usage status of the first terminal device) and the local terminal record 102 (at least one of the information representing the device status of the first terminal device and the line status), and the environment configuration change unit 121a changes the environment configuration of the remote operation system 1 according to the change in the combination (situation information) of the usage record 101 and the local terminal record 102. The environment configuration change unit 121a may change the environment configuration of the remote operation system 1 to the environment configuration represented by the selected environment configuration pattern 105, or may change it to an environment configuration obtained by modifying the environment configuration represented by the selected environment configuration pattern 105. Note that the environment configuration pattern 105 corresponding to the situation information is an environment configuration pattern estimated to be suitable for this situation information. Also, the process of step S14 of this embodiment is executed when it is determined in step S12 that the change in the situation information exceeds the standard, and is not executed when the change in the situation information does not exceed the standard. That is, the environment configuration change unit 121a of this embodiment does not change the environment configuration when the change in the situation information does not exceed the standard, and changes the environment configuration when the change in the situation information exceeds the standard. Hereinafter, a method for changing the environment configuration according to the situation information will be exemplified.

[0035] <Example 1>In Example 1, based on the received situation information, the environment configuration change unit 121a scores each of the plurality of environment configuration patterns 105, selects the environment configuration pattern 105 with the highest score, and changes the current environment configuration to the environment configuration specified by the selected environment configuration pattern 105. For example, points corresponding to each item of the usage record 101 and the local terminal record 102 may be defined for each of the plurality of environment configuration patterns 105. In this case, the environment configuration change unit 121a specifies the points corresponding to each item of the usage record 101 and the local terminal record 102 for each of the plurality of environment configuration patterns 105, adds or subtracts the specified points for each environment configuration pattern 105 to calculate a score, and may select the environment configuration pattern 105 with the highest or lowest calculated score. A specific example of this is shown. For example, as illustrated in FIG. 6, an applicability rank table 106 may be set for each of the plurality of environment configuration patterns 105, and points (importance) corresponding to each item of the usage record 101 and the local terminal record 102 may be defined. The applicability rank table 106 illustrated in FIG. 6 corresponds to each of the environment configuration patterns 105 (FIG. 5). The applicability rank table 106 in this example has a usage situation column 1061 representing the usage situation (for example, "during negotiation"), an environment configuration pattern name column 1062 representing the name of the environment configuration pattern (for example, "projector presentation"), an evaluation target item column 1063 representing the evaluation target items of the usage record 101 and the local terminal record 102, a determination condition column 1064 representing the determination conditions for each evaluation target item, and an importance column 1065 representing the points (importance) for each evaluation target item. As illustrated in FIG. 9, first, the environment configuration change unit 121a selects, from among the plurality of applicability rank tables 106 (FIG. 6), those in which the value of the usage situation described in the usage situation column 1061 matches the value (for example, "during negotiation") in the usage situation value column 1013 corresponding to the "user setting" in the usage situation item column 1012 of the usage record 101 (FIG. 4A) (step S141).Next, for the evaluation target item column 1063 of the selected applicability rank table 106 (Fig. 6), the environment configuration change unit 121a determines whether the usage record 101 and the local terminal record 102 satisfy the determination conditions in the determination condition column 1064. If they are satisfied, the importance in the importance column 1065 is specified as a point. The environment configuration change unit 121a adds up the points specified for each selected applicability rank table 106, and obtains a score (total importance value) for the selected applicability rank table 106. Here, the applicability rank table 106 corresponds to the environment configuration pattern 105 (Fig. 5) having the same value in the environment configuration pattern name column 1052 (for example, "projector presentation") as the environment configuration pattern name column 1062 (for example, "projector presentation") of the applicability rank table 106. The score obtained for the selected applicability rank table 106 becomes the score of the environment configuration pattern 105 corresponding to the selected applicability rank table 106. Fig. 7 exemplifies the scores (total importance values) calculated for a plurality of environment configuration patterns 105. In this example, the environment configuration change unit 121a selects the environment configuration pattern 105 corresponding to the applicability rank table 106 with the highest score, and changes the current environment configuration to the environment configuration specified by the selected environment configuration pattern 105. In the example of Fig. 7, scores (total importance values) are calculated for the applicability rank tables 106 whose environment configuration pattern names are "display presentation", "projector presentation", "video playback", "price management", and "delivery management", respectively. The respective scores (total importance values) are "14", "18", "5", "9", and "1", and the respective ranks are "2", "1", "4", "3", and "5". In this case, the environment configuration change unit 121a selects the environment configuration pattern 105 (environment configuration pattern name: projector presentation) corresponding to the applicability rank table 106 with the highest score, and changes the current environment configuration to the environment configuration specified by the selected environment configuration pattern 105 (step S142).

[0036] <Example 2> As illustrated in FIG. 9, in Example 2, the environment configuration change unit 121a identifies the usage status of the local terminal device 11 (the first terminal device) using the received situation information, and selects an environment configuration pattern 105 based on the identified usage status. The identification of the usage status of the local terminal device 11 may be performed based only on the usage status record 101, may be performed by estimating the usage status from the local terminal record 102, or may be performed by integrating the usage status identified from the usage status record 101 and the usage status estimated from the local terminal record 102 (step S141). Next, the environment configuration change unit 121a modifies the environment configuration specified by the environment configuration pattern 105 selected in step S141 based on at least one of the device status and the line status specified by the local terminal record 102. That is, the environment configuration is modified according to changes in the device status and the line status specified by the local terminal record 102. For example, the environment configuration is modified according to the availability / specification changes of the hardware and software constituting the local terminal device 11 specified by the local terminal record 102. For example, the value of some items of the environment configuration specified by the environment configuration pattern 105 selected in step S141 (for example, the environment configuration value column 1054 "264 (dpi)" corresponding to the "resolution" in the environment configuration item column 1053 of FIG. 5) is set to the value of some items of the device status and the line status in the local terminal record 102 (for example, the value corresponding to the "resolution" in the device / line item column 1022 of FIG. 4B, "1080×1920" in the device / line item value column 1023 (resolution in dpi display)) is modified. The environment configuration change unit 121a changes the current environment configuration to the modified environment configuration (step S142).

[0037] <Example 3> In Example 3, machine learning is performed using information based on past situation information (for example, the usage status record 101 and the local terminal record 102) and the corresponding correct environment configuration pattern 105 as learning data, and the environment configuration pattern 105 corresponding to the newly input situation information is estimated using the model obtained thereby. A specific example thereof is shown below.

[0038] Example 3-1: For example, the environment configuration changing unit 121a performs machine learning using the situation information and the correct environment configuration pattern 105 corresponding thereto as learning data, and obtains a model for estimating the environment configuration pattern 105 corresponding to the situation information with the situation information as input. The environment configuration changing unit 121a applies the sent situation information to the model to obtain an estimation result of the environment configuration pattern 105 corresponding to the situation information, and changes the current environment configuration to the environment configuration specified by the estimation result of the environment configuration pattern 105.

[0039] Example 3-2: For example, the environment configuration changing unit 121a performs machine learning using the situation information before and after update (for example, the usage record 101 and the local terminal record 102 before and after update) and the correct environment configuration pattern 105 corresponding thereto as learning data, and obtains a model for estimating the environment configuration pattern 105 corresponding to the situation information before and after update with the situation information before and after update as input. The environment configuration changing unit 121a applies the before and after update of the sent situation information to the model to obtain an estimation result of the environment configuration pattern 105 corresponding to the situation information before and after update, and changes the current environment configuration according to the environment configuration before and after update specified by the estimation result of the environment configuration pattern 105.

[0040] Example 3-3: For example, the environment configuration changing unit 121a performs machine learning using the difference between the situation information before and after update (for example, the difference items of the usage record 101 and the local terminal record 102 before and after update) and the difference of the correct environment configuration pattern 105 corresponding thereto as learning data, and obtains a model for estimating the difference of the environment configuration pattern 105 corresponding to the difference between the situation information before and after update with the difference between the situation information before and after update as input. The environment configuration changing unit 121a applies the difference between the before and after update of the sent situation information to the model to obtain an estimation result of the difference of the environment configuration pattern 105 corresponding to the difference between the situation information before and after update, and changes the current environment configuration according to the difference of the environment configuration specified by the estimation result of the difference of the environment configuration pattern 105.

[0041] The environment configuration change unit 121a sends information representing the changed environment configuration to the environment configuration application unit 121b.

[0042] <Step S15> The environment configuration application unit 121b (Fig. 3) receives the information representing the environment configuration sent from the environment configuration change unit 121a (Step S14) and the remote information sent from the remote information providing unit 124 (Step S103). The environment configuration application unit 121b changes the sent remote information according to the environment configuration, and generates and outputs remote information 104 (remote information UI2: second information) corresponding to the current environment configuration of the remote operation system 1. That is, the environment configuration application unit 121b changes at least a part of the remote information 104 (remote information UI2: second information) according to the change in the environment configuration (changes from the past remote information 104). That is, the remote information 104 corresponding to the changed environment configuration is at least partially different from the remote information 104 corresponding to the environment configuration before the change. For example, first, the environment configuration application unit 121b changes the environment configuration of the connected terminal device 12 (for example, hardware settings, software settings, etc.) to the changed environment configuration represented by the information sent from the environment configuration change unit 121a (setting change). For example, next, the environment configuration application unit 121b changes the remote information sent from the remote information providing unit 124 according to the changed environment configuration, and generates and outputs remote information 104 (remote information UI2: second information). Note that the remote information 104 corresponding to a specific environment configuration is as described above. The remote information 104 corresponding to the current environment configuration is sent to the communication interface 125. The communication interface 125 transmits the remote information 104 to the local terminal device 11 via the network 10. The process of Step S15 is executed as the process of Step S104 in the remote operation processes of Steps S101 to S105 described above. Also, the process of changing the environment configuration is executed in parallel with the remote operation process, and after Step S15, the process returns to Step S11.

[0043] <Step S16> Furthermore, in parallel with the processes of Steps S11 to S15 described above, the environment configuration change unit 121a (Fig. 3) monitors whether the local terminal device 11 has been disconnected from the connected terminal device 12.

[0044] <Step S17> When the environment configuration change unit 121a determines that the local terminal device 11 is not disconnected from the connection destination terminal device 12 (that is, when it determines that the local terminal device 11 is connected to the connection destination terminal device 12), the process of step S16 is continued. On the other hand, when the environment configuration change unit 121a determines that the local terminal device 11 has been disconnected from the connection destination terminal device 12, the process of step S18 is executed.

[0045] <Step S18> When the environment configuration change unit 121a determines that the local terminal device 11 has been disconnected from the connection destination terminal device 12, it returns the environment configuration of the remote operation system 1 to the initial state (default state). Then, it proceeds to step S15. As a result, the environment configuration of the local terminal device 11 that has been disconnected from the connection destination terminal device 12 returns to the initial state. Consequently, when operating the local terminal device 11 that has been disconnected from the connection destination terminal device 12, the user 100 will handle the local terminal device 11 with the initial state environment configuration. This can eliminate the inconvenience and problems caused by using the local terminal device 11 whose environment configuration has been changed to be suitable for remote operation for processes other than remote operation.

[0046] <Features of the First Embodiment> In the first embodiment, the environment configuration of the remote operation system 1 is changed according to a change in the status information indicating at least any one of the usage status, device status, and line status of the local terminal device 11 (first terminal device). Thereby, even when the situation regarding the local terminal device 11 changes, high operability can be achieved without manually changing the environment configuration. As a result, it is possible to achieve both operability and convenience when the situation regarding the local terminal device 11 changes.

[0047] In the first embodiment, for example, the environmental configuration of the remote operation system 1 is changed according to changes in the combination of the usage record 101 (information representing the usage status of the first terminal device) and the local terminal record 102 (at least one of the information representing the device status and the line status of the first terminal device). Thereby, even when the device status and the line status do not change, a decrease in operability due to a change in the usage status can be suppressed, and even when the usage status does not change, a decrease in operability due to a change in at least one of the device status and the line status can be suppressed.

[0048] In the first embodiment, when the change in the status information does not exceed the reference, the environmental configuration is not changed, and when the change in the status information exceeds the reference, the environmental configuration is changed. Thereby, it is possible to suppress a situation where the environmental configuration is frequently updated in response to a slight change in the status information (for example, the environmental configuration is updated every time the status information changes), which may instead reduce the operability.

[0049] [Modification Example 1 of the First Embodiment] In the first embodiment, the local terminal device 11 transmits the newly acquired situation information to the connection destination terminal device 12 only when the change in the situation information exceeds the standard (Fig. 8: Steps S11 - S13). Thereby, the environment configuration change unit 121a of this embodiment does not change the environment configuration when the change in the situation information does not exceed the standard, and changes the environment configuration when the change in the situation information exceeds the standard. However, the situation information acquired by the local terminal device 11 may be always transmitted to the connection destination terminal device 12, and the environment configuration change unit 121a of the connection destination terminal device 12 may not change the environment configuration when the change in the situation information does not exceed the standard, and change the environment configuration when the change in the situation information exceeds the standard. For example, as illustrated in Fig. 10, every time the situation information acquisition unit 112 of the local terminal device 11 (Fig. 2) acquires the situation information, it sends the situation information to the communication interface 115, and the communication interface 115 may transmit the situation information to the connection destination terminal device 12 via the network 10 (Steps S11, S13). In this case, the communication interface 125 of the connection destination terminal device 12 (Fig. 3) receives the situation information and sends it to the environment configuration change unit 121a of the environment configuration device 121. The environment configuration change unit 121a compares the obtained situation information with the situation information before that, and determines whether the change in the situation information exceeds the standard. Here, if the change in the situation information does not exceed the standard, the process returns to the process of Step S11. On the other hand, if the change in the situation information exceeds the standard, the process proceeds to Step S14 (Step S12’). Also by such processing, the environment configuration change unit 121a of the connection destination terminal device 12 does not change the environment configuration when the change in the situation information does not exceed the standard, and changes the environment configuration when the change in the situation information exceeds the standard. Note that without making the determination in Step S12’, the environment configuration change unit 121a may set the environment configuration according to the received situation information every time it receives the situation information.

[0050] [Modification Example 2 of the First Embodiment] The environmental configuration of the remote operation system 1 may be changed according to only the change of either the usage record 101 (information representing the usage status of the first terminal device) or the local terminal record 102 (at least one of the information representing the device status and the line status of the first terminal device). For example, either the usage record 101 or the local terminal record 102 may be omitted.

[0051] [Modification Example 3 of the First Embodiment] In the first embodiment, after step S15, the process returns to step S11, and the process of changing the environmental configuration is repeated. However, the process of changing the environmental configuration may end after step S15.

[0052] [Second Embodiment] The remote operation system further includes peripheral devices, and the environmental configuration of the remote operation system may be changed according to the change of the status information representing at least one of the usage status, device status, and line status of at least one of the local terminal device 11 and the peripheral devices. Hereinafter, the description will focus on the differences from the matters described so far, and the same reference numerals will be used for the matters described so far to simplify the description.

[0053] [Overall Configuration] As illustrated in FIG. 1, the remote operation system 2 of this embodiment includes a local terminal device 21 (first terminal device), peripheral devices 23-1,..., 23-N (first terminal devices), and a connected destination terminal device 12 (second terminal device). Here, each of the peripheral devices 23-1,..., 23-N is denoted as peripheral device 23-n. However, N is an integer of 1 or more, and n = 1,..., N. The local terminal device 21 and the connected destination terminal device 12 are configured to be communicable through the network 10. The peripheral devices 23-1,..., 23-N are connected to the local terminal device 21 by wire or wirelessly.

[0054] [Local Terminal Device 21 (First Terminal Device)] The hand-held terminal device 21 is a device directly operated by the user 100 and is arranged near the user 100 (for example, at hand). As illustrated in FIG. 2, the hand-held terminal device 21 of this embodiment includes a user interface 111, a situation information acquisition unit 112, a remote operation unit 113, a remote information control unit 114, a communication interface 115, a memory 116, and a peripheral device control unit 216. Information input to the hand-held terminal device 21 and information output from each part of the hand-held terminal device 21 are stored in the memory 116 one by one and read out as necessary. The hand-held terminal device 21 is a device configured by a general-purpose or dedicated computer including, for example, a processor, a memory, a communication device, and input / output devices such as a display and a speaker, executing a predetermined program. Specific examples of the hand-held terminal device 21 include a personal computer terminal device, a tablet terminal device, a smartphone terminal device, a smartwatch terminal device, and the like.

[0055] <Peripheral device 23-n> The peripheral device 23-n is connected to the hand-held terminal device 21 by wire or wirelessly and is a device used by the user 100 through the hand-held terminal device 21. As illustrated in FIG. 11, the peripheral device 23-n of this embodiment includes an input / output control unit 212-n and a user interface 231-n. Input to the user interface 231-n may be operation input (for example, mouse input or keyboard input), contact input (for example, touch panel input), non-contact input (for example, touchless input or gesture input), image input (for example, image reading), voice input, or other input. Output from the user interface 231-n is some kind of sensory information, and for example, may be image output, video output, virtual space output, text output, or voice output. Specific examples of the peripheral device 23-n include a display, a projector, a headset, a keyboard, a mouse, a microphone, a speaker, and the like.

[0056] <Preprocessing> The preprocessing of the second embodiment is the same as that of the first embodiment, except that peripheral devices 23-1, …, 23-N are taken into consideration. That is, the environmental configuration pattern 105 of the second embodiment may represent not only those exemplified in the first embodiment but also the environmental configuration of at least any one of peripheral devices 23-1, …, 23-N. For example, the environmental configuration may be not only those exemplified in the first embodiment but also the setting of at least any one of the inputs of peripheral devices 23-1, …, 23-N, the setting of the outputs, the setting of other user interfaces 231-n, or the setting of the correspondence between the input / output of the local terminal device 11 and the input / output of at least any one of peripheral devices 23-1, …, 23-N.

[0057] <Remote operation processing> The local terminal device 21 (FIGS. 1 and 2: the first terminal device) remotely operates the destination terminal device 12 (FIGS. 1 and 3: the second terminal device) to obtain the remote information UI2 (the second information), and outputs the output information UI1 (the first information) obtained from the remote information UI2 to the user interface 111 or the user interface 231-n of the peripheral device 23-n (FIG. 11: the first terminal device), and the user interface 111 or the user interface 231-n performs an output based on the output information UI1.

[0058] The differences between the remote operation processing of the first embodiment and that of the second embodiment are that the remote operation system 1 is replaced by the remote operation system 2, the local terminal device 11 is replaced by the local terminal device 21, and at least a part of the input and output of the user interface 111 is replaced by the input and output of the user interface 231-n of any of the peripheral devices 23-n (FIG. 11). For example, at least a part of the operation input to the user interface 111 may be replaced by the operation input to the user interface 231-n of any of the peripheral devices 23-n (step S101), or at least a part of the output from the user interface 111 may be replaced by the output from the user interface 231-n of any of the peripheral devices 23-n (step S105). In the former case, the user 100 performs an operation input for remote operation on the user interface 231-n of any of the peripheral devices 23-n (FIG. 11: the first terminal device). The user interface 231-n sends the content of the operation input to the input / output control unit 232-n. The input / output control unit 232-n sends the content of the operation input to the peripheral device control unit 216 of the local terminal device 21 (FIG. 2). The peripheral device control unit 216 sends the content of the operation input to the remote operation unit 113. In the latter case, the remote information control unit 114 obtains and outputs output information (UI1: the first information) for the output from the user interface 231-n based on the remote information 104 (remote information UI2: the second information). The output information (UI1: the first information) is sent to the peripheral device control unit 216, and the peripheral device control unit 216 sends the output information (UI1: the first information) to the input / output control unit 232-n of any of the peripheral devices 23-n (FIG. 11: the first terminal device). The output information (UI1: the first information) is output to the user interface 231-n. The user interface 231-n performs an output based on the output information. The user 100 perceives the output from the user interface 231-n.

[0059] <Processing for Changing Environment Configuration> The environment configuration changing unit 121a of the environment configuration device 121 of the connection destination terminal device 12 (Fig. 3) in the second embodiment changes the environment configuration of the remote operation system 2 in response to a change in at least one of the usage status, device status, and line status of at least one of the local terminal device 11 (first terminal device) and the peripheral devices 23-1, …, 23-N (first terminal device). The difference between the change process of the environment configuration in the second embodiment and the change process of the environment configuration in the first embodiment is that the remote operation system 1 is replaced by the remote operation system 2, the local terminal device 11 is replaced by the local terminal device 21, and at least a part of the input / output of the user interface 111 is replaced by the input / output of the user interface 231-n of any one of the peripheral devices 23-n (Fig. 11).

[0060] <Features of the Second Embodiment> In the second embodiment, the environment configuration of the remote operation system 2 is changed in response to a change in the status information representing at least one of the usage status, device status, and line status of at least one of the local terminal device 11 (first terminal device) and the peripheral devices 23-1, …, 23-N (first terminal device). Thereby, even when the situation regarding any of the local terminal device 11 and the peripheral devices 23-1, …, 23-N changes, high operability can be realized without manually changing the environment configuration. As a result, it is possible to achieve both operability and convenience when the situation regarding at least one of the local terminal device 11 and the peripheral devices 23-1, …, 23-N changes.

[0061] In the second embodiment as well, the same modifications as in Modifications 1 to 3 of the first embodiment are possible.

[0062] [Third Embodiment] In the first embodiment, the environment configuration device 121 was provided in the connection destination terminal device 12 (second terminal device). However, the environment configuration device may be distributed between the local terminal device (first terminal device) and the connection destination terminal device (second terminal device), and the environment configuration device of the distributed local terminal device and the environment configuration device of the connection destination terminal device may cooperate to realize the functions of the entire environment configuration device.

[0063] As illustrated in FIG. 12, the remote operation system 3 of this embodiment includes a local terminal device 31 (first terminal device) and a connection destination terminal device 32 (second terminal device). The remote operation system 3 may further include peripheral devices 23-1,..., 23-N (first terminal devices). The local terminal device 31 and the connection destination terminal device 32 are configured to be communicable through a network 10. The peripheral devices 23-1,..., 23-N are connected to the local terminal device 31 by wire or wirelessly. The difference between the local terminal device 31 of the third embodiment and the local terminal devices 11 of the first embodiment and 21 of the second embodiment is that the local terminal device 31 has an environment configuration device 311. The difference between the connection destination terminal device 32 of the third embodiment and the connection destination terminal devices 12 of the first and second embodiments is that the environment configuration device 121 is replaced by an environment configuration device 321. That is, in this embodiment, the environment configuration device 121 of the first embodiment is distributed between the environment configuration device 311 and the environment configuration device 321. For example, the environment configuration device 321 (FIG. 12) includes an environment configuration change unit 121a (FIG. 3) but does not include an environment configuration application unit 121b, and the environment configuration device 311 (FIG. 12) includes an environment configuration application unit 121b (FIG. 3) but does not include an environment configuration change unit 121a. The processing of steps S103-S105, the processing of steps S14 and S15 (FIGS. 8 and 10) in this case is replaced by, for example, the processing of steps S103'-S105' and the processing of steps S14' and S15' below.

[0064] <Step S103'> The connection destination terminal device 32 (FIG. 12) receives the remote operation information 103, extracts the remote information corresponding to the operation input from the storage unit 123, and transmits the extracted remote information to the local terminal device 31 via the network 10.

[0065] <Step S104'> The local terminal device 31 receives the transmitted remote information and sends it to the environment configuration application unit 121b of the environment configuration device 311. Information representing the current environment configuration is further sent to the environment configuration application unit 121b. The environment configuration application unit 121b of the environment configuration device 311 changes or maintains the transmitted remote information, and generates and outputs remote information 104 (output information UI1 (first information)) corresponding to the current environment configuration.

[0066] <Step S105’> The remote information 104 (output information UI1 (first information)) is output to the user interface 111 of the local terminal device 31, and the user interface 111 performs an output based on the remote information 104. Here, the remote information 104 may be output to the user interface 231-n of any peripheral device 23-n (FIG. 11), and the user interface 231-n may perform an output based on the remote information 104. The user 100 perceives the output from the user interface 111 or the user interface 231-n.

[0067] <Step S14’> The destination terminal device 32 receives the status information (for example, the usage record 101 and the local terminal record 102), and sends it to the environment configuration change unit 121a of the environment configuration device 321. The environment configuration change unit 121a changes the environment configuration of the remote operation system 3 according to the change in the status information. The destination terminal device 32 transmits the information representing the changed environment configuration to the local terminal device 31 via the network 10. The information representing the changed environment configuration is received by the local terminal device 31 and sent to the environment configuration application unit 121b of the environment configuration device 311.

[0068] <Step S15’> In addition to the information representing the changed environment configuration, the remote information sent from the connection destination terminal device 32 is input to the environment configuration application unit 121b of the environment configuration device 311. The environment configuration application unit 121b of the environment configuration device 311 changes the sent remote information and generates and outputs remote information 104 (output information UI1 (first information)) corresponding to the current environment configuration of the remote operation system 3. That is, the environment configuration application unit 121b changes at least a part of the remote information 104 (output information UI1 (first information)) according to the change in the environment configuration (changes from the past remote information 104). For example, first, the environment configuration application unit 121b of the environment configuration device 311 changes (sets) the environment configuration of the local terminal device 31 (for example, hardware settings, software settings, etc.) to the changed environment configuration sent from the connection destination terminal device 32. For example, next, the environment configuration application unit 121b of the environment configuration device 311 changes the sent remote information according to the changed environment configuration and generates and outputs remote information 104 (output information UI1 (first information)). The remote information 104 is output to the user interface 111, and the user interface 111 performs an output based on the remote information 104. Here, the remote information 104 may be output to the user interface 231-n of any peripheral device 23-n (FIG. 11), and the user interface 231-n may perform an output based on the remote information 104. At this time, the environment configuration application unit 121b of the environment configuration device 311 may change (set) the environment configuration of this peripheral device 23-n (for example, hardware settings, software settings, etc.) to the changed environment configuration sent from the connection destination terminal device 32, and the user interface 231-n of this peripheral device 23-n may perform an output based on the remote information 104.

[0069] [Modification Example 1 of the Fourth Embodiment] In the fourth embodiment, the environment configuration devices 311 and 321 are distributed between the local terminal device 31 (the first terminal device) and the connection destination terminal device 32 (the second terminal device), and the environment configuration device 311 of the distributed local terminal device 31 and the environment configuration device 321 of the connection destination terminal device 32 cooperate to realize the functions of the entire environment configuration devices 311 and 321. On the other hand, the environment configuration devices may be further distributed to peripheral devices, and the distributed environment configuration devices may cooperate to realize the functions of the entire environment configuration devices.

[0070] As illustrated in FIG. 13, the remote operation system 3' of this modification includes a local terminal device 31 (the first terminal device), a connection destination terminal device 32 (the second terminal device), and peripheral devices 33-1,..., 33-N (the first terminal device). The local terminal device 31 and the connection destination terminal device 32 are configured to be communicable through the network 10. The peripheral devices 33-1,..., 33-N are connected to the local terminal device 31 by wire or wirelessly. The difference between the peripheral device 23-n of the second and third embodiments and the peripheral device 33-n of this modification is that the peripheral device 33-n has an environment configuration device 331-n. That is, in this embodiment, the environment configuration device 121 of the first embodiment is distributed to the environment configuration devices 311, 321, and peripheral devices 33-1,..., 33-N. For example, the environment configuration device 321 (FIG. 13) includes the environment configuration change unit 121a (FIG. 3) but does not include the environment configuration application unit 121b, and the environment configuration devices 311 and 331-1,..., 331-N (FIG. 13) include the environment configuration application unit 121b (FIG. 3) but do not include the environment configuration change unit 121a. In this case, in this modification, a part of the processing performed by the environment configuration device 311 of the local terminal device 31 in the third embodiment may be executed by the environment configuration device 331-n of any one of the peripheral devices 33-n. That is, remote information and information representing the current environment configuration may be sent to the environment configuration application unit 121b of the environment configuration device 331-n of any one of the peripheral devices 33-n, and this environment configuration application unit 121b may change or maintain the sent remote information and generate and output remote information 104 (output information UI1 (the first information)) corresponding to the current environment configuration.

[0071] [Hardware Configuration] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry including a general-purpose processor, a specific-purpose processor, an integrated circuit, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the described functions. The processor includes transistors and other circuits and is regarded as circuitry or processing circuitry. The processor may be a programmed processor that executes a program stored in a memory.

[0072] In this specification, circuitry, unit, and means are hardware programmed to realize the described functions or hardware that executes. The hardware may be any hardware disclosed in this specification or any hardware known as being programmed or executing to realize the described functions.

[0073] When the hardware is a processor regarded as being of the circuitry type, the circuitry, means, or unit is a combination of hardware and software used to configure the hardware and / or the processor.

[0074] The program describing this processing content can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any of a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, etc. may be used.

[0075] In addition, the distribution of this program can be carried out, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Further, the program may be stored in the storage device of a server computer and transferred from the server computer to other computers via a network, so as to constitute a configuration for distributing this program.

[0076] A computer that executes such a program first stores, for example, the program recorded on a portable recording medium or the program transferred from a server computer once in its own storage device. Then, at the time of executing the process, this computer reads the program stored in its own recording medium and executes the process according to the read program. As another execution form of this program, it is also possible that the computer directly reads the program from the portable recording medium and executes the process according to the program. Further, each time a program is transferred from the server computer to this computer, it is also possible to sequentially execute the process according to the received program. Also, the transfer of the program from the server computer to this computer may not be performed, and the process may be executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. Furthermore, it is also possible to use a so-called SaaS (Software as a Service) type service in which a part of the server computer is used by the user together with the program to execute the processing of the terminal. Note that the program in this embodiment includes information for use in processing by an electronic computer that conforms to the program (data having a property of defining the processing of the computer although not a direct instruction to the computer).

[0077] In addition, in this embodiment, the present apparatus is configured by causing a predetermined program to be executed on a computer, but at least a part of these processing contents may be realized hardware-wise.

[0078] Note that the present invention is not limited to the above-described embodiments. For example, in the above-described embodiments, an example is shown in which the environment configuration changing unit of the environment configuration device is arranged in at least one of the local terminal device (first terminal device), the connected destination terminal device (second terminal device), or the peripheral device (first terminal device). However, the environment configuration changing unit of the environment configuration device may be arranged in a device other than the local terminal device, the connected destination terminal device, and the peripheral device. For example, the environment configuration changing unit of the environment configuration device may exist on the cloud. The above-described various processes are not only executed in time series according to the description, but may also be executed in parallel or individually according to the processing ability of the device that executes the processes or as necessary. Needless to say, other appropriate changes can be made without departing from the spirit of the present invention.

[0079] In the above-described embodiment, an example was shown in which the environmental configuration of the remote operation system is changed according to a change in the status information indicating at least any one of the usage status, device status, and line status of the local terminal device and / or peripheral devices (first terminal device). However, the environmental configuration of the remote operation system may be changed according to a change in the status information indicating at least any one of the usage status, device status, and line status of the connected terminal device (second terminal device). The usage status (usage scenario) of the connected terminal device means the situation and environment in which the connected terminal device is used. Examples of the usage status of the connected terminal device include the physical environment in which the connected terminal device is used, the social environment in which the connected terminal device is used, the purpose for which the connected terminal device is used and the functions used by the connected terminal device, the type of user of the connected terminal device, and specific situations and scenarios in which the connected terminal device is used. The environmental configuration of the remote operation system may be changed according to a change in the OS of the connected terminal device. That is, in a remote operation system including a first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device, an environmental configuration change unit that changes the environmental configuration of the remote operation system according to a change in the status information indicating at least any one of the usage status, device status, and line status of the second terminal device, and an environmental configuration application unit that changes at least a part of the first information and the second information according to the change in the environmental configuration may be provided.

[0080] In the above-described embodiments, at least a part of the first information for output from the user interface and the second information for obtaining the first information was changed according to the change in the environmental configuration. However, at least a part of the handling of the input to the user interface (processing for the input to the user interface) may be changed according to the change in the environmental configuration. For example, according to a change in at least a part of the hardware and software constituting the user interface or a change in the usage situation (change in the environmental configuration), the handling of the information input to the user interface may be changed. For example, when the user interface is changed from a personal computer terminal device to a smartphone terminal device or when the usage situation or the like changes, a virtual keyboard or input means specialized for a specific device, work, situation, etc. according to the changed environmental configuration is provided. The handling of the information input to the virtual keyboard or input means may be changed. For example, when the work performed by the user or the terminal device or interface to be operated is changed, the input language of the user interface (how to handle the input as a language) or the input key arrangement (how to handle the input as a key arrangement) may be changed so that input can be performed with a simpler operation than the standard keyboard input. For example, according to whether or not an AI operation is performed, it may be changed whether or not the input of the user interface is handled as an input of a special key corresponding to the AI operation. For example, according to a change in application operations including cursor operations and remote connection / disconnection, the setting of the tactile operation area or the like may be changed so that the input of the user interface becomes a toolbar, scroll, click, drag operation, etc. corresponding to the application operation. In addition, according to the situation of the user, the handling of the input of the user interface may be customized so that the input is suitable for the situation of the user. Further, according to a change in the situation information indicating at least any one of the usage situation, device situation, and line situation of the connected terminal device, the handling of the input of the user interface of the local terminal device and / or peripheral devices may be changed.For example, in response to a change in the OS of the destination terminal device, the input of the user interface of the local terminal device and / or the peripheral device may be treated as a key array corresponding to the OS after the change, or may be treated as a special key corresponding to a specific OS. That is, in a remote operation system including a first terminal device that outputs first information for output from the user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device, an environment configuration change unit that changes the environment configuration of the remote operation system in response to a change in the situation information indicating at least any one of the usage status, device status, and line status of the first terminal device and / or the second terminal device, and an environment configuration application unit that changes at least a part of the handling of the input to the user interface in response to the change in the environment configuration may be provided.

Industrial Applicability

[0081] By applying the present invention, for example, the environment configuration of the remote operation system can be automatically changed in response to a change in the usage status of the local terminal device. Specific examples are shown below. · When the user is using the remote operation system for drafting work, increase the resolution of the image output from the user interface of the local terminal device, and when using the remote operation system for form creation work, lower the resolution of the image output from the user interface of the local terminal device. · Set a high security level during the movement of the local terminal device, apply a privacy filter to the confidential information output from the user interface of the local terminal device, or require a click operation to display the confidential information. When returning to the company, cancel these settings (return the settings to the default state). · When the user is using the remote operation system to perform a presentation using a projector, display the presentation screen full screen on the projector, which is a peripheral device, and display a mouse cursor as a substitute for the laser pointer. After the presentation is finished, cancel this setting. · When the hand-held terminal device is placed on the desk, the image output from the user interface of the hand-held terminal device is set to be suitable for an external display. When the hand-held terminal device is placed in a conference room, the setting of the image output from the user interface of the hand-held terminal device is changed to a setting suitable for a smartphone screen.

[0082] By applying the present invention, for example, the environment configuration of the remote operation system can be automatically changed according to changes in the device status or line status of the hand-held terminal device. Specific examples are shown below. · When the hand-held terminal device is connected to a headset which is a peripheral device, the volume output from the headset is changed to a volume suitable for a Web conference. Otherwise, this setting is cancelled. · When the network between the hand-held terminal device and the connected destination terminal device is a mobile line, the communication volume is reduced by lowering the resolution or number of colors of the image output from the user interface of the hand-held terminal device, or making the background invisible. When this network is a Wi-Fi line, the resolution or number of colors of the image output from the user interface of the hand-held terminal device is increased, or the background is displayed. · When the power supply means of the hand-held terminal device is external power supply, the resolution, number of colors, etc. of the image output from the user interface of the hand-held terminal device are increased to improve the image quality. When the power supply means of the hand-held terminal device is an internal battery, the resolution, number of colors, etc. of the image output from the user interface of the hand-held terminal device are decreased to suppress power consumption.

Explanation of reference numerals

[0083] 1, 2, 3, 3’ Remote operation system 11, 21, 31 Hand-held terminal device 12, 32 Connected destination terminal device 23-n, 33-n Peripheral device 121, 311, 321, 331-n Environment configuration device 111, 231-n User interface

Claims

1. A first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device. In a remote operation system including: an environment configuration changing unit that changes the environment configuration of the remote operation system in response to a change in situation information indicating at least any one of the usage status, device status, and line status of the first terminal device; An environment configuration applying unit that changes at least a part of the first information and the second information in response to the change in the environment configuration. An environment configuration device having the above.

2. The environment configuration device according to Claim 1, wherein the situation information is a combination of information indicating the usage status of the first terminal device and at least any one of information indicating the device status and information indicating the line status of the first terminal device.

3. The environment configuration device according to Claim 1, wherein the environment configuration changing unit does not change the environment configuration when the change in the situation information does not exceed a reference, and changes the environment configuration when the change in the situation information exceeds the reference.

4. An environment configuration method of an environment configuration device, In a remote operation system including a first terminal device that outputs first information for output from a user interface to the user interface, and a second terminal device that is remotely operated by the first terminal device and transmits second information for obtaining the first information to the first terminal device, an environment configuration changing step of changing the environment configuration of the remote operation system in response to a change in situation information indicating at least any one of the usage status, device status, and line status of the first terminal device; An environment configuration applying step of changing at least a part of the first information and the second information in response to the change in the environment configuration. An environment configuration method having the above.

5. A program for causing a computer to function as the environment configuration device according to any one of Claims 1 to 3.

Images