Linked display system and linked display method

Abstract

Provided is a linked display method to be executed by establishing a communication connection between a first information device and a second information device, the linked display method comprising: a step of activating a linked operation between the first information device and the second information device; a step of disabling a power-saving mode of the first information device and switching screen display of a first display, which is included in the first information device, such that the screen display is always on; a step of transmitting, to the second information device, second split data containing a part of data displayed on the first display; a step of displaying, on the first display, first split data obtained after extracting the second split data from the display data; a step of receiving the second split data by the second information device; and a step of displaying the second split data on a second display included in the second information device.

Description

Linkage display system and linkage display method

[0001] The present invention relates to an linked display system and a linked display method, and more particularly to a linked display technology using a head mounted display (HMD) and a mobile terminal.

[0002] As a technology for displaying content that mixes text and images on separate devices, Patent Document 1 discloses a display system that "has an image display device that displays moving images from the content, a text display device that displays text and still images from the content, and a control device that controls the display of the image display device and the text display device, wherein the control device manages the content data by dividing it into a moving image portion, a text portion, and a still image portion, and causes the image display device to display moving images based on the data for the moving image portion, and causes the text display device to display text and still images based on the data for the text portion and the data for the still image portion (abstract excerpt)."

[0003] Japanese Patent Application Laid-Open No. 2007-334237

[0004] Even if the above display system splits what was originally one piece of data into multiple pieces and displays them on separate devices, it does not mean that users only need to look at one of them. For example, there are requests to view one piece of data while also viewing the other piece of data as needed, such as a list of materials and work procedures, an image (still image, video) and explanatory text, or an image and an enlarged image.

[0005] However, when a mobile device is not operated for a certain period of time, the device generally goes into a power saving mode, the display goes blank, and the screen goes dark to reduce battery consumption.

[0006] Therefore, when a mobile terminal is used in the display system described in Patent Document 1, there is a risk that the screen display of the mobile terminal may disappear while the user is looking at the screen of the display device linked to the mobile terminal. In such a case, the user must touch the screen or enter a password each time, which is inconvenient to use.

[0007] The present invention has been made in consideration of the above points, and aims to improve the usability of a linked display system and linked display method that links multiple information devices and displays data in divided form.

[0008] In order to solve the above-mentioned problems, the present invention has the configurations described in the claims. One example thereof is an linked display system in which a first information device and a second information device are communicatively connected, the first information device comprising a first display, a first communication device for transmitting and receiving data to and from the second information device, and a first processor for controlling the operation of the first display and the first communication device, the second information device comprising a second display, a second communication device for transmitting and receiving data to and from the first information device, and a second processor for controlling the operation of the second display and the second communication device, the first information device being configured to operate in a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a second communication mode in which the first information device is configured to operate in a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a power saving mode that turns off the screen display of the first display when no operation is performed on the first display for a predetermined period of time or longer; when a linked operation between the first information device and the second information device is started, the power saving mode of the first information device is disabled and the screen display of the first display is switched to always on; second split data consisting of a portion of the display data to be displayed on the first display is transmitted to the second information device; the first split data obtained by extracting the second split data from the display data is displayed on the first display; and when the second information device receives the second split data, it displays it on the second display.

[0009] According to the present invention, it is possible to improve the usability of a linked display system and a linked display method that link multiple information devices and display data in a divided manner. Objects, configurations, and effects other than those described above will be made clear in the following embodiments.

[0010] 1 is a system schematic diagram of an linked display system of an information device and a head-mounted display according to an embodiment of the present invention, where (a) shows the state before linkage and (b) shows the state after linkage. It is a front view of the exterior of a mobile terminal according to an embodiment of the present invention. It is a schematic configuration diagram of a mobile terminal according to an embodiment of the present invention. It is an exterior view of an HMD according to an embodiment of the present invention. It is a schematic configuration diagram of an HMD according to an embodiment of the present invention. It is a functional block diagram of a control program and a linked display app installed in a mobile terminal. It is a functional block diagram of a control program and a linked display app installed in an HMD. It is a diagram showing an example 1 of a data display screen by the linked display system, where (a) shows the screen of the mobile terminal before linkage, (b-1) shows the screen of the HMD after linkage, and (b-2) shows the screen of the mobile terminal after linkage. It is an explanatory diagram of the operation modes of a mobile terminal. It is a diagram comparing power consumption in each operation mode. It is a diagram showing the relationship between operation and power consumption of a mobile terminal in the first embodiment. It is a control flow diagram (first half) of the linked display system according to the first embodiment. It is a control flow diagram (second half) of the linked display system according to the first embodiment. It is a diagram showing the relationship between operation and power consumption of a mobile terminal in a second embodiment. FIG. 1 is a control flow diagram (first half) of an integrated display system according to a second embodiment. FIG. 2 is a control flow diagram (middle half) of an integrated display system according to a second embodiment. FIG. 3 is a control flow diagram (second half) of an integrated display system according to a second embodiment. FIG. 4 is a diagram showing a user's line of sight when the display is turned on by a trigger from the integrated display system, where (a) shows the field of view through the HMD display when the user is not looking directly at the mobile device screen, and (b) shows the field of view through the HMD display when the user is looking directly at the mobile device screen. FIG. 5 is a control flow diagram of an integrated display system according to a third embodiment. FIG. 6 is a diagram showing the relationship between the operation and power consumption of a mobile device in a fourth embodiment. FIG. 7 is a control flow diagram (first half) of an integrated display system according to a fourth embodiment. FIG. 8 is a control flow diagram (second half) of an integrated display system according to a fourth embodiment. FIG. 9 is a diagram showing an example of a data display screen according to a fifth embodiment, where (a) shows the screen of a mobile device before linkage, (b-1) shows the screen of an HMD after linkage, and (b-2) shows the screen of a mobile device after linkage. FIG. 10 is a control flow diagram showing the flow of processing according to a fifth embodiment.13A and 13B are diagrams showing examples of data display screens according to a sixth embodiment, in which (a) shows the screen of the mobile device before linkage, (b-1) shows the screen of the HMD after linkage, and (b-2) shows the screen of the mobile device after linkage. 13B are control flow diagrams showing the flow of processing according to the sixth embodiment. 13C are diagrams showing examples of viewpoint operation when a video is displayed on a mobile device.

[0011] The present invention is expected to contribute to diversification and technological improvement in labor-intensive industries, for example, and therefore contribute to Goal 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (increasing economic productivity through diversification, technological improvement, and innovation, particularly in labor-intensive industries and industries that add value to goods and services).

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same components are designated by the same reference numerals throughout the drawings, and redundant explanations will be omitted.

[0013] FIG. 1 is a system schematic diagram of a linked display system of a mobile terminal 2 and an HMD (head mounted display) 3 according to this embodiment, where (a) shows the system before linkage and (b) shows the system after linkage.

[0014] 1(a), before the linkage, user 1 operates only mobile terminal 2 (corresponding to the first information device). Mobile terminal 2 is connected for communication with web server 6 via network 5. Mobile terminal 2 receives data from web server 6 via first data acquisition path 41 and displays the data on mobile terminal 2.

[0015] After linking as shown in FIG. 1( b ), user 1 wears HMD 3 (corresponding to the second information device) and operates portable terminal 2. Portable terminal 2 is communicatively connected to web server 6 via network 5. Portable terminal 2 receives display data (content) from web server 6 via first data acquisition path 41. Portable terminal 2 is communicatively connected (linked) with HMD 3 and divides the display data acquired from web server 6. The divided data to be displayed on HMD 3 among the parts of the divided data is referred to as second divided data, and the data obtained by extracting the second divided data from the display data is referred to as first divided data. Portable terminal 2 then transfers the second divided data to HMD 3 via second data transfer path 42 during linking.

[0016] As another example of the data acquisition path, the HMD 3 may be communicatively connected to the web server 6, the web server 6 may divide the data to create third data, the portable terminal 2 may receive the data from the web server 6 via the first data acquisition path 41, and the HMD 3 may receive the data from the web server 6 via the third data acquisition path 43. In this case, the HMD 3 and the portable terminal 2 may also be communicatively connected via the second data transfer path 42.

[0017] (Information Device) An example of a mobile terminal will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a front view of the exterior of the mobile terminal 2 according to this embodiment, and Fig. 3 is a schematic diagram of the mobile terminal 2 according to this embodiment.

[0018] As shown in Fig. 2, the surface of the mobile terminal 2 is provided with a display 207 (corresponding to a "first display") on which a home button 230, an earpiece 231, a notification lamp 232, an in-camera 206, a distance measurement sensor 213, and a touch panel 208 are stacked. In addition, although not shown in Fig. 2, the mobile terminal 2 has a volume button and a power button on its side, a speaker 218 (see Fig. 3), a microphone 217 (see Fig. 3), a power and communication connector, and an earphone jack on its bottom, and an out-camera 205 (see Fig. 3), a fingerprint sensor, etc. on its back. Note that the locations of these components do not have to be as shown in Fig. 2.

[0019] As shown in FIG. 3, the mobile terminal 2 includes a processor 201, a RAM 202, a ROM 203, a storage 204, an outer camera 205, an inner camera 206, a display 207, a touch panel 208, an acceleration sensor 209, a gyro sensor 210, a geomagnetic sensor 211, a positioning sensor 212, a ranging sensor 213, a telephone network communication I / F 214, a LAN communication I / F 215, a BT communication I / F 216 (BT: Bluetooth (registered trademark)), a microphone 217, a speaker 218, a key input I / F 219, and a battery 220, which are connected to each other via a bus 221.

[0020] The ROM 203 includes a control program 251 that controls the basic operations of the mobile terminal 2 and an association display application 252 (installed on the mobile terminal 2), and these programs are loaded into the RAM 202 and executed by the processor 201. The association display application 252 may be stored in the storage 204.

[0021] The storage 204 can store data generated in the process and as a result of executing a processing program.

[0022] The telephone network communication I / F 214 is a type of communication device that realizes communication functions for mobile communications such as 4G and 5G.

[0023] The distance measurement sensor 213 may be a sensor that calculates distance to feature points such as the contours of a real object using a method such as a stereo camera, or a sensor that measures distance by irradiating light rays two-dimensionally such as a TOF (Time of Flight) method, as long as it can measure the distance to the mobile terminal 2.

[0024] The LAN communication I / F 215 is a type of communication device that realizes a communication function for wireless communication using the Wi-Fi (registered trademark) standard, etc. Note that other wireless communication means may also be used.

[0025] The BT communication I / F 216 is a type of communication device used for linking the portable terminal 2 and the HMD 3. Linking between the portable terminal 2 and the HMD 3 is not limited to BT, and other wireless communication means may be used.

[0026] (HMD) An example of the HMD 3 will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is an external view of the HMD 3 according to this embodiment, and Fig. 5 is a schematic configuration diagram of the HMD 3 according to this embodiment.

[0027] 4 , the HMD 3 mainly includes a controller 331, an outer camera 305, a distance measurement sensor 313, a display 307, a positioning sensor 312, a microphone 317, a speaker 318, and operation keys 319a mounted on a frame 330, and the frame 330 is worn on the head of the user 1. When worn, the display 307 and distance measurement sensor 313 are located in front of the eyes of the user 1.

[0028] The outer camera 305 is mounted to capture an image in front of the user 1's line of sight, and the distance measurement sensor 313 measures the distance between the image captured by the outer camera 305 (corresponding to the "forward image") and a real object in real space.

[0029] The components shown as a processor 301, a RAM 302, a ROM 303, and a storage 304 (see FIG. 5) are incorporated into the controller 331. The locations of these components do not have to be the same as those shown in FIG.

[0030] As shown in FIG. 5, the HMD 3 includes a processor 301, a RAM 302, a ROM 303, a storage 304, an outer camera 305, an inner camera 306, a display 307, an acceleration sensor 309, a gyro sensor 310, a geomagnetic sensor 311, a positioning sensor 312, a ranging sensor 313, a LAN communication I / F 315, a BT communication I / F 316, a microphone 317, a speaker 318, a key input I / F 319, and a battery 320, which are connected to each other via a bus 321.

[0031] FIG. 6 is a functional block diagram of the control program 251 and the linked display application 252 installed on the mobile terminal 2. As shown in FIG.

[0032] The control program 251 includes a first communication control unit 251a and a first display control unit 251b.

[0033] The first communication control unit 251 a communicates with the HMD 3 and the web server 6 .

[0034] The first display control unit 251b controls the display on the display 207 during the normal mode.

[0035] The link display application 252 includes a first link display unit 252a, a first operation acceptance unit 252b, and an operation mode control unit 252d.

[0036] The first link display unit 252a controls the display of the display 207 during link operation, and also resizes images.

[0037] First operation acceptance unit 252b accepts operations performed by touching touch panel 208. First operation acceptance unit 252b also includes first image recognition unit 252c. First image recognition unit 252c performs image recognition processing on the image captured by in-camera 206, and determines whether user 1 is looking at portable terminal 2 from the front.

[0038] The operation mode control unit 252d switches the operation among a normal mode in which all functions are on, a power saving mode, a linked power saving mode (screen always on), and a linked power saving mode (trigger return). The details of each mode will be described later.

[0039] The above-mentioned units are configured by the processor 201 loading a control program 251 and an associated display application 252 into the RAM 202 and executing them.

[0040] FIG. 7 is a functional block diagram of the control program 351 and the linked display application 352 installed in the HMD 3.

[0041] The control program 351 includes a second communication control unit 351a and a second display control unit 351b.

[0042] The second communication control unit 351 a communicates with the mobile terminal 2 and the web server 6 .

[0043] The second display control unit 351b controls the display on the display 307 when not linked with the portable terminal 2.

[0044] The link display application 352 includes a second link display unit 352a and a second operation receiving unit 352b.

[0045] The second link display unit 352a controls the display of the display 307 during link operation, and also resizes images.

[0046] The second operation receiving unit 352b includes a second image recognition unit 352c and a gaze detection unit (gaze detection sensor) 352d. The second image recognition unit 352c performs image recognition processing on the image captured by the outer camera 305 and recognizes the portable terminal 2.

[0047] The gaze detection unit 352d detects the area in which the eyes of the user 1 are captured from the image captured by the in-camera 306 and performs gaze tracking. The second operation acceptance unit 352b can determine whether the user 1 is looking at the portable device 2 by superimposing the gaze tracking result on the captured image of the portable device 2 recognized by the second image recognition unit 352c. Furthermore, the second operation acceptance unit 352b may determine, as a result of the gaze tracking, where the gaze point (viewpoint) of the user 1 is located within the angle of view of the out-camera 305, and accept operations by associating actions including the position and movement of the gaze point with operations for linked display. The in-camera 306 and the gaze detection unit 352d constitute a gaze sensor.

[0048] The above-mentioned units are configured by the processor 301 loading a control program 351 and an associated display application 352 into the RAM 302 and executing them.

[0049] 8 shows an example 1 of a data display screen by the linked display system, where (a) shows the screen of the mobile terminal 2 before linkage, (b-1) shows the screen of the HMD 3 after linkage, and (b-2) shows the screen of the mobile terminal 2 after linkage. Fig. 8 is an example of displaying data consisting only of text.

[0050] The data may be divided in advance when the data is created, and the user 1 may specify data to be transmitted to the HMD 3 by touching text on the touch panel 208. Alternatively, an area may be specified on the touch panel 208.

[0051] After the mobile terminal 2 and the HMD 3 are linked, it is preferable to enlarge (optimize) the data display size to match the screen size of each.

[0052] It is also preferable that the data displayed on the HMD 3 can be changed by controlling the line of sight of the HMD 3, for example, by blinking twice, the display data of the HMD 3 and the portable terminal 2 can be swapped.

[0053] 8 shows an example of a screen display of data consisting of only text, but part of the data may be an image. In this case, the image may be transmitted to the HMD 3, or the image may be displayed on the mobile terminal 2 and the text may be displayed on the HMD 3. The type of data and the type of divided data are not important.

[0054] FIG. 9 is an explanatory diagram of the operation modes of the mobile terminal.

[0055] In this embodiment, the operating states of the mobile terminal 2 include normal mode, power saving mode, linked power saving mode (screen always on) which is executed only during linked operation (linked operation), and linked power saving mode (trigger return).

[0056] The normal mode is a mode in which all functions are on. When the mobile device 2 is in use and no linked display is being displayed, the mobile device operates in the normal mode. If no operation is performed for a predetermined period of time in the normal mode, the mobile device switches to the power saving mode.

[0057] The power saving mode is a mode in which, for example, only wireless communication (telephone network communication I / F 214) is on and other functions are off, and is a mode to which the mobile terminal 2 transitions if no operation is performed for a predetermined period of time. The power saving mode is the operating mode with the lowest power consumption. Therefore, the screen display of the display 207 is also turned off. To turn on the screen display from the power saving mode, it is necessary to touch the touch panel 208 or operate the home button 230. Furthermore, if a passcode has been set, the user is required to enter the passcode.

[0058] During linkage, the portable terminal 2 may operate in a normal mode with the power saving mode disabled, or may operate in the power saving mode or the linked power saving mode.

[0059] The linked power saving mode includes a linked power saving mode (always screen on) and a linked power saving mode (trigger return). In the linked power saving mode (always screen on), only the minimum functions necessary for linked display with the HMD 3 are turned on, and other functions are turned off, thereby operating with less power consumption than in the normal mode. Specifically, the telephone network communication I / F 214, the device-to-device connection with the mobile terminal 2 via BT connection, and the screen display are turned on, while other functions are turned off. Voice recognition for voice operation may also be turned on. In the linked power saving mode (trigger return), like the power saving mode, the screen display is turned off when no operation is performed. In this case, trigger detection is required to turn on the screen display. By turning on the trigger detection function and detecting a trigger operation, for example, by the mobile terminal 2 or the HMD 3, the screen display can be switched on and off.

[0060] FIG. 10 is a diagram showing a comparison of power consumption in each operation mode.

[0061] Using normal mode as the standard, power consumption is lower in linked power saving mode (always on screen), and even lower in linked power saving mode (trigger recovery). Using linked power saving mode (trigger recovery) as the standard, power consumption is lower in power saving mode.

[0062] The control flow of the linked display system of each embodiment will be described below, but the required operation mode differs depending on the embodiment.

[0063] First Embodiment In the first embodiment, while a mobile terminal and an HMD are linked, the power saving mode is disabled, the screen of the mobile terminal is always on, and the mobile terminal operates in normal mode.

[0064] FIG. 11 is a diagram showing the relationship between the operation and power consumption of the mobile terminal in the first embodiment.

[0065] 11 , in the first embodiment, the linked operation between the portable terminal 2 and the HMD 3 is started at time t1 while the portable terminal 2 is in normal operation, that is, when the device connection starts and the portable terminal 2 receives an affirmative response to the HMD linked operation start signal, the power saving mode is switched from enabled to disabled. As a result, the power saving mode is disabled during the linked operation, and the screen of the portable terminal 2 remains on even if a certain period of time has passed without any operation.

[0066] When the linked operation ends at time t3, the power saving mode switches from disabled to enabled and normal operation resumes. Therefore, although power consumption is high because the power saving mode is disabled during linked operation, work efficiency does not decrease because the screen can be viewed at any time.

[0067] Fig. 12 is a control flow diagram (first half) of the linked display system according to the first embodiment. Fig. 13 is a control flow diagram (second half) of the linked display system according to the first embodiment.

[0068] As shown in FIG. 12, the mobile terminal 2 receives data from the web server 6 and displays it on the display 207 (S101).

[0069] When the linked display application 252 is started on the mobile terminal 2 (S102), the mobile terminal 2 transmits an HMD linked operation start signal to the HMD 3 (S103).

[0070] When pairing between the HMD 3 and the portable terminal 2 is completed and the HMD 3 receives a linkage operation start signal, the HMD 3 starts an HMD linkage operation (S104) and transmits an acknowledgment (ACK packet) to the portable terminal 2 (S105).

[0071] The first operation accepting unit 252b accepts a data division operation via the touch panel 208. The first link display unit 252a divides the data in accordance with the data division operation, selects the data in the divided area as second divided data (S106), and transmits it to the HMD 3 (S107).

[0072] When the second communication control unit 351a of the HMD 3 completes reception of the second divided data (S108), it transmits an acknowledgment to the portable terminal 2 (S109).

[0073] The second link display unit 352a of the HMD 3 resizes the second divided data to fit the size of the display 307 (S110) and displays the second divided data on the display 307 (S111).

[0074] On the other hand, when the mobile terminal 2 receives the affirmative response in step S109, the first link display unit 252a resizes the first divided data to fit the size of the display 207 (S112) and displays the first divided data on the display 207 (S113).

[0075] The operation mode control unit 252d of the portable terminal 2 disables the power saving mode (S114). As a result, the screen of the display 207 is always on, regardless of whether the portable terminal 2 is operated or not, until the linked display of the HMD 3 is released.

[0076] As shown in FIG. 13, the second linked display unit 352a of the HMD 3 continues the linked display until the HMD linked operation ends (S115: No).

[0077] On the other hand, when an instruction to end the HMD linked operation is input to the HMD 3 by gesture input or gaze input (S115: Yes), the HMD 3 transmits a linked operation end signal to the portable terminal 2 (S116).

[0078] When the mobile device 2 receives the linkage end signal, it ends the linkage (S117). The first linkage display unit 252a returns an affirmative response (S118), erases the first divided data that was being displayed (S119), and the first display control unit 251b redisplays the original data on the display 207 (S120). The operation mode control unit 252d enables the power saving mode (S121).

[0079] When the HMD 3 receives the positive response in step S118, it erases the display of the second divided data on the display 307 (S122).

[0080] According to the first embodiment, when linked display starts between the mobile device 2 and the HMD 3, the power saving mode of the mobile device 2 is disabled and the mobile device 2 always operates in normal mode. As a result, the screen of the mobile device 2 is always on during linkage regardless of whether the mobile device 2 is operated or not, so that even if the user 1 looks at the screen of the mobile device 2 at any timing during linkage, the first split data displayed on the screen can be viewed without operating the mobile device 2.

[0081] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed while working on the task at hand with almost no eye movement, which is expected to improve work efficiency.

[0082] Second Embodiment In the second embodiment, the power saving mode remains enabled while the portable terminal 2 and the HMD 3 are linked, and the screen display of the portable terminal 2 is turned on and the portable terminal 2 operates in normal mode upon detection of a trigger by the HMD 3. Generally, the portable terminal 2 switches to the power saving mode if no operation is performed for a predetermined period of time while the portable terminal 2 is running in normal mode. However, in the second embodiment and a third embodiment described later, activation of a linked operation also causes a transition from normal mode to linked power saving mode (trigger return).

[0083] The processing of the linked display system according to the second embodiment will be described below with reference to Figs. 14 to 17. Fig. 14 is a diagram showing the relationship between the operation of a mobile terminal and power consumption in the second embodiment. Fig. 15 is a control flow diagram (first half) of the linked display system according to the second embodiment. Fig. 16 is a control flow diagram (middle) of the linked display system according to the second embodiment. Fig. 17 is a control flow diagram (second half) of the linked display system according to the second embodiment.

[0084] 14 , in the second embodiment, a linked operation between the portable terminal 2 and the HMD 3 is initiated at time t1 while the portable terminal 2 is in normal operation, i.e., when the device connection starts and the portable terminal 2 receives an affirmative response to the HMD linked operation start signal, the portable terminal 2 transitions to linked power saving mode (trigger return) and turns off the screen display of the portable terminal 2. The activation of the linked operation and the transition to linked power saving mode (trigger return) do not have to occur simultaneously.

[0085] When the HMD 3 detects an operation, for example, a gesture operation, at time t2, the detection of the operation triggers the screen display of the portable terminal 2 to be switched from off to on. If no operation is performed for a certain period of time, the screen display is then turned off. This switching of the screen display from on to off is the same as the standard operation in power saving mode.

[0086] When the linked operation ends at time t3, the linked power saving mode (trigger return) is enabled and the device switches to normal mode, performing normal operation. Therefore, the linked power saving mode (trigger return) is enabled by the linked operation and the screen display is turned off, shortening the time the screen display is on and reducing power consumption, while the screen display is turned on using the detection of an operation as a trigger, so work efficiency does not decrease.

[0087] 15, steps S101 to S112 are the same as those in the first embodiment, and therefore will not be described again. The operation mode control unit 252d of the portable terminal 2 transitions to the linked power saving mode (trigger return) (S210).

[0088] As shown in FIG. 16, when the second link display unit 352a determines that the user 1 is looking at the portable terminal 2 (S201: Yes), the second link display unit 352a transmits a screen display on signal to the portable terminal 2 (S202).

[0089] On the other hand, when the second link display unit 352a determines that the user 1 is not looking at the portable terminal 2 (S201: No), the second link display unit 352a transmits a screen display off signal to the portable terminal 2 (S203).

[0090] When the mobile terminal 2 receives the screen display on signal (S204: on signal), the first link display unit 252a turns on the screen display of the display 207 (S205) and displays the first divided data (S206).

[0091] On the other hand, when the mobile terminal 2 receives the screen display off signal (S204: off signal), the first link display unit 252a turns off the screen display of the display 207 (S207).

[0092] The portable terminal 2 returns to step S204 and repeats the process until it receives an HMD linked operation end signal from the HMD 3 (S208: No).

[0093] The HMD 3 returns to step S201 and repeats the process until an instruction to end the HMD-linked operation is input by gesture input or gaze input (S115: No).

[0094] When an instruction to end the HMD linked operation is input to the HMD 3 by a gesture input or a gaze input (S115: Yes), the HMD 3 transmits a linked operation end signal to the portable terminal 2 (S116).

[0095] When the portable terminal 2 receives the linkage operation end signal (S208: Yes), it ends the HMD linkage operation (S117), and the first linkage display unit 252a returns an affirmative response (S118).

[0096] 17 , when the linked operation is completed, the portable terminal 2 turns on the screen display of the display 207 (S209), erases the first divided data that was displayed (remaining on the display 207) (S119), and the first display control unit 251b redisplays the original data on the display 207 (S120). Then, the operation mode control unit 252d cancels the linked power saving mode (trigger return) (S211). This causes the portable terminal 2 to transition to the normal mode.

[0097] When the HMD 3 receives the positive response in step S118, it erases the display of the second divided data on the display 307 (S122).

[0098] Figure 18 is a diagram showing the user's line of sight when the display is turned on by a trigger from the linked display system, where (a) shows the field of view through the display 307 of the HMD 3 when the user 1 is not looking directly at the screen of the mobile terminal 2, and (b) shows the field of view through the display 307 of the HMD 3 when the user 1 is looking directly at the screen of the mobile terminal 2.

[0099] In FIG. 18A, the mobile terminal 2 is visible in the left corner of the user 1's field of vision, but the user 1 is not looking directly at the mobile terminal 2, so no screen display-on signal (trigger signal) is output.

[0100] In FIG. 18(b), the display 207 of the portable terminal 2 is displayed directly in the center of the user 1's field of vision, so it is determined that the user is looking at the portable terminal 2 from the front, and a screen display on signal (trigger signal) is output.

[0101] 18(a) or 18(b) is determined by the second image recognition unit 352c performing image recognition processing on the image captured by the outer camera 305. The second image recognition unit 352c recognizes the type of subject and recognizes whether the portable terminal 2 is captured (corresponding to recognition of the image of the first information device), and further determines whether the image captured by the portable terminal 2 is directly facing the HMD 3 near the center of the image captured by the outer camera 305.

[0102] According to the second embodiment, the screen display of the portable terminal 2 and the HMD 3 is turned on only when the user 1 looks at the screen of the portable terminal 2 while the portable terminal 2 and the HMD 3 are linked, so the user 1 can view the first divided data without operating the portable terminal 2. In addition, the screen display is turned off when the user 1 is not looking at the screen of the portable terminal 2, so power consumption during linkage can be reduced compared to the first embodiment in which the screen display is always on.

[0103] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed while working on the task at hand with almost no eye movement, which is expected to improve work efficiency.

[0104] <Third Embodiment> In the third embodiment, the power saving mode remains enabled while the portable device 2 and the HMD 3 are linked, and the screen display of the portable device 2 is turned on and the device operates in normal mode upon detection of a trigger on the portable device 2. Generally, the portable device 2 switches to power saving mode if no operation is performed for a predetermined period of time while the device is running in normal mode. However, in the second and third embodiments, the device transitions from normal mode to linked power saving mode (trigger return) upon activation of a linked operation. The third embodiment differs from the second embodiment in that trigger detection is performed by the HMD 3 in the second embodiment, but is performed by the portable device 2 in the third embodiment.

[0105] The processing of the linked display system according to the third embodiment will be described with reference to Fig. 19. Fig. 19 is a control flow diagram of the linked display system according to the third embodiment.

[0106] Steps S101 to S112 and step S210 (see FIG. 15) are the same as those in the second embodiment, and therefore redundant explanations will be omitted.

[0107] 19 , the first image recognition unit 252c of the portable terminal 2 performs image recognition processing on the image captured by the in-camera 206 to recognize the face of the user 1. Then, when the first linked display unit 252a determines that the user 1 is looking at the portable terminal 2 from the front based on the size of the facial image of the user 1 and the orientation of the face (S301: Yes), it turns on the screen display of the display 207 (S204) and displays the first divided data (S205).

[0108] On the other hand, when the first link display unit 252a determines that the user 1 is not looking directly at the mobile terminal 2 (S301: No), it turns off the screen display of the display 207 (S207).

[0109] The first linked display unit 252a may determine whether the user 1 is looking at the mobile terminal 2 from the front, for example, if the first image recognition unit 252c can recognize both eyes from the image of the in-camera 206, it may determine that the user 1 is looking at the mobile terminal 2 from the front.

[0110] If the first link display unit 252a does not receive the HMD link operation end signal (S302: No), the process returns to step S301.

[0111] When the portable terminal 2 receives the HMD linked operation end signal (S302: Yes), the same processing as in the second embodiment is executed, and therefore a duplicated description will be omitted.

[0112] According to the third embodiment, the mobile terminal 2 and the HMD 3 are linked and the screen display is turned on only when the user 1 looks at the screen of the mobile terminal 2, so that the user 1 can view the first split data without operating the mobile terminal 2.

[0113] Furthermore, since the HMD 3 is worn on the head, it is required to be lightweight, making it difficult to install a large-capacity battery. According to the third embodiment, the trigger is detected by the portable terminal 2, so the HMD 3 does not need to detect the trigger for the process of switching the screen display on and off of the portable terminal 2 and transmit a screen display on signal or a screen display off signal to the portable terminal 2, and the power consumption of the HMD 3 can be reduced compared to the second embodiment.

[0114] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed while working on the task at hand with almost no eye movement, which is expected to improve work efficiency.

[0115] Fourth Embodiment In the fourth embodiment, the screen display of the portable terminal 2 is always on while the portable terminal 2 and the HMD 3 are linked, and the portable terminal 2 operates in a linked power saving mode (screen always on).

[0116] The processing of the linked display system according to the fourth embodiment will be described below with reference to Fig. 20 to Fig. 22. Fig. 20 is a diagram showing the relationship between the operation and power consumption of a mobile terminal according to the fourth embodiment. Fig. 21 is a control flow diagram (first half) of the linked display system according to the fourth embodiment. Fig. 22 is a control flow diagram (second half) of the linked display system according to the fourth embodiment.

[0117] 20 , in the fourth embodiment, the linked operation between the portable terminal 2 and the HMD 3 is started at time t1 while the portable terminal 2 is in normal operation. In other words, when the device connection starts and the portable terminal 2 receives an affirmative response to the HMD linked operation start signal, the portable terminal 2 enters the linked operation mode (constant screen on) and then transitions to the linked power saving mode. At this time, the screen display of the portable terminal 2 is always on.

[0118] When the linked operation ends at time t3, the device switches to normal mode and continues normal operation. Therefore, the linked operation switches to a linked power-saving mode (screen always on) that consumes less power than normal mode, saving power, while keeping the screen display always on, so work efficiency does not decrease.

[0119] As shown in FIG. 21, steps S101 to S113 are the same as those in the first embodiment, and therefore a duplicated description will be omitted.

[0120] The operation mode control unit 252d of the portable terminal 2 transitions to the linked power saving mode (screen always on) (S401).

[0121] As shown in FIG. 22, the second linked display unit 352a of the HMD 3 continues the linked display until the HMD linked operation ends (S115: No).

[0122] On the other hand, when an instruction to end the HMD linked operation is input to the HMD 3 by gesture input or gaze input (S115: Yes), the HMD 3 transmits a linked operation end signal to the portable terminal 2 (S116).

[0123] When the mobile device 2 receives the linkage end signal, it ends the linkage (S117). The first linkage display unit 252a returns an affirmative response (S118), erases the first divided data that was being displayed (S119), and the first display control unit 251b redisplays the original data on the display 207 (S120). The operation mode control unit 252d then cancels the linkage power-saving mode (constant screen on) (S402). This causes the mobile device 2 to transition to normal mode.

[0124] In the linked power saving mode (screen always on), the mobile terminal 2 can display the screen, perform voice search, wireless telephone, and BT communication, but, for example, applications, data synchronization, Wi-Fi, and location information acquisition are stopped, so power consumption can be reduced more than in normal mode.

[0125] Furthermore, the screen of the portable terminal 2 is always on regardless of whether the portable terminal 2 is operated or not, so even if the user 1 looks at the screen of the portable terminal 2 at any timing during linking, the first divided data displayed on the screen can be viewed without operating the portable terminal 2.

[0126] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed while working on the task at hand with almost no eye movement, which is expected to improve work efficiency.

[0127] Fifth Embodiment The fifth embodiment is an embodiment in which data consisting of only images is displayed in cooperation between the portable terminal 2 and the HMD 3 .

[0128] Figure 23 shows an example of a data display screen according to the fifth embodiment, where (a) shows the screen of the mobile terminal 2 before linkage, (b-1) shows the screen of the HMD 3 after linkage, and (b-2) shows the screen of the mobile terminal 2 after linkage.

[0129] Before linking, multiple thumbnail images are displayed on the screen of the portable terminal 2 (see FIG. 23(a)). Data to be transmitted to the HMD 3 is specified by selecting an image on the touch panel 208. Alternatively, the display order may be set in advance when the data is created, and the first image may be transmitted to the HMD 3.

[0130] After the linking, the data display size is enlarged (optimized) to fit the screen size on the HMD 3 and displayed (see FIG. 23(b-1)). Also, on the screen of the portable terminal 2, a mark indicating that the image has been selected is superimposed on the thumbnail image corresponding to the selected image (see FIG. 23(b-2)).

[0131] During linkage, the data displayed on the HMD 3 can be changed by controlling the line of sight of the HMD 3. For example, the data can be changed to the next data by blinking twice.

[0132] FIG. 24 is a control flow diagram showing the flow of processing according to the fifth embodiment.

[0133] After establishing a link connection in steps S101 to S105, image data A is selected from the thumbnail images displayed on the screen of the portable terminal 2 (S501) and transmitted to the HMD 3 (S502).

[0134] The HMD 3 receives the image data A (S503) and returns an affirmative response (S504).

[0135] The HMD 3 resizes the image data A (S505) and displays it on the display 307 (S506).

[0136] Upon receiving the positive response in step S504, the portable terminal 2 superimposes a frame on the image data A (S507), displays the image data A with the frame superimposed on the display 207 (S508), and disables the power saving mode (S114). The subsequent processing is the same as that shown in FIG. 13, so a repeated explanation will be omitted.

[0137] According to this embodiment, the portable terminal 2 and the HMD 3 are linked, and one image can be selected from multiple thumbnail images displayed on the portable terminal 2 and enlarged and displayed on the HMD 3, making it easier to view the image data A that you want to focus on.

[0138] Sixth Embodiment In the sixth embodiment, one image and image data consisting of a part of the image are displayed in cooperation between the portable terminal 2 and the HMD 3 .

[0139] Figure 25 is a diagram showing an example of a data display screen related to the sixth embodiment, where (a) shows the screen of the mobile terminal 2 before linkage, (b-1) shows the screen of the HMD 3 after linkage, and (b-2) shows the screen of the mobile terminal 2 after linkage.

[0140] Before linking, the HMD transmission data is specified by specifying the area to be enlarged on the touch panel 208. Alternatively, an enlargeable area is set in advance when creating the data, and the HMD transmission data is specified by the user 1 touching the area. Furthermore, the area may be specified by the line of sight through the HMD 3.

[0141] After the linking, a mark is superimposed on the enlarged portion of the original image on the display 207 of the mobile terminal 2 (see FIG. 25(b-2)).

[0142] In the HMD 3, the data display size is enlarged (optimized) to fit the screen size (see FIG. 25(b-1)).

[0143] The data displayed on the HMD 3 may be changed by operating the line of sight of the HMD 3. For example, two blinks may be used to change to the next data, change the enlargement position, change the magnification, or the like.

[0144] FIG. 26 is a control flow diagram showing the flow of processing according to the sixth embodiment.

[0145] After establishing a link connection in steps S101 to S105, an area to be enlarged in the image displayed on the screen of the portable terminal 2 is designated, and enlarged image data is designated (S601), and transmitted to the HMD 3 (S602).

[0146] The HMD 3 receives the enlarged image data (S603) and returns an affirmative response (S604).

[0147] The HMD 3 resizes the enlarged image data (S605) and displays it on the display 307 (S606).

[0148] Upon receiving the positive response in step S604, the portable terminal 2 superimposes a frame on the enlarged portion of the image data (S607), displays the image data with the superimposed frame on the display 207 (S608), and disables the power saving mode (S609). The subsequent processing is the same as that shown in FIG. 13, so a repeated explanation will be omitted.

[0149] According to this embodiment, the portable terminal 2 and the HMD 3 are linked, and a portion of the image displayed on the portable terminal 2 can be specified and enlarged and displayed on the HMD 3, making it easier to view the portion you want to focus on.

[0150] Seventh Embodiment The seventh embodiment is an embodiment relating to a variation of the operation example of the HMD 3. The operation example according to this embodiment may be used as appropriate in the first to sixth embodiments.

[0151] The operation of the HMD 3 is not limited to gaze detection, and may be performed by, for example, voice input, gesture input, tapping on a frame, or the like.

[0152] The number of divisions of the data being displayed on the portable terminal 2 is not limited to two. The data may be divided into three or more parts, and the divided data may be transmitted to the HMD 3 and displayed in parallel on the display 307. The divided data may also be displayed sequentially on the display 307.

[0153] Furthermore, when data is displayed on the screen of the portable terminal 2 and the user 1 looks at the image of the portable terminal 2 through the HMD 3, the data being displayed on the HMD 3 may be moved near the screen of the portable terminal 2.

[0154] Furthermore, if the data displayed on the screen of the mobile terminal 2 includes a "video", the video may be operated by eye gaze operation.

[0155] FIG. 27 is a diagram showing an example of viewpoint manipulation when a video is displayed on the mobile terminal 2.

[0156] A video may be displayed on the portable terminal 2, and text or the like may be displayed on the HMD 3, and the video may be controlled based on the position and time of the gaze point. The action of continuously looking at one point for a predetermined period of time while keeping the line of sight fixed on the portable terminal 2 may be used as a video playback operation.

[0157] Furthermore, while viewing the portable terminal 2 through the HMD 3 and keeping the line of sight fixed on the portable terminal 2, the action of moving the line of sight to the left may be regarded as a rewind operation, and the action of moving the line of sight to the right may be regarded as a fast-forward operation.

[0158] Furthermore, the video stopping operation may be an operation of continuing to look at a point other than the portable terminal 2 for a predetermined period of time through the HMD 3. The HMD 3 may be configured to customize video operations corresponding to user operations.

[0159] When the processor 301 of the HMD 3 detects an operation corresponding to the movement of the line of sight, it generates an instruction signal indicating the content of the operation and transmits it to the portable terminal 2. When the portable terminal 2 receives the instruction signal, it operates the video display on the display 207 of the portable terminal 2 in accordance with the operation content indicated by the instruction signal, and the text display of the HMD 3 also transitions in conjunction with this.

[0160] According to this embodiment, when the mobile terminal 2 and the HMD 3 are linked, the video being displayed on the HMD 3 can be played, fast-forwarded, rewound, and stopped by inputting the viewpoint, so that even when the user 1 is busy working and their hands are full, they can perform operations related to watching the video without operating the mobile terminal 2 with their hands.

[0161] The invention made by the inventor has been specifically described above based on the embodiments thereof, but it goes without saying that the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the invention.

[0162] For example, the first information device may be a tablet terminal instead of a smartphone, and the second information device may be smart glasses instead of a head-mounted display.

[0163] Furthermore, the trigger event for resuming the screen display of the portable device 2 is not limited to the recognition of a facial image of the user 1 from an image captured by the in-camera 206 of the portable device 2, or the recognition of an image of the portable device from an image captured by the out-camera 305 of the HMD 3 and the user's line of sight overlapping the image. For example, when a link destination is displayed on the display 307 of the HMD 3, blinking twice with the gaze point overlapping the link destination may cause the linked page to be displayed on the display 207 of the portable device 2. Furthermore, voice recognition of an utterance by the user 1, such as "screen on," may also be used as a trigger event.

[0164] Furthermore, the related data transmitted from the portable terminal 2 to the HMD 3 is not limited to an image corresponding to a thumbnail image, divided data of data being displayed on the portable terminal 2, or text in a video. For example, when a sentence in a first language is displayed on the display 207 of the portable terminal 2, the sentence translated into a second language may be displayed on the HMD 3 as related data.

[0165] For example, any combination of the above-described embodiments is included in the present invention. Also, modifications to add further functions to the above-described embodiments are possible.

[0166] Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0167] The programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each process is performed may also be changed.

[0168] The functions of the present invention described above may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Alternatively, they may be realized by software, such as by a processor unit interpreting and executing an operating program that realizes each function. Hardware and software may also be used in combination.

[0169] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected.

[0170] The embodiments include the following: (First Aspect) A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls operations of the first display and the first communication device, and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls operations of the second display and the second communication device, the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a power saving mode in which the screen display of the first display is turned off when a predetermined period of time has passed since the first information device was last operated; when a linked operation between the first information device and the second information device is initiated, the power saving mode of the first information device is disabled and the screen display of the first display is switched to always on; second split data consisting of a portion of the display data to be displayed on the first display is transmitted to the second information device; the first split data obtained by extracting the second split data from the display data is displayed on the first display; and the second information device receives the second split data and displays it on the second display.(Form of the second aspect) A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls operations of the first display and the first communication device, and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls operations of the second display and the second communication device, the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a first linked power saving mode in which the screen display of the first display is turned off when a predetermined period of time has elapsed since the first information device was last operated; when linked operation between the first information device and the second information device is started, the first linked power saving mode of the first information device is enabled; when a trigger signal is input to turn on the screen display of the first display while the screen display of the first display is off, the first information device transmits second split data consisting of a portion of the display data to be displayed on the first display to the second information device; the first split data obtained by extracting the second split data from the display data is displayed on the first display; when a trigger signal is input to turn on the screen display of the first display while the screen display of the first display is off, the second information device turns on the screen display of the first display; and when the second information device receives the second split data, it displays it on the second display.(Embodiment of the Third Aspect) A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls operations of the first display and the first communication device, and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls operations of the second display and the second communication device, the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a second linked power-saving mode in which the first display is always on and other specified functions are turned off to operate with less power than in the normal mode, and when linked operation between the first information device and the second information device is activated, the first information device switches to the second linked power-saving mode, transmits second divided data consisting of a part of the display data to be displayed on the first display to the second information device, and displays the first divided data after the second divided data is extracted from the display data on the first display, and when the second information device receives the second divided data, it displays it on the second display. (Form of the fourth aspect) A linked display method executed by connecting a first information device and a second information device for communication, the linked display method including: a step of starting a linked operation between the first information device and the second information device; a step of disabling a power saving mode of the first information device and switching the screen display of a first display provided in the first information device to be always on; a step of transmitting second split data consisting of a part of display data to be displayed on the first display to the second information device; a step of displaying the first split data on the first display after extracting the second split data from the display data; a step of the second information device receiving the second split data; and a step of displaying the second split data on a second display provided in the second information device.

[0171] 1: User 2: Portable terminal 3: HMD 5: Network 6: Web server 41: First data acquisition path 42: Second data transfer path 43: Third data acquisition path 201: Processor 202: RAM 203: ROM 204: Storage 205: Outer camera 206: Inner camera 207: Display 208: Touch panel 209: Acceleration sensor 210: Gyro sensor 211: Geomagnetic sensor 212: Positioning sensor 213: Distance measurement sensor 214: Telephone network communication I / F 215: LAN communication I / F 216: Bluetooth communication I / F 217: Microphone 218: Speaker 219: Key input I / F 220: Battery 221: Bus 230: Home button 231: Earpiece 232: Notification lamp 251: Control program 251a: First communication control unit 251b: First display control unit 252: Linked display application 252a: First linked display unit 252b: First operation acceptance unit 252c: First image recognition unit 252d: Operation mode control unit 301: Processor 302: RAM 303: ROM 304: Storage 305: Outer camera 306: Inner camera 307: Display 309: Acceleration sensor 310: Gyro sensor 311: Geomagnetic sensor 312: Positioning sensor 313: Distance measurement sensor 315: LAN communication I / F 316: BT communication I / F 317: Microphone 318 : Speaker 319 : Key input I / F 319a : Operation key 320 : Battery 321 : Bus330: Frame 331: Controller 351: Control program 351a: Second communication control unit 351b: Second display control unit 352: Linked display application 352a: Second linked display unit 352b: Second operation acceptance unit 352c: Second image recognition unit 352d: Gaze detection unit

Claims

1. A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls the operation of the first display and the first communication device; and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls the operation of the second display and the second communication device. the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a power saving mode in which the screen display of the first display is turned off when a predetermined period of time has passed since the first information device was last operated; when a linked operation between the first information device and the second information device is initiated, the power saving mode of the first information device is disabled and the screen display of the first display is switched to always on; second split data consisting of a portion of the display data to be displayed on the first display is transmitted to the second information device; the first split data obtained by extracting the second split data from the display data is displayed on the first display; and the second information device receives the second split data and displays it on the second display.

2. A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls the operation of the first display and the first communication device; and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls the operation of the second display and the second communication device. the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a first linked power saving mode in which the screen display of the first display is turned off when a predetermined period of time has passed since the first information device was last operated; when linked operation between the first information device and the second information device is started, the first linked power saving mode of the first information device is enabled, second divided data consisting of a portion of the display data to be displayed on the first display is transmitted to the second information device, the first divided data obtained by extracting the second divided data from the display data is displayed on the first display, and when a trigger signal to turn on the screen display of the first display is input while the screen display of the first display is off, the screen display of the first display is turned on; and when the second information device receives the second divided data, it displays it on the second display.

3. An integrated display system according to claim 2, wherein the second information device transmits the trigger signal to the first information device when it detects a trigger event that turns on the screen display of the first display.

4. An interconnected display system as described in claim 2, wherein the second information device is a head-mounted display or smart glasses equipped with an outer camera that captures images of real space and an eye-gaze sensor that detects the line of sight of the user of the second information device, and the second processor recognizes the image of the first information device by performing image recognition processing on the image captured by the outer camera, and transmits the trigger signal to the first information device when it determines that the line of sight overlaps with the first information device.

5. An integrated display system as described in claim 4, wherein the second processor generates an instruction signal for playing, fast-forwarding, rewinding, or stopping the video on the first display based on the gaze movement determined to be overlapping, and transmits the instruction signal to the first information device, and the first information device operates the video based on the instruction signal.

6. An linked display system according to claim 2, wherein the first information device determines that the trigger signal has been input when it detects a trigger event that turns on the screen display of the first display.

7. An linked display system as described in claim 2, wherein the first information device is a mobile terminal further equipped with an in-camera on the surface on which the first display is provided, and the first processor performs image recognition processing on an image captured by the in-camera and determines that a trigger signal has been input when it recognizes a facial image of the user of the second information device.

8. A linked display system in which a first information device and a second information device are communicatively connected, wherein the first information device comprises a first display, a first communication device that transmits and receives data to and from the second information device, and a first processor that controls the operation of the first display and the first communication device; and the second information device comprises a second display, a second communication device that transmits and receives data to and from the first information device, and a second processor that controls the operation of the second display and the second communication device. the first information device has a normal mode in which the screen display of the first display is on and device-to-device communication with the second information device is maintained, and a second linked power-saving mode in which the first display is always on and other specified functions are turned off to operate with less power than in the normal mode, and when linked operation between the first information device and the second information device is started, the first information device switches to the second linked power-saving mode, transmits second divided data consisting of a part of the display data to be displayed on the first display to the second information device, displays the first divided data after extracting the second divided data from the display data on the first display, and when the second information device receives the second divided data, displays it on the second display.

9. An linked display system according to claim 1, wherein the first display displays a plurality of thumbnail images, and when the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device, and the second processor displays the received image data corresponding to the selected thumbnail image on the second display.

10. An linked display system according to claim 2, wherein the first display displays a plurality of thumbnail images, and when the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device, and the second processor displays the received image data corresponding to the selected thumbnail image on the second display.

11. An integrated display system according to claim 8, wherein the first display displays a plurality of thumbnail images, and when the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device, and the second processor displays the received image data corresponding to the selected thumbnail image on the second display.

12. An integrated display system as described in claim 1, wherein, when the first processor receives a selection of a partial area of ​​an image displayed on the first display, it transmits divided data within the selected area to the second information device, and the second processor displays the received divided data on the second display.

13. An integrated display system as described in claim 2, wherein, when the first processor receives a selection of a partial area of ​​an image displayed on the first display, it transmits divided data within the selected area to the second information device, and the second processor displays the received divided data on the second display.

14. An integrated display system as described in claim 8, wherein, when the first processor receives a selection of a partial area of ​​an image displayed on the first display, it transmits divided data within the selected area to the second information device, and the second processor displays the received divided data on the second display.

15. A linked display method executed by connecting a first information device and a second information device for communication, comprising: a step of starting linked operation between the first information device and the second information device; a step of disabling the power saving mode of the first information device and switching the screen display of the first display provided in the first information device to always on; a step of transmitting second divided data consisting of a part of the display data to be displayed on the first display to the second information device; a step of displaying the first divided data on the first display after extracting the second divided data from the display data; a step of receiving the second divided data by the second information device; and a step of displaying the second divided data on the second display provided in the second information device.

Images