Collaborative display system and collaborative display method

Abstract

The cooperative display method is executed by performing communication connection on a first information device and a second information device, and includes: a step of starting a cooperative operation between the first information device and the second information device; switching the screen display of a first display of the first information device to be always on by disabling the power saving mode of the first information device; transmitting, to a second information device, second divided data including a portion of the data displayed on the first display; displaying, on a first display, first divided data obtained by cutting out the second divided data from the display data; a step in which the second information device receives the second divided data; and displaying the second divided data on a second display provided in the second information device.

Description

Technical Field

[0001] The present invention relates to a collaborative display system and a collaborative display method, and in particular to a collaborative display technology using a head mounted display (HMD: Head Mount Display) and a portable terminal. Background Art

[0002] As a technology for displaying content in which text and images are mixed on various devices, the following display system is disclosed in Patent Document 1, "having: an image display device for displaying animation in the content; a text display device for displaying text and still images in the content; and a control device for controlling the display of the image display device and the text display device, the control device divides the data of the content into an animation part, a text part, and a still image part for management, and causes the image display device to display the animation based on the data of the animation part, and causes the text display device to display the text and the still image (abstract excerpt) based on the data of the text part and the data of the still image part."

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2007-334237 Summary of the invention

[0006] In the above display system, the data originally contained in one piece is divided into multiple pieces. Even if the data is displayed on each device, it is not necessary for the user to only see one of them. For example, when viewing the data of one piece, there is a need to see the data of the other piece as needed, such as a material list and a work process, an image (still image, animation) and an explanation text, an image and an enlarged image, etc.

[0007] However, when a portable terminal is not operated for a certain period of time, it generally shifts to a power saving mode to reduce battery consumption, and the display disappears and the screen becomes dark.

[0008] Therefore, when the portable terminal is used in the display system described in Patent Document 1, the screen display of the portable terminal may disappear while the user is viewing the screen of the display device that cooperates with the portable terminal. In such a case, the user needs to touch the screen and input a password each time, which is a fact that the usability is poor.

[0009] The present invention has been made in view of the above-mentioned points, and an object of the present invention is to improve the usability of a cooperative display system and a cooperative display method for causing a plurality of information devices to cooperate with each other to display data in a divided manner.

[0010] In order to solve the above-mentioned problems, the present invention has a structure described in the claims. When giving an example, it is a collaborative display system formed by communicating with a first information device and a second information device, characterized in that the first information device has: a first display; a first communication device that sends and receives data with the second information device; and a first processor that controls the operation of the first display and the first communication device, and the second information device has: a second display; a second communication device that sends and receives data with the first information device; and a second processor that controls the operation of the second display and the second communication device, and the first information device has a screen display of the first display that is turned on and maintains a normal mode of inter-device communication with the second information device. The invention relates to a power saving mode in which the screen display of the first display is turned off when no operation is performed on the first information device for a period of time exceeding a predetermined period. When the collaborative action between the first information device and the second information device is started, the power saving mode of the first information device is invalidated and the screen display of the first display is switched to always on. The second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after the second split data is cut out from the display data is displayed on the first display. The second information device displays the second split data on the second display when receiving it.

[0011] According to the present invention, the usability of a cooperative display system and a cooperative display method for displaying divided data by making a plurality of information devices cooperate can be improved. The objects, structures, and effects other than those described above will become more apparent in the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a system schematic diagram of a coordinated display system of an information device and a head mounted display according to the present embodiment, wherein (a) shows before the coordination and (b) shows after the coordination.

[0013] Figure 2 It is a front view of the appearance of the portable terminal involved in this embodiment.

[0014] Figure 3 It is a schematic structural diagram of the portable terminal involved in this embodiment.

[0015] Figure 4 This is an external view of the HMD according to the present embodiment.

[0016] Figure 5 It is a schematic structural diagram of the HMD involved in this embodiment.

[0017] Figure 6This is a functional block diagram of a control program and a collaborative display application installed on a portable terminal.

[0018] Figure 7 This is a functional block diagram of the control program installed in the HMD and the cooperative display application.

[0019] Figure 8 The figures show a data display screen example 1 using a collaborative display system, wherein (a) shows a screen of a portable terminal before collaboration, (b-1) shows a screen of an HMD after collaboration, and (b-2) shows a screen of a portable terminal after collaboration.

[0020] Fig. 9 This is an illustration of the operation mode of the portable terminal.

[0021] Fig.10 This is a diagram showing a comparison of power consumption in each operation mode.

[0022] Fig.11 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the first embodiment.

[0023] Fig.12 This is a control flow chart (first half) of the collaborative display system according to the first embodiment.

[0024] Fig.13 This is a control flow chart (second half) of the collaborative display system according to the first embodiment.

[0025] Fig.14 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the second embodiment.

[0026] Fig.15 This is a control flow chart (first half) of the collaborative display system according to the second embodiment.

[0027] Fig.16 This is a control flow chart of the collaborative display system according to the second embodiment (middle).

[0028] Fig.17 This is a control flow chart of the collaborative display system according to the second embodiment (second half).

[0029] Fig.18 The diagram shows the user's field of view when the display is turned on due to a trigger using a collaborative display system, (a) shows the field of view of the display through the HMD when the user is not looking at the screen of the portable terminal from the front, and (b) shows the field of view of the display through the HMD when the user is looking at the screen of the portable terminal from the front.

[0030] Fig.19 This is a control flow chart of the collaborative display system according to the third embodiment.

[0031] Fig. 20 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the fourth embodiment.

[0032] Fig.21 This is a control flow chart (first half) of the collaborative display system according to the fourth embodiment.

[0033] Fig. 22 This is a control flow chart of the collaborative display system according to the fourth embodiment (second half).

[0034] Fig.23 The figures show examples of data display screens according to the fifth embodiment, wherein (a) shows a screen of a portable terminal before collaboration, (b-1) shows a screen of an HMD after collaboration, and (b-2) shows a screen of a portable terminal after collaboration.

[0035] Fig.24 It is a control flow chart showing the flow of processing involved in the fifth embodiment.

[0036] Fig.25 The figures show examples of data display screens according to the sixth embodiment, wherein (a) shows a screen of a portable terminal before collaboration, (b-1) shows a screen of an HMD after collaboration, and (b-2) shows a screen of a portable terminal after collaboration.

[0037] Fig.26 It is a control flow chart showing the flow of processing involved in the sixth embodiment.

[0038] Fig. 27 It is a diagram showing an example of viewpoint operation when an animation is displayed on a portable terminal. DETAILED DESCRIPTION

[0039] For example, the present invention is expected to diversify and improve technology for labor-intensive industries, and therefore can be expected to contribute to 8.2 of the Sustainable Development Goals (SDGs) advocated by the United Nations (improving economic productivity through diversification, technology improvement, and innovation, focusing on industries that further increase the value of goods and services and labor-intensive industries).

[0040] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In all the drawings, the same components are denoted by the same reference numerals, and repeated descriptions are omitted.

[0041] Figure 1 1 is a system schematic diagram of a coordinated display system of a portable terminal 2 and an HMD (head mounted display) 3 according to the present embodiment, wherein (a) shows before the coordination and (b) shows after the coordination.

[0042] exist Figure 1 Before the collaboration shown in (a), the user 1 only operates the portable terminal 2 (corresponding to the first information device). The portable terminal 2 is connected to the web server 6 via the network 5. The portable terminal 2 receives data from the web server 6 via the first data acquisition path 41 and displays it on the portable terminal 2.

[0043] exist Figure 1 After the collaboration shown in (b), the user 1 wears the HMD 3 (equivalent to the second information device) and operates the portable terminal 2. The portable terminal 2 is connected to the WEB server 6 via the network 5. The portable terminal 2 receives the display data (content) from the WEB server 6 via the first data acquisition path 41. The portable terminal 2 is connected to the HMD 3 for communication (collaboration) and divides the display data obtained from the WEB server 6. The divided data displayed on the HMD 3 among the divided data is called the second divided data, and the data after the second divided data is cut out from the display data is called the first divided data. In addition, the portable terminal 2 transmits the second divided data to the HMD 3 via the second data transmission path 42 during the collaboration.

[0044] As another example of the data acquisition path, the HMD 3 may be communicatively connected to the WEB server 6, the data may be divided in the WEB server 6 to create the third data, the portable terminal 2 may receive the data from the WEB server 6 via the first data acquisition path 41, and the HMD 3 may receive the data from the WEB server 6 via the third data acquisition path 43. Even in this case, the HMD 3 and the portable terminal 2 may be communicatively connected via the second data transmission path 42.

[0045] (Information Equipment)

[0046] Reference Figure 2 as well as Figure 3 , illustrating an example of a portable terminal. Figure 2 is a front view of the appearance of the portable terminal 2 involved in this embodiment, Figure 3 It is a schematic configuration diagram of the portable terminal 2 according to the present embodiment.

[0047] like Figure 2 As shown, the surface of the portable terminal 2 is provided with a home button 230, an earpiece 231, a notification light 232, an internal camera 206, a distance sensor 213, and a display 207 (corresponding to the "first display") stacked with a touch panel 208. Figure 2 Although not shown in the figure, the portable terminal 2 has a volume button and a power button on the side, and a speaker 218 on the bottom (see Figure 3 )、Microphone 217 (refer to Figure 3 ), power supply and communication connector, headphone jack, and an external camera 205 on the back (refer to Figure 3), fingerprint sensor, etc. In addition, the configuration location may not be as follows Figure 2 shown.

[0048] like Figure 3 As shown, the portable terminal 2 includes a processor 201, a RAM 202, a ROM 203, a storage device 204, an external camera 205, an internal camera 206, a display 207, a touch panel 208, an acceleration sensor 209, a gyro sensor 210, a geomagnetic sensor 211, a positioning sensor 212, a distance sensor 213, a telephone network communication I / F 214, a LAN communication I / F 215, a BT communication I / F 216 (BT: Bluetooth (registered trademark))), a microphone 217, a speaker 218, a key input I / F 219, and a battery 220, which are interconnected via a bus 221.

[0049] The ROM 203 includes a control program 251 for controlling basic operations of the portable terminal 2 and a cooperative display application 252 (installed in the portable terminal 2 ), and the processor 201 expands and executes these programs in the RAM 202 . The cooperative display application 252 may also be stored in the storage device 204 .

[0050] The storage device 204 can store data generated during the execution of the processing program and the results thereof.

[0051] The telephone network communication I / F 214 is a type of communication device that realizes the communication function of mobile communications such as 4G and 5G.

[0052] The distance measuring sensor 213 can be a sensor that calculates the distance based on feature points such as the outline of a real object in a manner similar to a stereo camera, or a sensor that measures the distance by irradiating light two-dimensionally in a TOF (Time Of Flight) manner. Any sensor can be used to measure the distance to the portable terminal 2.

[0053] The LAN communication I / F 215 is a type of communication device that realizes a communication function using wireless communication using the Wi-Fi (registered trademark) standard, etc. In addition, other wireless communication units may also be used.

[0054] The BT communication I / F 216 is a type of communication device used in the cooperation between the portable terminal 2 and the HMD 3. The cooperation between the portable terminal 2 and the HMD 3 is not limited to BT, and other wireless communication means may be used.

[0055] (HMD)

[0056] Reference Figure 4 as well as Figure 5 , illustrating an example of HMD3. Figure 4 is an appearance diagram of HMD 3 involved in this embodiment, Figure 5 It is a schematic structural diagram of the HMD 3 involved in this embodiment.

[0057] like Figure 4 As shown, the HMD 3 mainly carries a controller 331, an external camera 305, a distance sensor 313, a display 307, a positioning sensor 312, a microphone 317, a speaker 318, and an operation key 319a on a frame 330, and the frame 330 is worn on the head of the user 1. When worn, the display 307 and the distance sensor 313 are located in front of the eyes of the user 1.

[0058] The external camera 305 is installed so as to capture the front of the user 1's line of sight, and the distance measuring sensor 313 measures the distance to a real object in the real space captured by the photographic image (equivalent to the “front image”) of the external camera 305 .

[0059] The controller 331 is embedded with a processor 301, a RAM 302, a ROM 303, and a storage device 304 (see Figure 5 ) shown in the components. In addition, the configuration location may not be as shown Figure 4 shown.

[0060] like Figure 5 As shown, HMD 3 includes a processor 301, RAM 302, ROM 303, a storage device 304, an external camera 305, an internal camera 306, a display 307, an acceleration sensor 309, a gyro sensor 310, a geomagnetic sensor 311, a positioning sensor 312, a distance sensor 313, a LAN communication I / F 315, a BT communication I / F 316, a microphone 317, a speaker 318, a key input I / F 319, and a battery 320, which are interconnected via a bus 321.

[0061] Figure 6 This is a functional block diagram of the control program 251 and the cooperative display application 252 installed in the portable terminal 2 .

[0062] The control program 251 includes a first communication control unit 251 a and a first display control unit 251 b .

[0063] The first communication control unit 251 a communicates with the HMD 3 and the WEB server 6 .

[0064] The first display control unit 251 b performs display control on the display 207 in the normal mode.

[0065] The coordinated display application 252 includes a first coordinated display unit 252a, a first operation accepting unit 252b, and an operation mode control unit 252d.

[0066] The first cooperative display unit 252a performs display control of the display 207 during the cooperative operation and also performs image size adjustment.

[0067] The first operation accepting unit 252b accepts an operation performed by a touch operation of the touch panel 208. The first operation accepting unit 252b also includes a first image recognition unit 252c. The first image recognition unit 252c performs image recognition processing on the image captured by the internal camera 206 to determine whether the user 1 is looking at the portable terminal 2 from the front.

[0068] The operation mode control unit 252d switches the operation among a normal mode in which all functions are turned on, a power saving mode, a cooperative power saving mode (screen is always on), or a cooperative power saving mode (trigger recovery). The contents of each mode will be described later.

[0069] The above-mentioned components are configured by the processor 201 loading the control program 251 and the cooperative display application 252 into the RAM 202 and executing them.

[0070] Figure 7 This is a functional block diagram of the control program 351 and the cooperative display application 352 installed in the HMD 3 .

[0071] The control program 351 includes a second communication control unit 351 a and a second display control unit 351 b .

[0072] The second communication control unit 351 a communicates with the portable terminal 2 and the WEB server 6 .

[0073] The second display control unit 351 b performs display control on the display 307 when not cooperating with the portable terminal 2 .

[0074] The coordinated display application 352 includes a second coordinated display unit 352a and a second operation accepting unit 352b.

[0075] The second cooperative display unit 352a performs display control of the display 307 during the cooperative operation and also performs image size adjustment.

[0076] The second operation accepting unit 352b includes a second image recognition unit 352c and a sight line detection unit (sight line detection sensor) 352d. The second image recognition unit 352c performs image recognition processing on the image captured by the external camera 305 to recognize the portable terminal 2.

[0077] The sight line detection unit 352d detects the area where the eyes of the user 1 are photographed from the image captured by the internal camera 306, and performs sight line tracking. The second operation receiving unit 352b can judge whether the user 1 is looking at the portable terminal 2 by overlapping the result of sight line tracking with the photographic image of the portable terminal 2 recognized by the second image recognition unit 352c. In addition, it is also possible to judge where the gaze point (viewpoint) of the user 1 is located in the field of view of the external camera 305 as a result of the sight line tracking, and accept the operation by associating the position of the gaze point, the movement of the gaze point, and the operation of the coordinated display. The sight line sensor is composed of the internal camera 306 and the sight line detection unit 352d.

[0078] The above-mentioned components are configured by the processor 301 loading the control program 351 and the cooperative display application 352 into the RAM 302 and executing them.

[0079] Figure 8 The figures show a data display screen example 1 using the collaborative display system, wherein (a) shows the screen of the portable terminal 2 before collaboration, (b-1) shows the screen of the HMD 3 after collaboration, and (b-2) shows the screen of the portable terminal 2 after collaboration. Figure 8 This is an example of displaying data consisting only of text.

[0080] The data is divided in advance when the data is created, and the user 1 may designate the data to be sent to the HMD 3 by touching text on the touch panel 208. Alternatively, the segment may be designated on the touch panel 208.

[0081] It is preferable that the portable terminal 2 and the HMD 3 cooperate with each other to enlarge (optimize) the data display size in accordance with the size of each screen.

[0082] In addition, it is preferable that the data displayed on the HMD 3 can be changed by the line of sight operation of the HMD 3, for example, the display data of the HMD 3 and the portable terminal 2 can be exchanged by blinking twice.

[0083] Figure 8 Although a screen display example of data consisting of only text is shown, a part of the data may be an image. In this case, the image may be sent to the HMD 3, or the image may be displayed on the portable terminal 2 and the text may be displayed on the HMD 3. There is no limitation on the type of data or the type of segmented data.

[0084] Fig. 9 This is an illustration of the operation mode of the portable terminal.

[0085] In the present embodiment, the operating states of the portable terminal 2 include a normal mode, a power saving mode, a cooperative power saving mode (screen always on) executed only in cooperation (cooperative operation), and a cooperative power saving mode (trigger recovery).

[0086] The normal mode is a mode in which all functions are turned on. When the portable terminal 2 is in use and no coordinated display is performed, the normal mode is operated. When no operation is performed for a predetermined time in the normal mode, the mode is switched to the power saving mode.

[0087] The power saving mode is, for example, a mode in which only wireless communication (telephone network communication I / F 214) is turned on and other functions are turned off, and is a mode to which the portable terminal 2 is transferred when no operation is performed for a predetermined time. The power saving mode is an operation mode with the lowest power consumption. Therefore, the screen display of the display 207 is also turned off. In order to turn the screen display on from the power saving mode, it is necessary to touch the touch panel 208, operate the home button 230, etc. In addition, if a password is set, the password is requested to be entered.

[0088] In cooperation, the portable terminal 2 has two modes: one in which the power saving mode is disabled and the portable terminal 2 operates in the normal mode; and one in which the portable terminal 2 operates in the power saving mode or the cooperative power saving mode.

[0089] In the collaborative power saving mode, there are collaborative power saving mode (screen is always on) and collaborative power saving mode (trigger recovery). In the collaborative power saving mode (screen is always on), only the minimum functions required for collaborative display with HMD3 are turned on, and other functions are turned off, so that it operates more power-saving than the normal mode. Specifically, the device connection and screen display with the portable terminal 2 connected by the telephone network communication I / F214 and BT are turned on, and other functions are turned off. In addition, the sound recognition for voice operation can also be turned on. The collaborative power saving mode (trigger recovery) is similar to the power saving mode, and the display of the screen is turned off when no operation is performed. In this case, in order to turn on the display of the screen, it is necessary to detect a trigger. By turning on the function of detecting the trigger, for example, detecting an operation that is triggered by the portable terminal 2 or HMD3, the display of the screen can be switched on / off.

[0090] Fig.10 This is a diagram showing a comparison of power consumption in each operation mode.

[0091] Compared with the normal mode, the power consumption in the cooperative power saving mode (screen always on) is lower, and the power consumption in the cooperative power saving mode (trigger recovery) is even lower. Compared with the cooperative power saving mode (trigger recovery), the power consumption in the power saving mode is lower.

[0092] Hereinafter, a control flow of the cooperative display system according to each embodiment will be described, but the required operation mode is different in each embodiment.

[0093] <First Embodiment>

[0094] The first embodiment is an embodiment in which the power saving mode is disabled in cooperation between the portable terminal and the HMD, and the screen of the portable terminal is always on and operates in the normal mode.

[0095] Fig.11 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the first embodiment.

[0096] like Fig.11 As shown, in the first embodiment, the portable terminal 2 starts the coordinated operation of the portable terminal 2 and the HMD 3 at time t1 in the normal operation, that is, when the devices start to be connected and the portable terminal 2 receives a positive response to the HMD coordinated operation start signal, the power saving mode is switched from valid to invalid. As a result, the power saving mode becomes invalid during the coordinated operation, so even if the screen of the portable terminal 2 is not operated for more than a certain period of time, the screen remains turned on as it is.

[0097] After the cooperative operation is completed at time t3, the power saving mode is switched from invalid to valid, and normal operation is performed. Therefore, in the cooperative operation, the power saving mode becomes invalid, so although the power consumption is high, the screen can be viewed at any time, so the work efficiency is not reduced.

[0098] Fig.12 This is a control flow chart (first half) of the collaborative display system according to the first embodiment. Fig.13 This is a control flow chart (second half) of the collaborative display system according to the first embodiment.

[0099] like Fig.12 As shown, the portable terminal 2 receives data from the WEB server 6 and displays it on the display 207 (S101).

[0100] When the cooperative display application 252 is started in the portable terminal 2 ( S102 ), an HMD cooperative operation start signal is transmitted from the portable terminal 2 to the HMD 3 ( S103 ).

[0101] When pairing between the HMD 3 and the portable terminal 2 is completed and the HMD 3 receives the collaboration start signal, the HMD collaboration is started in the HMD 3 ( S104 ), and an affirmative response (ACK packet) is sent to the portable terminal 2 ( S105 ).

[0102] The first operation accepting unit 252b accepts the data division operation via the touch panel 208. The first cooperative display unit 252a divides the data according to the data division operation, selects the data in the divided area as the second divided data (S106), and transmits it to the HMD 3 (S107).

[0103] When reception of the second divided data is completed ( S108 ), the second communication control unit 351 a of the HMD 3 transmits an affirmative response to the portable terminal 2 ( S109 ).

[0104] The second coordinated display unit 352 a of the HMD 3 adjusts the size of the second divided data to match the size of the display 307 ( S110 ), and displays the second divided data on the display 307 ( S111 ).

[0105] On the other hand, when the portable terminal 2 receives an affirmative response in step S109, the first cooperative display unit 252a adjusts the size of the first divided data to match the size of the display 207 (S112), and displays the first divided data on the display 207 (S113).

[0106] The operation mode control unit 252d of the portable terminal 2 invalidates the power saving mode (S114). Thus, until the coordinated display of the HMD 3 is canceled, the screen of the display 207 is always on regardless of whether the portable terminal 2 is operated.

[0107] like Fig.13 As shown, the second cooperative display unit 352a of the HMD 3 continues to perform the cooperative display until the HMD cooperative operation is completed (S115: No).

[0108] On the other hand, when the HMD 3 inputs an instruction to end the HMD cooperation operation by gesture input or line of sight input ( S115 : Yes), the HMD 3 transmits a cooperation end signal to the portable terminal 2 ( S116 ).

[0109] When receiving the collaborative operation end signal, the portable terminal 2 ends the collaborative operation (S117). The first collaborative display unit 252a sends back an affirmative response (S118), and the display of the first segmented data that has been displayed disappears (S119), and the first display control unit 251b displays the original data on the display 207 again (S120). The operation mode control unit 252d enables the power saving mode (S121).

[0110] When receiving the affirmative response of step S118 , the HMD 3 turns off the display of the second divided data displayed on the display 307 ( S122 ).

[0111] According to the first embodiment, when the collaborative display starts in the portable terminal 2 and the HMD 3, the power saving mode of the portable terminal 2 becomes invalid and the portable terminal 2 always operates in the normal mode. Therefore, in the collaboration, regardless of whether the portable terminal 2 is operated or not, the screen of the portable terminal 2 is always turned on, so even if the user 1 views the screen of the portable terminal 2 at any time during the collaboration, the user can view the first segmented data displayed on the screen without operating the portable terminal 2.

[0112] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed without moving the eyes while working nearby, and improvement in work efficiency can be expected.

[0113] <Second Embodiment>

[0114] The second embodiment is an embodiment in which the screen display of the portable terminal 2 is turned on and operates in the normal mode by trigger detection of the HMD 3 while the power saving mode is maintained in the collaboration of the portable terminal 2 and the HMD 3. In addition, generally, when no operation is performed for a predetermined time during the execution of the normal mode in the portable terminal 2, the power saving mode is switched to, but in the second embodiment and the third embodiment described later, the normal mode is also switched to the collaborative power saving mode (trigger recovery) by starting the collaborative action.

[0115] Below, refer to Figures 14 to 17 , describing the processing of the collaborative display system involved in the second embodiment. Fig.14 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the second embodiment. Fig.15 This is a control flow chart (first half) of the collaborative display system according to the second embodiment. Fig.16 This is a control flow chart of the collaborative display system according to the second embodiment (middle). Fig.17 This is a control flow chart of the collaborative display system according to the second embodiment (second half).

[0116] like Fig.14 As shown, in the second embodiment, the portable terminal 2 starts the collaborative operation of the portable terminal 2 and the HMD 3 at time t1 during normal operation, that is, when the device starts to connect and the portable terminal 2 receives an affirmative response to the HMD collaborative operation start signal, it shifts to the collaborative power saving mode (trigger recovery), and the screen display of the portable terminal 2 is turned off. The startup of the collaborative operation and the shift to the collaborative power saving mode (trigger recovery) may not be simultaneous.

[0117] At time t2, when the HMD 3 detects an operation based on a gesture, for example, the screen display of the portable terminal 2 is switched from off to on, triggered by the detection of the operation. After that, if no operation is performed for a certain period of time, the screen display is turned off. The switching of the screen display from on to off is the same as the standard operation of the power saving mode.

[0118] After the coordinated action ends at time t3, the coordinated power saving mode (trigger recovery) is enabled and the mode is switched to the normal mode to perform the normal action. Therefore, the coordinated power saving mode (trigger recovery) becomes enabled and the screen display is turned off through the coordinated action, so the time the screen display is turned on is shortened and the power consumption is suppressed. At the same time, the screen display is turned on by detecting the operation as a trigger, so the work efficiency is not reduced.

[0119] like Fig.15 As shown, steps S101 to S112 are the same as those in the first embodiment, and thus repeated descriptions are omitted. The operation mode control unit 252d of the portable terminal 2 shifts to the cooperative power saving mode (triggering recovery) (S210).

[0120] like Fig.16 As shown, when the second collaborative display unit 352a determines that the user 1 is viewing the portable terminal 2 (S201: Yes), it transmits a screen display on signal to the portable terminal 2 (S202).

[0121] On the other hand, when the second collaborative display unit 352a determines that the user 1 is not viewing the portable terminal 2 (S201: No), it transmits a screen display off signal to the portable terminal 2 (S203).

[0122] When the portable terminal 2 receives the screen display on signal (S204: on signal), the first cooperative display unit 252a turns on the screen display of the display 207 (S205), and displays the first divided data (S206).

[0123] On the other hand, when the portable terminal 2 receives the screen display off signal (S204: off signal), the first cooperative display unit 252a turns off the screen display of the display 207 (S207).

[0124] The process returns to step S204 and repeats the process until the portable terminal 2 receives the HMD cooperative operation end signal from the HMD 3 ( S208 : No).

[0125] The process returns to step S201 and repeats the process until the HMD 3 receives an instruction to end the HMD cooperative operation through a gesture input or a line of sight input ( S115 : No).

[0126] When the HMD 3 receives an instruction to end the HMD cooperation through a gesture input or a line of sight input ( S115 : Yes), the HMD 3 transmits a cooperation end signal to the portable terminal 2 ( S116 ).

[0127] When the mobile terminal 2 receives the cooperation end signal (S208: Yes), the HMD cooperation ends (S117), and the first cooperation display unit 252a sends back an affirmative response (S118).

[0128] like Fig.17 As shown in FIG. 1 , after the cooperative operation is completed, the portable terminal 2 turns on the screen display of the display 207 (S209), makes the display of the first segmented data displayed (remaining on the display 207) disappear (S119), and the first display control unit 251b displays the original data on the display 207 again (S120). In addition, the operation mode control unit 252d cancels the cooperative power saving mode (triggering recovery) (S211). As a result, the portable terminal 2 transitions to the normal mode.

[0129] When receiving the affirmative response of step S118 , the HMD 3 turns off the display of the second divided data displayed on the display 307 ( S122 ).

[0130] Fig.18 The diagram shows the user's line of sight when the display is turned on due to a trigger using the collaborative display system, (a) shows the field of view of the display 307 across the HMD 3 when the user 1 is not looking at the screen of the portable terminal 2 from the front, and (b) shows the field of view of the display 307 across the HMD 3 when the user 1 is looking at the screen of the portable terminal 2 from the front.

[0131] exist Fig.18 In (a), although the portable terminal 2 appears in the left corner of the visual field of the user 1, since the portable terminal 2 is not being viewed from the front, the screen display start signal (trigger signal) is not output.

[0132] exist Fig.18 In (b), the display 207 of the portable terminal 2 is facing the center of the visual field of the user 1, so it is determined that the portable terminal 2 is being viewed from the front, and a screen display start signal (trigger signal) is output.

[0133] About Fig.18 Which state (a) or (b) is selected is realized by the second image recognition unit 352c performing image recognition processing on the image captured by the external camera 305. The type of the subject is recognized to determine whether it is reflected in the portable terminal 2 (equivalent to the recognition of the first information device image), and then it is determined whether the image captured by the portable terminal 2 is facing the HMD 3 near the center of the image captured by the external camera 305.

[0134] According to the second embodiment, in the collaboration between the portable terminal 2 and the HMD 3, the screen display is turned on only when the user 1 is viewing the screen of the portable terminal 2, so the user 1 can view the first divided data without operating the portable terminal 2. In addition, when the user 1 is not viewing the screen of the portable terminal 2, the screen display is turned off, so compared with the first embodiment in which it is always turned on, the power consumption in the collaboration can be reduced.

[0135] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed without moving the eyes while working nearby, and improvement in work efficiency can be expected.

[0136] <Third Embodiment>

[0137] The third embodiment is an embodiment in which the screen display of the portable terminal 2 is turned on and the portable terminal 2 operates in the normal mode by detecting a trigger of the portable terminal 2 while the power saving mode is maintained in the collaboration between the portable terminal 2 and the HMD 3. In addition, generally, when no operation is performed for a predetermined time during the execution of the normal mode in the portable terminal 2, the portable terminal 2 switches to the power saving mode, but in the second embodiment and the third embodiment, the collaborative action is started to migrate from the normal mode to the collaborative power saving mode (trigger recovery). In addition, the difference from the second embodiment is that in the second embodiment, the HMD 3 detects the trigger, but in the third embodiment, the portable terminal 2 detects the trigger.

[0138] Reference Fig.19 , describing the processing of the collaborative display system involved in the third embodiment. Fig.19 This is a control flow chart of the collaborative display system according to the third embodiment.

[0139] Steps S101 to S112, and step S210 (see Fig.15 ) is the same as the second embodiment, so repeated description is omitted.

[0140] exist Fig.19 In the example, the first image recognition unit 252c of the portable terminal 2 performs image recognition processing on the image captured by the internal camera 206 to recognize the face of the user 1. Furthermore, when the first cooperative display unit 252a determines that the user 1 is looking at the portable terminal 2 from the front based on the size of the face image of the user 1 and the face orientation (S301: Yes), it turns on the screen display of the display 207 (S204) and displays the first segmented data (S205).

[0141] On the other hand, when it is determined that the user 1 is not looking at the portable terminal 2 from the front ( S301 : No), the first cooperative display unit 252 a turns off the screen display of the display 207 ( S207 ).

[0142] For example, when the first image recognition unit 252c can recognize both eyes based on the image of the internal camera 206, the first cooperative display unit 252a may determine whether the user 1 is looking at the portable terminal 2 from the front as the user 1 is looking at the portable terminal 2 from the front.

[0143] When the first collaborative display unit 252a has not received the HMD collaborative operation end signal (S302: No), the process returns to step S301.

[0144] When the portable terminal 2 receives the HMD cooperative operation end signal ( S302 : Yes), the same processing as that of the second embodiment is executed, and thus repeated description is omitted.

[0145] According to the third embodiment, in the cooperation between the portable terminal 2 and the HMD 3 , the screen display is turned on only when the user 1 views the screen of the portable terminal 2 . Therefore, the user 1 can view the first divided data without operating the portable terminal 2 .

[0146] In addition, HMD 3 is required to be lightweight because it is worn on the head, and it is difficult to carry a large-capacity battery. According to the third embodiment, the trigger is detected by the portable terminal 2, so HMD 3 does not need to detect the trigger of the switching process of the screen display on and off of the portable terminal 2 and send the screen display on signal and the screen display off signal to the portable terminal 2, so the power consumption of HMD 3 can be reduced compared with the second embodiment.

[0147] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed without moving the eyes while working nearby, and improvement in work efficiency can be expected.

[0148] <Fourth Implementation Method>

[0149] The fourth embodiment is an embodiment in which the screen display of the portable terminal 2 is always on in the cooperation between the portable terminal 2 and the HMD 3, and the portable terminal 2 operates in a cooperative power saving mode (screen always on).

[0150] Below, refer to Figure 20 to Figure 22 , describing the processing of the collaborative display system involved in the fourth embodiment. Fig. 20 This is a diagram showing the relationship between the operation and power consumption of the portable terminal in the fourth embodiment. Fig.21 This is a control flow chart (first half) of the collaborative display system according to the fourth embodiment. Fig. 22 This is a control flow chart (second half) of the collaborative display system according to the fourth embodiment.

[0151] like Fig. 20As shown, in the fourth embodiment, the portable terminal 2 starts the collaborative operation of the portable terminal 2 and the HMD 3 at time t1 in the normal operation, that is, when the device starts to connect and the portable terminal 2 receives a positive response to the HMD collaborative operation start signal, it shifts to the collaborative operation mode (screen is always on) and migrates to the collaborative power saving mode. At this time, the screen display of the portable terminal 2 becomes always on.

[0152] After the cooperative operation is completed at time t3, the system switches to the normal mode and performs normal operation. Therefore, the system saves power by switching to the cooperative power saving mode (screen always on) which consumes less power than the normal mode, and the screen display is always on, so the operating efficiency is not reduced.

[0153] like Fig.21 As shown, steps S101 to S113 are the same as those in the first embodiment, and thus repeated descriptions are omitted.

[0154] The operation mode control unit 252d of the portable terminal 2 shifts to the cooperative power saving mode (screen is always on) (S401).

[0155] like Fig. 22 As shown, the second cooperative display unit 352a of the HMD 3 continues to perform the cooperative display until the HMD cooperative operation is completed (S115: No).

[0156] On the other hand, when the HMD 3 receives an instruction to end the HMD cooperation through a gesture input or a line of sight input ( S115 : Yes), the HMD 3 transmits a cooperation end signal to the portable terminal 2 ( S116 ).

[0157] When the portable terminal 2 receives the collaborative operation end signal, it ends the collaborative operation (S117). The first collaborative display unit 252a sends back an affirmative response (S118), and the display of the displayed first segmented data disappears (S119), and the first display control unit 251b displays the original data on the display 207 again (S120). In addition, the operation mode control unit 252d cancels the collaborative power saving mode (the screen is always on) (S402). As a result, the portable terminal 2 shifts to the normal mode.

[0158] In the collaborative power saving mode (screen always on), the portable terminal 2 can perform screen display, voice search, wireless phone, and BT communication, but since synchronization of applications and data, Wi-Fi, acquisition of location information, etc. are stopped, power consumption can be suppressed compared to the normal mode.

[0159] Furthermore, the screen of the portable terminal 2 is always open regardless of whether the portable terminal 2 is operated. Therefore, even if the user 1 views the screen of the portable terminal 2 at any timing during collaboration, the user 1 can view the first divided data displayed on the screen without operating the portable terminal 2 .

[0160] Furthermore, since the second divided data is displayed on the HMD 3, the second divided data can be viewed without moving the eyes while working nearby, and improvement in work efficiency can be expected.

[0161] <Fifth Implementation Method>

[0162] The fifth embodiment is an embodiment in which the portable terminal 2 and the HMD 3 display data consisting only of images in cooperation with each other.

[0163] Fig.23 The figures show examples of data display screens according to the fifth embodiment, wherein (a) shows the screen of the portable terminal 2 before collaboration, (b-1) shows the screen of the HMD 3 after collaboration, and (b-2) shows the screen of the portable terminal 2 after collaboration.

[0164] Before the collaboration, a plurality of thumbnail images are displayed on the screen of the portable terminal 2 (see Fig.23 (a)). By selecting an image on the touch panel 208, the data to be sent to the HMD 3 is designated. Alternatively, the display order may be set in advance when creating the data, and the leading image may be sent to the HMD 3.

[0165] After the coordination, the HMD 3 displays the data in an enlarged size (optimized) in accordance with the screen size (see Fig.23 (b-1)) In addition, on the screen of the portable terminal 2, a mark indicating that the image has been selected is superimposed on the thumbnail image corresponding to the selected image (see Fig.23 (b-2)).

[0166] In collaboration, the data displayed on the HMD 3 can be changed by eye-line operation of the HMD 3. For example, by blinking twice, the data can be changed to the next data.

[0167] Fig.24 This is a control flow chart showing the flow of processing according to the fifth embodiment.

[0168] After the collaborative connection is established in steps S101 to S105 , image data A is selected from thumbnail images displayed on the screen of the portable terminal 2 ( S501 ) and transmitted to the HMD 3 ( S502 ).

[0169] HMD 3 receives the image data A ( S503 ), and returns an affirmative response ( S504 ).

[0170] The HMD 3 resizes the image data A ( S505 ), and displays the image data A on the display 307 ( S506 ).

[0171] The portable terminal 2 that receives the affirmative response of step S504 superimposes the frame on the portion of the image data A (S507), displays the image data A superimposed with the frame on the display 207 (S508), and disables the power saving mode (S114). Fig.13 The processing shown is the same, so repeated description is omitted.

[0172] According to the present embodiment, the portable terminal 2 and the HMD 3 cooperate to select one image from a plurality of thumbnail images displayed on the portable terminal 2 and display it in an enlarged manner on the HMD 3 , so that the image data A that the user wants to focus on can be easily visually recognized.

[0173] <Sixth Implementation Method>

[0174] The sixth embodiment is an embodiment in which one image and image data including a part of the image are displayed in coordination on the portable terminal 2 and the HMD 3 .

[0175] Fig.25 The figures show examples of data display screens according to the sixth embodiment, wherein (a) shows the screen of the portable terminal 2 before collaboration, (b-1) shows the screen of the HMD 3 after collaboration, and (b-2) shows the screen of the portable terminal 2 after collaboration.

[0176] Before the collaboration, the user 1 specifies the area to be enlarged on the touch panel 208 and specifies the HMD to send data. Alternatively, the enlarged area is pre-set when creating data, and the user 1 touches it to specify the HMD to send data. Furthermore, the area can be specified through the HMD 3 by sight.

[0177] After the collaboration, a mark is superimposed on the enlarged portion of the original image on the display 207 of the portable terminal 2 (see Fig.25 (b-2)).

[0178] In HMD 3, the data display size is enlarged (optimized) to match the screen size and displayed (see Fig.25 (b-1)).

[0179] The data displayed on the HMD 3 may be changed by the line of sight operation of the HMD 3. For example, by blinking twice, the data may be changed to the next data, the magnification position may be changed, the magnification may be changed, and the like may be changed.

[0180] Fig.26 It is a control flow chart showing the flow of processing involved in the sixth embodiment.

[0181] After the cooperative connection is established in steps S101 to S105 , a section to be enlarged is designated in the image displayed on the screen of the portable terminal 2 to designate enlarged image data ( S601 ), and the enlarged image data is transmitted to the HMD 3 ( S602 ).

[0182] The HMD 3 receives the enlarged image data ( S603 ), and returns an affirmative response ( S604 ).

[0183] The HMD 3 resizes the enlarged image data ( S605 ), and displays the enlarged image data on the display 307 ( S606 ).

[0184] The portable terminal 2 that receives the affirmative response of step S604 superimposes the frame on the enlarged portion of the image data (S607), displays the image data superimposed with the frame on the display 207 (S608), and disables the power saving mode (S609). Fig.13 The processing shown is the same, so repeated description is omitted.

[0185] According to the present embodiment, the portable terminal 2 and the HMD 3 can cooperate to designate a portion of an image displayed on the portable terminal 2 and enlarge the image and display it on the HMD 3 , so that the portion to be focused on can be easily visually recognized.

[0186] <Seventh Implementation Method>

[0187] The seventh embodiment is an embodiment related to a modification of the operation example of the HMD 3. The operation example according to this embodiment can also be appropriately used in the first to sixth embodiments described above.

[0188] The operation of the HMD 3 is not limited to line of sight detection. For example, voice input, gesture input, tapping of the frame, etc. may also be used.

[0189] The number of divisions of the data displayed on the portable terminal 2 is not limited to 2. The data may be divided into 3 or more, and the divided data may be sent to the HMD 3 and displayed in parallel on the display 307. The divided data may be displayed on the display 307 in sequence.

[0190] Furthermore, when data is displayed on the screen of the portable terminal 2 and the user 1 is viewing an image of the portable terminal 2 through the HMD 3 , the data displayed on the HMD 3 may be moved to the vicinity of the screen of the portable terminal 2 .

[0191] Furthermore, when the data displayed on the screen of the portable terminal 2 includes “animation”, the animation can also be operated by line-of-sight operation.

[0192] Fig. 27 1 is a diagram showing an example of viewpoint operation when a moving image is displayed on the portable terminal 2.

[0193] The animation may be displayed on the portable terminal 2, and the text or the like may be displayed on the HMD 3, and the animation may be operated by the position and time of the gaze point. The animation playback operation may be performed by continuously looking at a point for a predetermined time while keeping the line of sight on the portable terminal 2.

[0194] Alternatively, the user may visually recognize the portable terminal 2 through the HMD 3 and set the action of moving the line of sight to the left while focusing on the portable terminal 2 as a rewind operation, and the action of moving the line of sight to the right while focusing on the portable terminal 2 as a fast forward operation.

[0195] Furthermore, an action of continuously looking at a point other than the portable terminal 2 for a predetermined period of time across the HMD 3 may be set as an animation stop operation. In addition, the HMD 3 may be capable of customizing an animation operation corresponding to a user operation.

[0196] When the processor 301 of the HMD 3 detects an operation corresponding to the movement of the line of sight, it generates an instruction signal indicating the content of the operation and transmits it to the portable terminal 2. When the portable terminal 2 receives the instruction signal, it operates the animation display of the display 207 of the portable terminal 2 according to the content of the operation indicated by the instruction signal, and in conjunction with it, the text display of the HMD 3 also changes.

[0197] According to the present embodiment, with respect to an animation displayed on the HMD 3 in a state where the portable terminal 2 and the HMD 3 are coordinated, the animation can be played, fast-forwarded, rewound, and stopped by means of viewpoint input, so that the user 1 can perform operations related to viewing the animation without having to operate the portable terminal 2 with his hands even when his hands are occupied during operation.

[0198] As mentioned above, the invention completed by the present inventors has been specifically described based on the embodiments thereof, but the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the gist thereof.

[0199] For example, the first information device may be a tablet terminal instead of a smartphone, and the second information device may be smart glasses instead of a head mounted display.

[0200] In addition, the trigger event for restarting the screen display of the portable terminal 2 is not limited to the case where the face image of the user 1 can be recognized from the image taken by the internal camera 206 of the portable terminal 2, the portable terminal image can be recognized from the image taken by the external camera 305 of the HMD 3, and the line of sight of the user 1 is covered. For example, when the link destination is displayed on the display 307 of the HMD 3, when the user blinks twice in a state where the gaze point overlaps with the link destination, the page of the link destination can be displayed on the display 207 of the portable terminal 2. In addition, the voice recognition of the user 1's voice such as "screen open" can also be used as a trigger event.

[0201] In addition, the related data sent from the portable terminal 2 to the HMD 3 is not limited to the image corresponding to the thumbnail image, the segmented data of the data displayed on the portable terminal 2, and the text in the animation. For example, when a text in a first language is displayed on the display 207 of the portable terminal 2, the text translated into a second language may be displayed on the HMD 3 as the related data.

[0202] For example, any combination of the above-described embodiments is included in the present invention. In addition, modifications in which functions are added to the above-described embodiments are also possible.

[0203] In addition, the numerical values, messages, etc. that appear in the text and drawings are merely examples, and the effects of the present invention will not be impaired even if different examples are used.

[0204] In addition, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. In addition, the order of performing each process may be reversed and executed.

[0205] The functions of the present invention described above may also be implemented in hardware by designing part or all of them using, for example, an integrated circuit. Alternatively, they may be implemented in software by interpreting and executing an action program that implements each function, etc., by a processor unit, etc. Hardware and software may also be used in combination.

[0206] In addition, the control lines and information lines shown in the drawings are those considered necessary for explanation, and not necessarily all the control lines and information lines on the product are shown. It can also be considered that almost all the structures are actually connected to each other.

[0207] The implementation includes the following aspects.

[0208] (The first viewpoint)

[0209] A collaborative display system is formed by communicating with a first information device and a second information device, wherein:

[0210] The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device.

[0211] The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device.

[0212] The first information device has a normal mode in which the screen display of the first display is turned on and maintains device communication with the second information device, and a power saving mode in which the screen display of the first display is turned off when no operation on the first information device is performed for more than a predetermined time. When the collaborative action between the first information device and the second information device is started, the power saving mode of the first information device is invalidated and the screen display of the first display is switched to always on, and the second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after cutting out the second split data from the display data is displayed on the first display, and the second information device displays the second split data on the second display when receiving it.

[0213] (Method from the second point of view)

[0214] A collaborative display system is formed by communicating with a first information device and a second information device, wherein:

[0215] The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device.

[0216] The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device.

[0217] The first information device has a normal mode in which the screen display of the first display is turned on and the device communication with the second information device is maintained, and a first cooperative power saving mode in which the screen display of the first display is turned off when the time for which no operation on the first information device is performed becomes longer than a predetermined time. When the cooperative action between the first information device and the second information device is started, the first cooperative power saving mode of the first information device is made effective. When a trigger signal for turning on the screen display is input while the screen display of the first display is turned off, second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after the second split data is cut out from the display data is displayed on the first display. When a trigger signal for turning on the screen display is input while the screen display of the first display is turned off, the screen display of the first display is turned on. When the second information device receives the second split data, it is displayed on the second display.

[0218] (Third point of view)

[0219] A collaborative display system is formed by communicating with a first information device and a second information device, wherein:

[0220] The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device.

[0221] The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device.

[0222] The first information device has a normal mode in which the screen display of the first display is turned on and the device communication with the second information device is maintained, and a second collaborative power saving mode in which the first display is always turned on and other predetermined functions are turned off to save power compared with the normal mode. When the collaborative operation between the first information device and the second information device is started, the first information device is switched to the second collaborative power saving mode, and the second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after the second split data is cut out from the display data is displayed on the first display, and the second information device displays on the second display when receiving the second split data.

[0223] (Method of the fourth viewpoint)

[0224] A collaborative display method is performed by establishing a communication connection between a first information device and a second information device, wherein the collaborative display method comprises:

[0225] A step of starting a cooperative operation between a first information device and a second information device;

[0226] The step of disabling the power saving mode of the first information device and switching the screen display of the first display of the first information device to always be on;

[0227] a step of transmitting second divided data including a part of the display data displayed on the first display to the second information device;

[0228] The step of displaying the first divided data obtained by cutting out the second divided data from the display data on the first display;

[0229] The second information device receives the second divided data; and

[0230] The step of displaying the second divided data on a second display included in the second information device.

[0231] (Explanation of symbols)

[0232] 1: user; 2: portable terminal; 3: HMD; 5: network; 6: WEB server; 41: first data acquisition path; 42: second data transmission path; 43: third data acquisition path; 201: processor; 202: RAM; 203: ROM; 204: storage device; 205: external camera; 206: internal camera; 207: display; 208: touch panel; 209: acceleration sensor; 210: gyro sensor; 211: geomagnetic sensor; 212: Positioning sensor; 213: Distance sensor; 214: Telephone network communication I / F; 215: LAN communication I / F; 216: BT communication I / F; 217: Microphone; 218: Speaker; 219: Key input I / F; 220: Battery; 221: Bus; 230: Home button; 231: Earpiece; 232: Notification light; 251: Control program; 251a: 1st communication control unit; 251b: 1st display control unit; 252: Collaborative display application; 252a: 1st 1 cooperative display unit; 252b: first operation receiving unit; 252c: first image recognition unit; 252d: action mode control unit; 301: processor; 302: RAM; 303: ROM; 304: storage device; 305: external camera; 306: internal camera; 307: display; 309: acceleration sensor; 310: gyro sensor; 311: geomagnetic sensor; 312: positioning sensor; 313: distance sensor; 315: LAN communication I / F; 31 6: BT communication I / F; 317: microphone; 318: speaker; 319: key input I / F; 319a: operation key; 320: battery; 321: bus; 330: framework; 331: controller; 351: control program; 351a: second communication control unit; 351b: second display control unit; 352: collaborative display application; 352a: second collaborative display unit; 352b: second operation receiving unit; 352c: second image recognition unit; 352d: line of sight detection unit.

Claims

1. A collaborative display system, comprising: a first information device and a second information device connected to each other by communication, wherein: The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device. The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device. The first information device has a normal mode in which the screen display of the first display is turned on and maintains device communication with the second information device, and a power saving mode in which the screen display of the first display is turned off when no operation on the first information device is performed for more than a predetermined time. When the collaborative action between the first information device and the second information device is started, the power saving mode of the first information device is invalidated and the screen display of the first display is switched to always on, and the second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after cutting out the second split data from the display data is displayed on the first display, and the second information device displays the second split data on the second display when receiving it.

2. A collaborative display system, comprising a first information device and a second information device connected to each other for communication, wherein: The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device. The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device. The first information device has a normal mode in which the screen display of the first display is turned on and the device communication with the second information device is maintained, and a first cooperative power saving mode in which the screen display of the first display is turned off when the time for which the operation on the first information device is not performed becomes longer than a predetermined time. When the cooperative action between the first information device and the second information device is started, the first cooperative power saving mode of the first information device is made effective, the second split data including a part of the display data displayed on the first display is sent to the second information device, the first split data after the second split data is cut out from the display data is displayed on the first display, and when a trigger signal for turning on the screen display is input while the screen display of the first display is turned off, the screen display of the first display is turned on. When receiving the second divided data, the second information device displays the second divided data on the second display.

3. The collaborative display system according to claim 2, wherein: The second information device transmits the trigger signal to the first information device when detecting a trigger event for turning on screen display of the first display.

4. The collaborative display system according to claim 2, wherein: The second information device is a head mounted display or smart glasses having an external camera for photographing a real space and a line of sight sensor for detecting a line of sight of a user of the second information device. The second processor performs image recognition processing on the image captured by the external camera to recognize the first information device image, and sends the trigger signal to the first information device if it is determined that the line of sight overlaps with the first information device.

5. The collaborative display system according to claim 4, wherein: The second processor generates an instruction signal for playing, fast-forwarding, rewinding, or stopping the animation on the first display by determining the activity as the overlapping line of sight, and transmits the instruction signal to the first information device. The first information device operates the animation according to the instruction signal.

6. The collaborative display system according to claim 2, wherein: The first information device determines that the trigger signal has been input when detecting a trigger event for turning on screen display of the first display.

7. The collaborative display system according to claim 2, wherein: The first information device is a portable terminal having a built-in camera on the surface of the first display. The first processor determines that a trigger signal has been input when the first processor recognizes a face image of a user of the second information device by performing image recognition processing on the image captured by the internal camera.

8. A collaborative display system, comprising: a first information device and a second information device connected to each other for communication; The first information device comprises: a first display; a first communication device for transmitting and receiving data with a second information device; and a first processor for controlling the operation of the first display and the first communication device. The second information device comprises: a second display; a second communication device for transmitting and receiving data with the first information device; and a second processor for controlling the operation of the second display and the second communication device. The first information device has a normal mode in which the screen display of the first display is turned on and the device communication with the second information device is maintained, and a second collaborative power saving mode in which the first display is always turned on and other predetermined functions are turned off to save power compared with the normal mode. When the collaborative operation between the first information device and the second information device is started, the first information device is switched to the second collaborative power saving mode, and the second split data including a part of the display data displayed on the first display is sent to the second information device, and the first split data after the second split data is cut out from the display data is displayed on the first display, and the second information device displays on the second display when receiving the second split data.

9. The collaborative display system according to claim 1, wherein: The first display displays a plurality of thumbnail images, When the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device. The second processor displays the received image data corresponding to the selected thumbnail image on the second display.

10. The collaborative display system according to claim 2, wherein: The first display displays a plurality of thumbnail images, When the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device. The second processor displays the received image data corresponding to the selected thumbnail image on the second display.

11. The collaborative display system according to claim 8, wherein: The first display displays a plurality of thumbnail images, When the first processor receives a selection of at least one of the plurality of thumbnail images, it transmits image data corresponding to the selected thumbnail image to the second information device. The second processor displays the received image data corresponding to the selected thumbnail image on the second display.

12. The collaborative display system according to claim 1, wherein: When the first processor receives a selection of a part of an area in the image displayed on the first display, it sends the segmented data in the selected area to the second information device. The second processor displays the received divided data on the second display.

13. The collaborative display system according to claim 2, wherein: When the first processor receives a selection of a part of an area in the image displayed on the first display, it sends the segmented data in the selected area to the second information device. The second processor displays the received divided data on the second display.

14. The collaborative display system according to claim 8, wherein: When the first processor receives a selection of a part of an area in the image displayed on the first display, it sends the segmented data in the selected area to the second information device. The second processor displays the received divided data on the second display.

15. A collaborative display method, executed by connecting a first information device and a second information device to each other in a communication manner, wherein: The collaborative display method comprises: A step of starting a cooperative operation between a first information device and a second information device; The step of disabling the power saving mode of the first information device and switching the screen display of the first display of the first information device to always be on; a step of transmitting second divided data including a part of the display data displayed on the first display to the second information device; The step of displaying the first divided data obtained by cutting out the second divided data from the display data on the first display; The second information device receives the second divided data; and The step of displaying the second divided data on a second display included in the second information device.

Images