Portable terminal and head mount display and cooperative display system of the same

Abstract

To activate features of a portable terminal and an HMD, and achieve user-friendly operation and an easy-to-see display form, when the respective devices display information in cooperation with each other.SOLUTION: In a portable display system for connecting a portable terminal and a head mount display, and making the devices display information in cooperation with each other, the portable terminal displays information on a first display, transmits at least a part of the displayed information to the head mount display, and the head mount display shares and displays information transmitted from the portable terminal on a second display. Here, the information displayed on the portable terminal is hierarchical information, and the information displayed on the head mount display is set to be low order information in the hierarchical information. Whether a user is at rest or moving is detected by a sensor provided on the portable terminal, and when the user is at rest, display of the information on the portable terminal is canceled, and the information is displayed only on the head mount display.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a technique for connecting a mobile terminal and a head mounted display to display information in cooperation with each other. [Background technology]

[0002] Mobile devices such as smartphones can perform various operations by connecting to a network and installing various applications, and can record, play back, and display data. A head mounted display (HMD) is a device worn on the user's head and displays images on a glasses- or goggle-like display. HMDs are equipped with multiple sensors, such as a camera, a sensor that measures the distance to an object, and a GPS sensor that measures the position, as well as a CPU that processes images and a battery. Like mobile devices, HMDs can also install and upgrade applications to be used.

[0003] Furthermore, new functions and usage methods have been proposed by linking a mobile terminal with an HMD. For example, Patent Document 1 describes a configuration in which the screen displayed by the HMD and a terminal device (mobile information terminal) is shared, and an application is delivered from an external server device to the HMD by operating the screen of the terminal device, in order to easily manage applications (hereinafter, apps) used by the HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 138153 Summary of the Invention [Problem to be solved by the invention]

[0005] The use and range of applications of HMDs are expected to expand dramatically. However, because HMDs are originally worn on the head, there are limitations on the size of the screen that can be displayed, processing power, resolution, weight, etc.

[0006] Mobile devices have features such as large displays, advanced processing, and data storage, and can be used in a variety of ways, for which many applications have been developed. However, because the display screen is large, there is a possibility that people around you may be able to peek at what you are doing, so you need to be careful when using them from the perspective of privacy.

[0007] The above-mentioned Patent Document 1 is an example of a form in which an HMD and a mobile terminal are used in cooperation, but the operation for acquiring an application is performed only using an easy-to-operate mobile terminal, and there is no sufficient consideration yet to sharing functions by utilizing the characteristics of each device. For example, with regard to the display of hierarchical information and privacy protection, it is assumed that the two devices will display the information in the same way, and there is no particular consideration given to user usability.

[0008] An object of the present invention is to realize user-friendly operation and an easy-to-view display format by utilizing the characteristics of each device when displaying information by linking a mobile terminal and an HMD. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention employs the following method. The present invention relates to a mobile terminal that connects to a head mounted display and displays information in cooperation with the head mounted display, the mobile terminal comprising a first display that displays information, a memory that stores information to be displayed on the first display, a first control unit that controls the display of the first display, and a first external device interface that transmits and receives information between the head mounted display, and the first control unit controls at least a portion of the information to be displayed on the first display to be transmitted from the first external device interface to the head mounted display, and to be displayed on a second display provided to the head mounted display in a shared manner.

[0010] The present invention also provides a head-mounted display that is connected to a mobile terminal and worn on a user's head to display information in cooperation with the mobile terminal, the head-mounted display comprising a second display that displays information, a second control unit that controls the display of the second display, and a second external device interface that transmits and receives information to and from the mobile terminal, and the second control unit receives at least a portion of the information to be displayed on the mobile terminal via the second external device interface and shares and displays the information on the second display.

[0011] The present invention also relates to a linked display system in which a mobile terminal and a head-mounted display are connected and linked to display information, in which the mobile terminal displays information on a first display and transmits at least a portion of the displayed information to the head-mounted display, and the head-mounted display shares and displays the information transmitted from the mobile terminal on a second display. Effect of the Invention

[0012] According to the present invention, when a mobile terminal and an HMD are linked to display information, the characteristics of each device can be utilized to realize user-friendly operation and an easy-to-view display format. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing a cooperative display system in which a mobile terminal and an HMD are connected (Example 1). [Diagram 2] A diagram showing the appearance of the HMD. [Diagram 3] FIG. 2 is a diagram showing the internal configuration of a mobile terminal. [Figure 4] FIG. 2 is a diagram showing the internal configuration of an HMD. [Diagram 5] FIG. 4 is a diagram showing types of data input and output between the mobile terminal and the HMD. [Figure 6] FIG. 11 is a diagram showing an example of a display of an operation log on a mobile terminal. [Figure 7] 13A and 13B are diagrams showing other display examples of the operation log on the mobile terminal. [Figure 8] 13 is a diagram showing an example of hierarchical display of detailed information of an operation log. [Figure 9] FIG. 13 is a diagram showing an example of display shared between a mobile terminal and an HMD. [Figure 10] 13A and 13B are diagrams showing modified examples of the display on the mobile terminal. [Figure 11] 4 is a flowchart showing an operation procedure of the mobile terminal. [Figure 12] 4 is a flowchart showing an operation procedure of the HMD. [Figure 13] FIG. 13 is a diagram for explaining gaze touch using a camera of a mobile terminal (Example 2). [Figure 14] FIG. 13 is a diagram showing an example of a face image of a user captured by a camera of a mobile terminal. [Figure 15] 11 is a flowchart showing an operation procedure of gaze touch using the camera of the mobile terminal. [Figure 16] A diagram explaining gaze touch using an HMD camera. [Figure 17] FIG. 13 shows an example of an image of a mobile device captured by a camera of an HMD. [Figure 18] FIG. 13 is a diagram showing an example of determining the viewpoint position from a terminal image captured by an HMD camera. [Figure 19] 11 is a flowchart showing the operation procedure of gaze touch using the camera of the HMD. [Figure 20] FIG. 11 is a block diagram for automatically implementing an anti-peeping function (third embodiment). [Figure 21] 11 is a flowchart for automatically implementing a peep prevention function. [Figure 22] 11 is a flowchart showing a case where a user selects and executes a peep prevention function. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings. In the following embodiment, a cooperative display system in which a mobile terminal 1 and a head mounted display (HMD) 2 are connected and perform display operations in cooperation with each other will be described. EXAMPLES

[0015] 1 is a schematic diagram showing a linked display system in which a mobile terminal 1 and an HMD 2 are connected. The HMD 2 is worn on the head of a user U, and is wirelessly connected to the mobile terminal 1, and operates in cooperation while transmitting and receiving data and commands. The user U can view an image displayed on a display 10 of the mobile terminal 1, and can also view an image displayed on a display 30 of the HMD 2. A LAN (Local Area Network) or telephone network communication may be used for the wireless connection 9 here. Other communication methods such as Bluetooth (registered trademark) for short-range communication or an infrared communication unit may also be used, and the method may be selected according to the amount of information to be transmitted.

[0016] 2 is a diagram showing the external appearance of the HMD 2. The HMD 2 worn by the user is composed of components such as a display 30, a camera 40 (and a microphone), a touch sensor 37, bone conduction type earphones (audio output unit) 43, a CPU 32, and a battery 42. Details of each component will be described later.

[0017] 3 is a diagram showing the internal configuration of the mobile terminal 1. The mobile terminal 1 will be described as being, for example, a smartphone, but the mobile terminal may also be a mobile information terminal such as a tablet terminal.

[0018] Display 10 displays various images and information, and display control unit 11 controls the display on display 10. CPU (main control unit) 12 controls the overall operation of mobile terminal 1, and memory 13 stores operation programs and the like executed by CPU 12. Data storage memory 14 stores unique data of mobile terminal 1. Device ID information 15 stores unique device information for identifying mobile terminal 1.

[0019] The external device IF 16 is an interface for inputting and outputting data to and from the HMD 2 via a wireless connection 9. Connection authentication with the HMD 2 is performed via the external device IF 16, and data is transmitted to the mobile terminal 1, thereby determining whether or not the HMD 2 is connected to the mobile terminal 1. The touch sensor 17 is used to input a user's operation on the mobile terminal 1, and can be configured integrally with the display 10.

[0020] The camera 20 has an outer camera that captures the outside scene and an inner camera that captures the user U looking at the display 10, and a camera operation abnormality detection unit 20a detects abnormalities in the camera 20. Various sensors 21 detect the state (rotation, position, acceleration, etc.) of the portable terminal 1. The battery 22 supplies power to the portable terminal 1, and a battery capacity sensor 22a detects the remaining capacity of the battery 22. A user authentication unit 23 identifies the user of the portable terminal 1, and an external communication unit 24 communicates with an external server (such as a database) via an external network 4.

[0021] 4 is a diagram showing the internal configuration of the HMD 2. The display 30 displays an image input from the mobile terminal 1, and can superimpose image data generated within the HMD 2 and an image captured by the camera 40. A display control unit 31 controls the display on the display 30. A CPU (main control unit) 32 controls the overall operation of the HMD 2, and a memory 33 stores operation programs executed by the CPU 32, etc. Device ID information 35 stores unique device information for identifying the HMD 2.

[0022] The external device IF 36 transmits and receives data to and from the portable terminal 1 via a wireless connection 9. The touch sensor 37 inputs operations performed on the HMD 2 by the user.

[0023] The camera 40 captures the scene outside the HMD 2, and the camera operation abnormality detection unit 40a detects abnormalities in the camera 40. The various sensors 41 are sensors such as gyros that measure the position and distance of an object captured by the camera 40, and measure the movement, acceleration, and rotation of the HMD 2. The sensor operation abnormality detection unit 41a detects abnormalities in the operation of the various sensors 41. The battery 42 supplies power to the HMD 2, and the battery capacity sensor 42a detects the remaining capacity of the battery 7.

[0024] The audio output unit 43 drives bone conduction type earphones, and the audio output abnormality detection unit 43 a detects an abnormality in the operation of the audio output unit 43 .

[0025] FIG. 5 is a diagram showing types of data input / output between the portable terminal 1 and the HMD 2 via the external device IFs 16 and 36. As shown in FIG.

[0026] The data output from the portable terminal 1 shown in (a) to the HMD 2 includes a unique ID for identifying the portable terminal 1, image data for displaying information on the HMD 2, image data for displaying an AR (augmented reality) image taking into account the front-to-back relationship in the depth direction using camera image data and ranging data input from the HMD 2, and commands responding to operational input from the HMD 2.

[0027] The data output from the HMD2 shown in (b) to the mobile terminal 1 includes a unique ID for identifying the HMD2, image data captured by the camera 40, detection data from the various sensors 41 (position, distance, acceleration, rotation, battery capacity, user authentication, etc.), and operation input data for the HMD2 (touch operation, gesture, voice input, etc.).

[0028] <Mobile device operation> Various applications operating on the mobile terminal 1 are stored in the memory 13, and a plurality of application icons that can be selected by the user are displayed on the home screen that appears at startup. Touching these displayed icons launches the corresponding application.

[0029] The mobile terminal 1 has an operation log display function that records when, where, which application, and how which device was used, and can display it as needed. Below, an "operation log app" that displays the "operation log" operated using the mobile terminal 1 and the HMD 2 will be described as an example.

[0030] For this purpose, the "operation log app" automatically starts the operation of acquiring the operation log from the time the mobile terminal 1 is started. With this function, operation logs such as the startup time of the app for which an operation instruction was input, the app name, the operation content, the location information, and the use of the HMD 2 are stored in the data storage memory 14.

[0031] When displaying the stored operation log, the operation log information stored in the data storage memory 14 is displayed as a list on the display 10. When an operation log displayed by the touch sensor 17 is selected, the details of the selected operation log are displayed.

[0032] Note that the operation log information stored in the data storage memory 14 is transmitted to, for example, an external app company 5a via the external communication unit 24 and the network 4 and used for investigating the usage status of the mobile terminal 1. Regarding other apps stored in the mobile terminal 1 as well, information such as the usage status is similarly transmitted to an external app company 5b.

[0033] <Operation of the HMD> Various data are sent from the portable terminal 1 to the HMD 2 via the external device IFs 16 and 36. This data includes operation log information, which is also displayed on the display 30 of the HMD 2. However, in this embodiment, the HMD 2 shares and displays detailed information (lower-level information and additional information in the hierarchical information) of an item selected by the user through a touch operation from among the operation logs displayed on the portable terminal 1. This makes it possible to realize an easy-to-view display format by taking advantage of the characteristics of each device (the portable terminal 1 and the HMD 2). Alternatively, as in the third embodiment described below, if there is a possibility that displaying the information on the portable terminal 1 may cause people around to peek at it, the information is shared and displayed on the HMD 2. This has the effect of protecting the user's privacy.

[0034] FIG. 6 is a diagram showing an example of the display of an operation log in the mobile terminal 1. The mobile terminal 1 records the operation contents of the user using an operation log application. FIG. 6 shows an example of the operation contents of one day displayed in chronological order. For example, if the first operation on the date: 2019 / 11 / 28 is to ring an alarm in the clock application of the mobile terminal 1, the operation log displays the following: time: 06:05, application (app): clock, operation contents: alarm, address, HMD2 unused (HMD icon not displayed). Next, the time: 08:30, application: SMS (Short Message Service), operation contents: 5 messages sent and received, HMD2 used (HMD icon displayed). The operation contents are displayed sequentially in the same manner, and the final operation of the day displays the following: time: 23:30, application: e-book application, operation contents: reading, HMD2 unused (HMD icon not displayed).

[0035] Note that, here, the operation log is displayed based on the date: 2019 / 11 / 28, but you can also select and display recorded information other than the date, such as the number of rows and columns to be displayed, or the time.

[0036] Fig. 7 is a diagram showing another display example of the operation log on the mobile terminal 1. While Fig. 6 displays the operation contents for one day, Fig. 7 displays the operation contents for multiple days. That is, this is an example of displaying in a format that allows past operation logs to be easily referred to by stacking multiple days' worth of data, each day being one page, and turning the pages like a diary.

[0037] The operation logs shown in FIG. 6 and FIG. 7 are information related to the operation content, and furthermore, detailed information executed by the operation can be displayed.

[0038] 8 is a diagram showing an example of hierarchical display of detailed information of an operation log. Here, an example is shown in which the date, time, application, operation content, etc. are selected by touching with a finger, and detailed information is displayed.

[0039] For example, suppose that on the full-screen display (a), you touched "14 photos" in the camera app at 17:30 (Touch 1). As a result, the recorded photos are displayed by type, as shown in (b) (display example for Touch 1). Here, six categories are displayed, and the number of photos in each category is also displayed.

[0040] If you touch this displayed number information again (Touch 2), a list of photos in the touched category will be displayed, as shown in (c) (Example of display with Touch 2). In this way, by digging deeper into the hierarchy of the displayed content, you can get a concrete understanding of the operations you performed at that time. You can also look back on those operations later, like keeping a diary.

[0041] Up to this point, an example in which the operation log is displayed only on the mobile terminal 1 has been shown, but a part of the information can also be transmitted to the HMD 2 and shared and displayed on the HMD 2. The mobile terminal 1 can display a lot of information as shown in Figs. 6 to 8. However, if information on a lower level is displayed simultaneously with the operation log, the display area of ​​each becomes small, making the displayed information difficult to see. Therefore, the HMD 2 is used to share and display, for example, detailed information on a lower level. Since the HMD 2 is a device mounted on the user's head, the character size that can be displayed is smaller and the content is less than that of the mobile terminal 1. Therefore, the detailed information on a lower level of the hierarchy caused by the operation of Touch 1 out of the operation log displayed on the mobile terminal 1 described in Fig. 8 is shared and displayed on the HMD 2.

[0042] 9 is a diagram showing an example of display shared between the mobile terminal 1 and the HMD 2. Here, the entire screen of the operation log is displayed on the mobile terminal 1 as in (a), and detailed information at a lower level is displayed on the display 30 of the HMD 2 as in (b). As detailed information, for example, as described above, categories of photo types are displayed as in FIG. 8(b), or multiple photos included in a category are displayed in sequence as thumbnails or photo images as in FIG. 8(c). This makes it easier for the user to view the information compared to when it is displayed only on the mobile terminal 1.

[0043] Furthermore, the HMD 2 has a different optical system from the mobile terminal 1, and is characterized in that the display contents are less likely to be seen by others in the vicinity. By displaying on the HMD 2 information that the user does not want others to see among the information displayed on the mobile terminal 1 (for example, photos), only the user can see it, which is advantageous in terms of privacy protection. Also, since the amount of information that the mobile terminal 1 can display is limited, by sharing the display information with the HMD 2, it can be used as a multiple display, improving usability.

[0044] In this embodiment, the operation log is shared between the mobile terminal 1 and the HMD 2. The user can select display items such as photos and SMS message contents in the operation log app settings for the operation log displayed on the HMD 2, or display detailed contents of the operation log displayed on the mobile terminal 1. When there are people around, it is also possible to display the selected items on the HMD 2 and display detailed contents of the operation log. Of course, the display on the HMD 2 is limited to when the HMD 2 is connected to the mobile terminal 1, and the display data is transmitted from the mobile terminal 1 to the HMD 2. The operation when there are other people around will be described in detail in the third embodiment.

[0045] Fig. 10 is a diagram showing a modified example of the display on the mobile terminal 1. Compared with the display example shown in Fig. 6, only the operations using the HMD2 are displayed in the operation log, and the operations not using the HMD2 are deleted. For example, the operation log at time: 06:05 (app: clock, operation: alarm), the operation log at time: 17:30 (app: camera, operation: 14 photos), and the operation log at time: 23:30 (app: e-book app, operation: reading) displayed in Fig. 6 did not use the HMD2, and are not shown in Fig. 10. It goes without saying that by selecting an operation log using the HMD displayed here, the details can be displayed as shown in Fig. 8, or the operation can be shared and displayed on the HMD2 as shown in Fig. 9.

[0046] Next, the operation flows of the mobile terminal 1 and the HMD 2 for implementing the shared display function of this embodiment will be described. 11 is a flowchart showing the operation procedure of the mobile terminal 1. The operation of the mobile terminal 1 is started (S100), an operation log application is launched, and a display function of an operation log operated on the mobile terminal 1 is executed (S101).

[0047] A connection check is performed between the mobile terminal 1 and the HMD 2 (S102). To this end, the mobile terminal 1 checks whether data is being normally transmitted and received between the mobile terminal 1 and the HMD 2, and whether the HMD 2 is being worn by the user, based on sensor information (such as acceleration information) from the HMD 2. This connection check is performed periodically or when an event occurs on the mobile terminal 1 (such as selection of a display item). If a connection error is determined in S102, communication with the HMD 2 is terminated and the connection is severed (S111).

[0048] If the connection is determined to be OK in S102, the user selects a display method for the operation log by selecting whether to display the operation log solely on the mobile terminal 1 (examples of Figs. 6 to 8) or to display the operation log shared between the mobile terminal 1 and the HMD 2 (example of Fig. 9) (S103).

[0049] The mobile terminal 1 creates full display data of the operation log summary (display of the contents of the operation log explained in FIG. 6). In addition, among the data, display data of the operation log summary to be displayed on the HMD 2 when the HMD 2 is used is also created (S104). The created full display data of the operation log summary is displayed on the display 10 of the mobile terminal 1 (S105).

[0050] When the user touches and selects an item for which the user wishes to display details from among the operation logs displayed on the display 10 (S106), detailed display data is created (S107). For example, when the operation log is touched as shown in Fig. 8(a) (touch 1), the data to be displayed by touch 1 is extracted from the data storage memory 14 and created.

[0051] Next, the process branches based on the display method of the operation log selected in S103 (S108). If the operation log is to be displayed solely on the mobile terminal 1, the process returns to S105, and the detailed display data created in S107 is displayed on the display 10 of the mobile terminal 1. Furthermore, in S106, if the user touches an item for which details are to be displayed (touch 2), detailed data to be displayed by touch 2 is created and displayed.

[0052] On the other hand, in the branch of S108, if the display is shared between the mobile terminal 1 and the HMD 2, the display destination is switched so that part of the display data is displayed on the HMD 2 (S109). Then, part of the display data is output to the HMD 2 via the external device IF 16 (S110). For example, as shown in FIG. 9, the detailed data selected by touch 1 is output to the HMD 2. The HMD 2 receives this detailed data and displays it on the display 30.

[0053] Fig. 12 is a flowchart showing the operation procedure of the HMD 2. Here, a case will be described in which the operation log is shared and displayed on the HMD 2 as shown in Fig. 9. The operation of the HMD 2 is started (S200), and a connection check with the portable terminal 1 is performed (S201).

[0054] The camera operation abnormality detection unit 40a and the sensor operation abnormality detection unit 41a check the operation of various components (camera, sensor, display, etc.) mounted on the HMD 2 (S202). Also, the battery capacity sensor 42a checks the remaining capacity of the battery 42 of the HMD 2 (S203).

[0055] If a connection error is determined in S201, the occurrence of the error is displayed on the display 30 (S211). If an operation error is found in any of the components in S202, or if the remaining battery charge is low in S203, error information is sent to the portable terminal 1 (S210), and the error information is displayed on the display 30 of the HMD 2 (S211). Note that by outputting the error information to the portable terminal 1 in S210, the HMD 2 may be instructed to display the error in S211 by the portable terminal 1.

[0056] If no abnormality is found in the checks of S201 to S203, the check results of the HMD 2 are output to the portable terminal 1 (S204). The portable terminal 1 then confirms that the HMD 2 is worn by the user, that various operations are normal, and that the display data output from the portable terminal 1 can be displayed.

[0057] The HMD 2 acquires display data from the mobile terminal 1 via the external device IF 36 (S205). This corresponds to step S110 in FIG. 11. The acquired display data is displayed on the display 30 of the HMD 2 (S206). As a result, for example, an image as shown in FIG. 9(b) (after touch 1) is displayed.

[0058] On the display screen of S206, the user selects the next item (icon) to be displayed by a touch operation (S207). For example, in FIG. 9(b), a touch operation of touch 2 is accepted. Alternatively, the user may select to end the display. The user's selection operation is performed by the touch sensor 37 of the HMD 2 or by gesture input.

[0059] Information on the item selected in S207 is output to the mobile terminal 1 via the external device IF 36 (S208). The mobile terminal 1 creates detailed data of the selected item, and the HMD 2 acquires the detailed data from the mobile terminal 1 (S209). Then, the process returns to S206, and the acquired detailed data is displayed on the display 30. As a result, for example, a display responsive to the operation of Touch 2 in FIG. 9(b) is performed.

[0060] The "operation log" used in the above description is hierarchical information, and is a suitable example for explaining the operation of reading out information of a lower level. In other words, the operation log is configured by recording the instruction operations performed by the user on the mobile terminal in chronological order, together with information on the location at that time, linked devices, etc.

[0061] Sharing the information displayed by the mobile device and the HMD is not limited to operation logs. As another example, it can be applied to the case where, while reading a book on the mobile device, the meaning of words that are not understood can be displayed on the HMD, or when a map is displayed on the mobile device, detailed information about a specific place on the map can be displayed on the HMD. Also, while watching a movie on the mobile device, it is possible to check the blogs of the characters or search for background geographical information on the HMD. In short, it becomes possible to share and display additional information on the HMD so as not to obstruct the screen of the mobile device.

[0062] In this way, in the first embodiment, the mobile terminal and the HMD cooperate to configure the display screen. In this case, the HMD is treated as a child screen of the mobile terminal, and information that cannot be displayed on the screen of the mobile terminal or information that is lower in the hierarchy is displayed on the HMD side. Therefore, the display contents of the mobile terminal are maintained as they are, and the contents can be confirmed by looking at the mobile terminal, while more detailed information can be viewed at the same time on the HMD. This makes it possible to realize easy-to-use operation and an easy-to-view display form by taking advantage of the characteristics of each device. EXAMPLES

[0063] In the first embodiment, a method of using the touch sensor 17, 37 of the mobile terminal 1 or HMD 2 to select a corresponding item when the user wishes to display detailed contents of the displayed operation log has been described. Generally, a camera is often mounted on the mobile terminal 1 or HMD 2. In the second embodiment, a method of detecting the user's line of sight using the camera and selecting an item (hereinafter, referred to as "gaze touch") will be described.

[0064] 13 is a diagram explaining gaze touch using the camera 20 of the mobile terminal 1. Gaze touch is a method of detecting which item the user's eyeball direction (gaze) is looking at and selecting that item. (a) shows a state in which the user U is looking at the display screen of the mobile terminal 1. The camera (in-camera) 20 mounted on the mobile terminal 1 is used to detect which position (which item) on the display screen the user U is looking at, in other words, where the gaze is located.

[0065] (b) shows an example of the screen of the mobile terminal 1. In detecting the line of sight of the user U, first, for example, five cross points (X1 to X5) are displayed in sequence at predetermined time intervals on the screen of the mobile terminal 1. The user gazes at these displayed cross points in order, following instructions from the mobile terminal 1. The direction of the eyeball at that time, i.e., the line of sight direction, is detected by the camera 20 of the mobile terminal 1, and the reference position and angle of the user's line of sight are found from the line of sight directions to these five cross points.

[0066] For example, the position of the face and the positions and orientations of the pupils and irises of the eyes when the user gazes at cross point X1 in the center of the screen are used as a reference, and the difference in the positions and orientations of the pupils and irises of the eyes when the user gazes at cross points X2 and X3 at the top edge of the screen is detected. Similarly, the difference in the positions and orientations of the pupils and irises of the eyes when the user gazes at X4 and X5 at the bottom edge of the screen is detected using X1 in the center of the screen as a reference. Furthermore, using the same method, the difference in the positions and orientations when the user gazes at the center and the left and right edges is detected, and these detection values ​​are stored in data storage memory 14.

[0067] (c) shows a state in which user U is actually gazing at a certain position on the screen. The user's gaze position (the position of the face and the positions and directions of the pupils and irises of the eyes) captured by camera 20 is compared with the gaze position relative to the cross point, which is a reference position, to calculate the current gaze position of the user. Then, a marker M indicating the gaze point of the user's gaze is displayed at the calculated position on the screen.

[0068] In gaze detection, the values ​​of the angle, position, and distance between the mobile terminal 1 and the face and eyeballs of the user U are important. When used for gaze touch, the above-mentioned reference direction of gaze needs to be corrected by checking the fluctuation (precision) of the angle, position, and distance between the mobile terminal 1 and the face and eyeballs, as well as the pupils and irises, using the camera 20 and distance sensor 21 of the mobile terminal 1.

[0069] FIG. 14 is a diagram showing an example of a user's face image (user image) captured by the camera (in-camera) 20 of the mobile terminal 1. (a) shows a state in which the user U is gazing at a reference position on the screen (cross point X1 in FIG. 13). From this image, the line of sight direction of the user U is obtained and used as a reference value. (b) shows a state in which the user U is currently gazing at a desired position (selected position) on the screen. When the gaze position changes, the image of the user U, particularly the position, angle, etc. of the user's face and eyeballs, changes.

[0070] In this example, it can be seen that the face size is smaller and the distance is greater in the gaze image (b) at the selected position compared to the gaze image (a) at the reference position. Also, the center position of the eyes is shifted downward as shown by the dashed line L1→L2. Note that the distance to the face can be detected by the distance sensor 21 instead of the difference between the camera images.

[0071] Here, the measurement values ​​of the face and eye positions and distances obtained in the gaze image of the selected position in (b) are set as the reference values ​​obtained when gazing at the five cross points (X1 to X5) in Fig. 13(a), and the difference between them is calculated and corrected to calculate the position on the screen that the user is currently gazing at. In other words, it is possible to detect which position on the screen the user is gazing at by using the camera 20 of the mobile terminal 1.

[0072] FIG. 15 is a flowchart showing an operational procedure of gaze touch using the camera 20 of the mobile terminal 1. As shown in FIG.

[0073] The operation of the mobile terminal 1 is started (S300), and the gaze touch application is started (S301). The initial setting of the gaze touch is started (S302). An image for the initial setting of the gaze touch is displayed on the display 10 of the mobile terminal 1 (S303). For example, cross points X1 to X5 as shown in FIG. 13(a) are displayed in sequence.

[0074] The user sequentially gazes at the images (cross points) displayed on the display 10 through the HMD 2 that the user is wearing (S304). An image of the eyeball of the user who is gazing is captured by the camera 20 mounted on the mobile terminal 1 (S305).

[0075] A reference value for the gaze direction is calculated from the information of the user image acquired when the cross point X1 is displayed, and is set as the initial position of the gaze touch (S306). The calculation result of the gaze position when gazing at each cross point X1 to X5 in sequence is stored together with the information on the initial position of the gaze touch. The positional relationship (angle, position, distance) between the mobile terminal 1 and the eyeball at this time is acquired by the sensor 21 of the mobile terminal 1. Furthermore, when the HMD 2 is worn, information on the distance to the mobile terminal 1 and the positions and directions of the pupils and irises of the eyeballs are also acquired by the eye tracking camera (S307).

[0076] The initial position calculated in S306 and the positional relationship information between the portable terminal 1 and the eyeball when gazing at each cross point obtained in S307 are linked to the device ID 35 of the HMD 2 and user information, and are stored in the data storage memory 14 of the portable terminal 1 (S308).

[0077] This completes the initial setting (S309). Note that the average value of the gaze points of cross points X2 to X5 may be calculated, and if it is found to be in roughly the same position as the gaze point of cross point X1, it may be determined that each has been correctly detected.

[0078] Next, an actual eye-gaze touch operation is performed. The user gazes at a desired operation position on the screen of the portable terminal 1 (S310). The camera 20 mounted on the mobile terminal 1 captures an image of the eyeball of the user who is gazing (S311).

[0079] The user's gaze direction is obtained from the acquired user image information, and compared with the positional relationship information at each point of gaze stored in the data storage memory 14 in S308 (to obtain the difference), to calculate the user's gaze point (S312).

[0080] The position of the gaze point, i.e., the gaze touch position, is displayed with a marker on the screen of the mobile terminal 1 (S313). If the gaze point remains at the position where the marker is displayed for a certain period of time or more, an operation similar to clicking or touching with a fingertip is performed on the position where the marker is present.

[0081] In order to accurately perform the above-mentioned gaze touch operation, it is necessary to constantly check the distance between the mobile terminal 1 and the face, the angle between the face and the eyes, the position, the distance, etc., and, if any changes occur, it is necessary to correct the change in gaze position due to the change in the face or eyes from the reference point. Therefore, based on the initial setting position, the angle, position, distance, etc., between the mobile terminal and the face or eyes are constantly monitored, and when a deviation of a certain amount or more occurs, the deviation is corrected based on the value calculated from the gaze movement range obtained in the initial setting.

[0082] This eye contact correction involves displaying an icon that is frequently used in the application, such as the icon for ending the operation log, in a fixed position as a landmark, and checking at a fixed frequency whether the icon is accurately understood, thereby correcting any deviations that can be corrected.

[0083] The gaze touch operation can also be performed using the camera 40 mounted on the HMD 2. This will be described below.

[0084] 16 is a diagram for explaining gaze touch using the camera 40 of the HMD 2. In order to specify the gaze position, for example, images of cross points X1 to X3 serving as references are displayed on the screen of the mobile terminal 1 as shown in FIG. 16. For example, point X1 is displayed in the center of the screen, point X2 in the upper left corner of the screen, and point X3 in the lower right corner of the screen. The user U gazes at these displayed cross points according to instructions from the mobile terminal 1, and takes an image with the camera 40 of the HMD 2. The image taken by the camera 40 is then sent to the mobile terminal 1.

[0085] The mobile terminal 1 obtains a reference center position from the acquired camera image of each cross point. For example, the position of cross point X1 in the center of the screen is compared with the average position of cross points X2 and X3 in the corners of the screen, and if they are approximately the same position, they are set as the reference position for gaze touch. If an eye tracking camera is mounted on the HMD 2 to detect the position and angle of the pupil and iris of the eye, that information may be used. Note that for gaze detection, the angle, position, and distance between the mobile terminal 1 and the HMD 2, and the position and angle between the HMD 2 and its own eyeball (position and angle when mounted on the head) are important.

[0086] 17 is a diagram showing an example of an image of the portable terminal 1 captured by the camera 40 of the HMD 2. From the image of the portable terminal 1 captured by the camera 40 of the HMD 2 (herein referred to as the "terminal image"), the gaze point of the user U on the screen of the portable terminal 1 is detected. In adjusting the initial position, if the image is captured as in the terminal image 1 of (a), the entire screen of the portable terminal 1 is visible, and information for detecting which point the user is gazing at can be obtained, which is appropriate.

[0087] However, for example, if the distance between the mobile terminal 1 and the HMD 2 is too far, the entire mobile terminal 1 will be captured as a small image, as in the terminal image 2 in (b). This will cause an error in detecting the gaze point, making it difficult to detect the gaze point accurately. In this case, an instruction is issued for the HMD 2 to move closer to the mobile terminal 1.

[0088] In addition, in the case of terminal image 3 in (c), the center of shooting is shifted toward the upper left of mobile terminal 1, so an instruction is given to move it toward the lower right. Conversely, in the case of terminal image 4 in (d), the center of shooting is shifted toward the lower right of mobile terminal 1, so an instruction is given to move it toward the upper left.

[0089] The terminal image 5 in (e) captures the entire screen of the mobile terminal 1 to the maximum extent possible, indicating that it is in a position where it cannot be approached any closer. In this way, the mobile terminal 1 is adjusted so that it is captured in the state of the terminal image 1 in (a).

[0090] As in the case of using the camera 20 of the mobile terminal 1 described above, it is necessary that the angle, position, distance, etc. between the mobile terminal 1 and the HMD 2 are within a certain range. Therefore, it is necessary to constantly check the changes in the angle and position between the mobile terminal 1 and the HMD 2 and make corrections to the reference position of the line of sight set above.

[0091] 18 is a diagram showing an example of determining the viewpoint position from a terminal image captured by the camera 40 of the HMD 2. (a) shows a case where the reference position of the screen of the mobile terminal 1 is determined, and (b) shows a case where the user's viewpoint position is determined from an actually acquired terminal image.

[0092] In the terminal image of (a), the distance and angle of the HMD 2 relative to the screen of the mobile terminal 1 are detected using three cross eyedroppers X1 to X3 to determine the reference position. The difference between the values ​​at that time and the terminal image of (b) is detected. In this example, the center of the screen (cross point X1) of the terminal image of (b) is in the center of the range captured by the HMD 2, so it can be seen that the mobile terminal 1 is almost directly in front of the HMD 2. Also, because the screen of the mobile terminal 1 is captured large, it can be seen that the distance to the HMD 2 is closer than when the terminal image of (a) was captured.

[0093] Here, if the screen of the mobile terminal 1 is deformed into a trapezoid or diamond shape, it is because the mobile terminal 1 is being held at an angle. Even in such a case, the state of the angle can be detected by detecting the position information of the four corners of the screen of the mobile terminal 1. Then, it becomes possible to detect where on the screen of the mobile terminal 1 the gaze is being directed in (b) relative to the gaze point position obtained by setting the reference position in (a). Note that if an eye tracking camera is mounted on the HMD 2 and the positions and directions of the pupils and irises of the eyes can be measured, the accuracy of determining the gaze point can be improved by using these as parameters.

[0094] FIG. 19 is a flowchart showing the operation procedure of gaze touch using the camera 40 of the HMD 2.

[0095] The operation of the HMD 2 is started (S400), the gaze touch application is started, and a command to start up the application is also sent to the mobile terminal 1 (S401). The initial setting of the gaze touch is started (S402). The mobile terminal 1 displays an image (cross points) for the initial setting of the gaze touch on its display 10 (S403). For example, cross points X1 to X3 as shown in FIG. 16 are displayed.

[0096] The user gazes at the image displayed on the mobile terminal 1 via the HMD 2 (S404). An image of the portable terminal 1 that the user is gazing at is taken by the camera 40 mounted on the HMD 2, and is sent to the portable terminal 1 via the external device IF 36. The portable terminal 1 acquires the image of the portable terminal taken by the HMD 2 (terminal image) (S405).

[0097] The mobile terminal 1 calculates the position and angle of the HMD 2 from the position of the displayed image and the terminal image captured by the HMD 2, and sets the reference position for eye-gaze touch (S406). At this time, if the mobile terminal 1 and HMD 2 are too far apart or the angle is too oblique, an instruction is given to the user to correct the holding position of the mobile terminal 1. Then, information on the angle and position between the mobile terminal 1 and HMD 2, and information on the angle of the pupil and iris of the eye are obtained by an eye tracking camera mounted on the HMD 2 (S407).

[0098] The initial position calculated in S406, and the angle and position information between the portable terminal 1 and the HMD 2 acquired in S407 are linked with the device ID 35 of the HMD 2 and user information, and are stored in the data storage memory 14 (S408). This completes the initial setting (S409).

[0099] Next, an actual eye-gaze touch operation is performed. The user gazes at a desired operation position on the screen of the portable terminal 1 through the HMD 2 (S410). An image of the mobile terminal 1 being gazed at (terminal image) is acquired by the camera 40 mounted on the HMD 2, and is sent to the mobile terminal (S411).

[0100] The mobile terminal 1 compares the acquired terminal image with the angle and position information of the HMD 2 stored in the data storage memory 14 in S408, and calculates where on the screen of the mobile terminal 1 the gaze is being directed (S412).

[0101] The position of the gaze point, i.e., the gaze touch position, is displayed with a marker on the screen of the mobile terminal 1 (S413). If the gaze point remains at the position where the marker is displayed for a certain period of time or more, an operation similar to clicking or touching with a fingertip is performed on the position where the marker is present.

[0102] Based on the initial setting position, the distance and angle between the mobile device and the HMD, as well as the angle of the HMD itself when worn by the user, are constantly monitored, and if any deviation occurs beyond a certain level, the deviation is corrected.

[0103] In the above example, the image captured by the camera 40 of the HMD 2 is sent to the portable terminal 1, and the distance and angle are calculated on the portable terminal 1 side, but this is not limited to the above, and these processes can also be performed on the HMD 2 side or on an external server via a network.

[0104] According to the second embodiment, a user can select a desired display item by simply gazing at the display screen of a mobile device, and can display detailed information related to the selected item. Therefore, the convenience of a user operation for switching the display contents is improved. EXAMPLES

[0105] In the first embodiment, a part of the details of the operation log displayed on the mobile terminal 1 is displayed on the HMD 2. In some cases, the display of the operation log on the mobile terminal 1 may not be desired to be seen by others from the viewpoint of privacy. In this respect, the HMD 2 has the advantage that, due to the difference in the optical system, the image is not focused on the lenses of the glasses, which are the display surface, making it difficult for people around to see what is being displayed. Therefore, in the third embodiment, a configuration will be described in which privacy is taken into consideration and the HMD 2 is actively used for display.

[0106] In this embodiment, the acquired operation log information is displayed by appropriately switching between the mobile terminal 1 and the HMD 2. In order to prevent the displayed operation log from being seen by people in the vicinity, an "anti-peeping function" is added. In an environment where the operation log is likely to be peeked at or when it is not desired to show it, the "anti-peeping function" hides the operation log information on the mobile terminal 1 and displays it on the HMD 2. The "anti-peeping function" may be implemented automatically or may be implemented by the user.

[0107] First, a case where the peep prevention function is automatically performed will be described. 20 is a diagram for explaining a configuration for automatically implementing the anti-peeping function. A user U is wearing an HMD 2 and looking at the display screen of a portable terminal 1. There is a possibility that a person Un is present around the user U and is peeking at the screen of the portable terminal 1.

[0108] In such a case, the camera 20 (which is an in-camera that takes pictures of the user U) mounted on the mobile terminal 1 captures an image of the user's surroundings when the mobile terminal 1 is being used. If a person Un other than the user U appears in the captured camera image C1, it is determined that there is a possibility of peeping. At this time, the camera 40 of the HMD 2 may be used to capture an image in front of the user, and it may be possible to detect whether a person Un who may be peeping is present in the captured camera image C2.

[0109] The key point here is that it is desirable for the photographing cameras 20, 40 to be equipped with wide-angle lenses. The degree of the wide angle is required to be equal to or larger than the viewing angle at which the display 10 of the mobile terminal 1 can be seen from all around.

[0110] In addition, the direction of the face of a person Un other than the photographed user U is detected, and the angle with the display 10 of the mobile terminal 1 is detected. If the angle is greater than or equal to the viewing angle of the display 10, it can be determined that the possibility of peeking is low.

[0111] In order to identify the user U appearing in the camera images C1 and C2, it is possible to use face recognition, etc. Also, when the peep prevention function is activated, it is important to monitor by operating the cameras 20 and 40 constantly or at regular intervals.

[0112] Operating the cameras 20 and 40 all the time or continuously at regular intervals may result in unintended camera photography, and conversely, it is necessary to consider the privacy of others. Therefore, it is necessary to clearly indicate on the mobile terminal 1 that the camera is operating or that the anti-peeping function is operating so that people around you are aware of it. For example, it is desirable to clearly indicate this by turning on a lamp on the mobile terminal 1.

[0113] FIG. 21 shows a flow chart for automatically implementing the peep prevention function. The operation of the mobile terminal 1 is started (S500), an operation log application is started, and a display function of an operation log operated on the mobile terminal 1 is executed (S501). The user is authenticated (S502), and the connection with the HMD 2 is checked (S503). If a connection error is detected, communication with the HMD 2 is terminated, and the connection is cut off.

[0114] It is determined whether the user is moving or not based on acceleration information from the sensor 21 of the portable terminal 1 (S504). If the user is moving, the process proceeds to S509, and since there is little possibility of peeking, the image is displayed using the screen of the portable terminal 1. If it is determined in S504 that the user is stationary, there is a possibility that the user may be peeked at, and therefore the "peeping prevention function" is executed (S505).

[0115] An image of the user's surroundings is captured by the camera 20 of the mobile terminal 1. Alternatively, an image of the surroundings of the mobile terminal 1 may be captured by the camera 40 of the HMD 2 (S506). The captured camera image is used to determine whether or not there is a person other than the user. Furthermore, the angle between the captured person and the display 10 of the mobile terminal 1 is used to determine whether or not there is a possibility of peeking (S507).

[0116] If it is determined in S507 that there is a possibility of peeping, the display of the operation log on the mobile terminal 1 is stopped, and display is performed only on the HMD 2 (S508). Also, if an operation (selection) is performed to display more detailed information on the information of the operation log displayed on the HMD 2, the more detailed information is displayed only on the HMD 2. On the other hand, if it is determined in S507 that there is no possibility of peeking, the operation log is displayed on the portable terminal 1 as it is, or using both the portable terminal 1 and the HMD 2 (S509).

[0117] Then, the process returns to S504 to determine again whether the user is moving, and in S507, the display is switched based on the determination of the possibility of peeking. In this manner, display using only the HMD 2 continues until the possibility of peeking disappears.

[0118] Next, a case where the peep prevention function is selected and implemented by the user will be described. FIG. 22 shows a flowchart in the case where the peep prevention function is selected and implemented by the user. The operation of the mobile terminal 1 is started (S600), an operation log application is started, and a display function of an operation log operated on the mobile terminal 1 is executed (S601). A connection check is performed between the mobile terminal 1 and the HMD 2 (S602).

[0119] The user selects the setting (ON / OFF) of the "peeping prevention function" (S603). The sensor 21 (acceleration sensor) of the mobile terminal 1 determines whether the user is moving or stationary (S604). If the user is moving, the process proceeds to S607. If the user is stationary in S604, the user determines the setting state of the "peeping prevention function" of the mobile terminal 1 (S605). If it is not set (S605, Off), the process proceeds to S607.

[0120] When the "peeping prevention function" is set (S605, On), the display of the operation log on the mobile terminal 1 is stopped, and the display is performed only on the HMD 2 (S606). Also, when an operation (selection) is performed to display more detailed information on the information of the operation log displayed on the HMD 2, the more detailed information is displayed only on the HMD 2. In S607, since the user is on the move and there is little possibility that people around him / her will peek at the user, and the user has not selected the anti-peek function, the operation log is displayed on the mobile terminal 1 as is, or using both the mobile terminal 1 and the HMD 2.

[0121] After that, the process returns to S604, where it is determined again whether the user is moving and the display is switched. In this manner, the display using only the HMD 2 continues until there is no longer any possibility of peeking.

[0122] It goes without saying that the above-mentioned anti-peeping function is not limited to the case where an operation log is displayed, but can be applied to the display of various types of information.

[0123] According to the third embodiment, in the cooperative display of a mobile terminal and an HMD, information that the user does not want others to see is displayed only on the HMD, preventing it from being seen by people around the mobile terminal, thereby protecting the user's privacy. [Explanation of symbols]

[0124] 1: Mobile device, 2: Head-mounted display (HMD), 10,30:Display, 11, 31: display control unit, 12,32: CPU (main control unit), 14: Data storage memory, 16,36: External device IF (interface), 17,37: Touch sensor, 20,40: Camera, 21, 41: Various sensors.

Claims

1. A mobile device that connects to a head mounted display and displays information in cooperation with the head mounted display. a first display for displaying information; a memory for storing information to be displayed on the first display; A first control unit that controls a display of the first display; a first external device interface for transmitting and receiving information to and from the head mounted display; A mobile terminal characterized in that the first control unit transmits at least a portion of the information to be displayed on the first display from the first external device interface to the head-mounted display, and controls the information to be shared and displayed on a second display provided on the head-mounted display.

2. A head mounted display that is connected to a mobile terminal and is worn on a user's head to display information in cooperation with the mobile terminal, a second display for displaying information; A second control unit that controls a display of the second display; a second external device interface for transmitting and receiving information to and from the mobile terminal; A head-mounted display characterized in that the second control unit receives at least a portion of the information displayed on the mobile terminal via the second external device interface, and shares and displays the information on the second display.

3. In a linked display system in which a mobile terminal and a head mounted display are connected and linked to display information, the mobile terminal displays information on a first display and transmits at least a portion of the displayed information to the head mounted display; A cooperative display system, characterized in that the head mounted display shares and displays information transmitted from the mobile terminal on a second display.

4. In the linked display system according to claim 3 , The information displayed on the mobile terminal is hierarchical information, The linked display system is characterized in that the information displayed on the head mounted display is lower-level information among the hierarchical information.

5. In the linked display system according to claim 3 , A linked display system, wherein the information displayed on the head mounted display is additional information to the information displayed on the mobile terminal.

6. In the linked display system according to claim 3 , the mobile terminal has a touch sensor that detects a touch operation by a user on the first display on which information is displayed, The linked display system, wherein the information displayed on the head-mounted display is detailed information about an item selected by a user, detected by the touch sensor.

7. In the linked display system according to claim 3 , the mobile terminal has a first camera that detects a line of sight of a user with respect to the first display on which information is displayed; A collaborative display system characterized in that the information displayed on the head-mounted display is detailed information about an item in the user's line of sight detected by the first camera.

8. In the linked display system according to claim 3 , the head mounted display has a second camera that detects a line of sight of a user with respect to the first display of the mobile terminal on which information is displayed, The linked display system is characterized in that the information displayed on the head-mounted display is detailed information about an item in the user's line of sight detected by the second camera.

9. In the linked display system according to claim 3 , Detecting whether the user is stationary or moving using a sensor provided in the mobile terminal; A coordinated display system characterized in that, when a user is stationary, the display of information on the first display of the mobile terminal is stopped and information is displayed only on the second display of the head-mounted display.

10. In the linked display system according to claim 3 , Detecting whether the user is stationary or moving using a sensor provided in the mobile terminal; Taking an image of the user's surroundings with a first camera of the mobile terminal or a second camera of the head mounted display; A coordinated display system characterized in that, when the user is stationary and another person is captured in the camera image around the user, the display of information on the first display of the mobile terminal is stopped and information is displayed only on the second display of the head-mounted display.

Images