Mobile terminal, head-mounted display, and their linked display system

Abstract

To activate features of a portable terminal and an HMD, and achieve user-friendly operation and an easy-to-see display form, when the respective devices display information in cooperation with each other.SOLUTION: In a portable display system for connecting a portable terminal and a head mount display, and making the devices display information in cooperation with each other, the portable terminal displays information on a first display, transmits at least a part of the displayed information to the head mount display, and the head mount display shares and displays information transmitted from the portable terminal on a second display. Here, the information displayed on the portable terminal is hierarchical information, and the information displayed on the head mount display is set to be low order information in the hierarchical information. Whether a user is at rest or moving is detected by a sensor provided on the portable terminal, and when the user is at rest, display of the information on the portable terminal is canceled, and the information is displayed only on the head mount display.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a technology for connecting a mobile terminal and a head-mounted display to display information in cooperation with each other. [Background technology]

[0002] Mobile devices such as smartphones can connect to a network and install various applications to perform a variety of functions, including recording, playing back, and displaying data. A head-mounted display (HMD) is a device worn on the user's head that displays images on a glasses- or goggle-like display. HMDs are equipped with multiple sensors, including a camera, a sensor to measure the distance to an object, and a GPS sensor to measure position, as well as a CPU for image processing and a battery. Like mobile devices, HMDs can also have applications installed and updated.

[0003] Furthermore, new functions and usage methods have been proposed by linking a mobile terminal and an HMD. For example, Patent Document 1 describes a configuration in which the screen displayed by the HMD and a terminal device (mobile information terminal) is shared, and apps are distributed from an external server device to the HMD by operating the screen of the terminal device, in order to easily manage applications (hereinafter referred to as apps) used by the HMD. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 138153 Summary of the Invention [Problem to be solved by the invention]

[0005] The use and range of applications of HMDs are expected to expand dramatically. However, because HMDs are originally worn on the head, there are limitations on the size of the screen that can be displayed, processing power, resolution, weight, etc.

[0006] Mobile devices have features such as large displays, advanced processing, and data storage, and are used in a variety of ways, for which many applications have been developed. However, because the display screen is large, there is a risk that people around you may be able to see what you are doing, so they must be used with caution from the perspective of privacy.

[0007] The above-mentioned Patent Document 1 is an example of a form in which an HMD and a mobile device are used in cooperation with each other, but the operation for acquiring an application is performed using an easy-to-operate mobile device, and sufficient consideration has not yet been given to sharing functions by taking advantage of the characteristics of each device. For example, with regard to the display of hierarchical information and privacy protection, it is assumed that the information will be displayed in the same way on both devices, and no particular consideration has been given to user usability.

[0008] An object of the present invention is to realize user-friendly operation and an easy-to-view display format by utilizing the characteristics of each device when displaying information by linking a mobile terminal and an HMD. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention employs the following method. The present invention relates to a mobile terminal that connects to a head-mounted display and displays information in cooperation with the head-mounted display, and includes a first display that displays information, a memory that stores information to be displayed on the first display, a first control unit that controls the display of the first display, and a first external device interface that transmits and receives information to and from the head-mounted display, and the first control unit controls the transmission of at least a portion of the information to be displayed on the first display from the first external device interface to the head-mounted display, and the transmission of the information to and from the head-mounted display on a second display provided in the head-mounted display.

[0010] The present invention also provides a head-mounted display that is connected to a mobile terminal, worn on a user's head, and displays information in cooperation with the mobile terminal, and includes a second display that displays the information, a second control unit that controls the display of the second display, and a second external device interface that transmits and receives information to and from the mobile terminal, and the second control unit receives at least a portion of the information to be displayed on the mobile terminal via the second external device interface and displays it on the second display in a shared manner.

[0011] The present invention also provides a linked display system in which a mobile terminal and a head-mounted display are connected and linked to display information, in which the mobile terminal displays information on a first display and transmits at least a portion of the displayed information to the head-mounted display, and the head-mounted display shares and displays the information transmitted from the mobile terminal on a second display. [Effects of the Invention]

[0012] According to the present invention, when a mobile terminal and an HMD are linked to display information, the characteristics of each device can be utilized to realize user-friendly operation and an easy-to-view display format. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram showing a cooperative display system in which a mobile terminal and an HMD are connected (Example 1). [Figure 2] A diagram showing the appearance of the HMD. [Figure 3] FIG. 2 is a diagram showing the internal configuration of a mobile terminal. [Figure 4] FIG. 2 is a diagram showing the internal configuration of an HMD. [Figure 5] FIG. 2 is a diagram showing types of data input and output between a mobile terminal and an HMD. [Figure 6] FIG. 10 is a diagram showing an example of display of an operation log on a mobile terminal. [Figure 7] FIG. 10 is a diagram showing another example of display of the operation log on the mobile terminal. [Figure 8] FIG. 10 is a diagram showing an example of hierarchical display of detailed information of an operation log. [Figure 9] FIG. 10 is a diagram showing an example of display shared between a mobile device and an HMD. [Figure 10] 10A and 10B are diagrams showing modified examples of the display on the mobile terminal. [Figure 11] 10 is a flowchart showing an operation procedure of the mobile terminal. [Figure 12] 10 is a flowchart showing the operation procedure of the HMD. [Figure 13] FIG. 10 is a diagram for explaining gaze touch using a camera of a mobile terminal (Example 2). [Figure 14] FIG. 10 is a diagram showing an example of a face image of a user captured by a camera of a mobile terminal. [Figure 15] 10 is a flowchart showing the operation procedure of gaze touch using the camera of the mobile terminal. [Figure 16] A diagram explaining gaze touch using an HMD camera. [Figure 17] FIG. 10 is a diagram showing an example of an image of a mobile device captured by a camera of an HMD. [Figure 18] FIG. 10 is a diagram showing an example of determining the viewpoint position from a terminal image captured by an HMD camera. [Figure 19] 10 is a flowchart showing the operation procedure of gaze touch using the camera of the HMD. [Figure 20] FIG. 10 is a diagram showing a configuration for automatically implementing an anti-peeping function (third embodiment). [Figure 21] 10 is a flowchart for automatically implementing the peeping prevention function. [Figure 22] 10 is a flowchart illustrating a case where a user selects and executes a peeping prevention function. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following description will discuss an embodiment of the present invention, in which a mobile terminal 1 and a head-mounted display (HMD) 2 are connected and perform display operations in cooperation with each other. [Example]

[0015] FIG. 1 is a schematic diagram showing a linked display system in which a mobile terminal 1 and an HMD 2 are connected. The HMD 2 is worn on the head of a user U and is wirelessly connected to the mobile terminal 1, operating in a linked manner while sending and receiving data and commands. The user U can view images displayed on a display 10 of the mobile terminal 1 and can also view images displayed on a display 30 of the HMD 2. A local area network (LAN) or telephone network communication may be used for the wireless connection 9 here. Other communication methods may also be used, such as Bluetooth (registered trademark) for short-range communication or an infrared communication unit, and the method may be selected according to the amount of information to be transmitted.

[0016] 2 is a diagram showing the appearance of the HMD 2. The HMD 2 worn by the user is composed of components such as a display 30, a camera 40 (and a microphone), a touch sensor 37, bone conduction earphones (audio output unit) 43, a CPU 32, and a battery 42. Details of each component will be described later.

[0017] 3 is a diagram showing the internal configuration of the mobile terminal 1. The mobile terminal 1 will be described as being a smartphone, for example, but other mobile information terminals such as tablet terminals are also applicable.

[0018] Display 10 displays various images and information, and display control unit 11 controls the display on display 10. CPU (main control unit) 12 controls the overall operation of mobile terminal 1, and memory 13 stores operation programs executed by CPU 12, etc. Data storage memory 14 stores unique data of mobile terminal 1. Device ID information 15 stores unique device information for identifying mobile terminal 1.

[0019] The external device IF 16 is an interface for inputting and outputting data to and from the HMD 2 via a wireless connection 9. Connection authentication with the HMD 2 is performed via the external device IF 16, and data is sent to the mobile terminal 1, thereby determining whether the HMD 2 is connected to the mobile terminal 1. The touch sensor 17 inputs user operations on the mobile terminal 1, and can be configured integrally with the display 10.

[0020] The camera 20 has an outer camera that captures the outside scene and an inner camera that captures the user U looking at the display 10, and a camera operation abnormality detection unit 20a detects abnormalities in the camera 20. Various sensors 21 detect the state (rotation, position, acceleration, etc.) of the portable terminal 1. The battery 22 supplies power to the portable terminal 1, and a battery capacity sensor 22a detects the remaining capacity of the battery 22. The user authentication unit 23 identifies the user of the portable terminal 1, and the external communication unit 24 communicates with an external server (such as a database) via the external network 4.

[0021] 4 is a diagram showing the internal configuration of the HMD 2. The display 30 displays an image input from the portable terminal 1, and can also superimpose image data generated within the HMD 2 and an image captured by the camera 40. A display control unit 31 controls the display on the display 30. A CPU (main control unit) 32 controls the overall operation of the HMD 2, and a memory 33 stores operation programs executed by the CPU 32, etc. Device ID information 35 stores unique device information for identifying the HMD 2.

[0022] The external device IF 36 transmits and receives data to and from the portable terminal 1 via a wireless connection 9. The touch sensor 37 inputs user operations on the HMD 2.

[0023] A camera 40 captures the scene outside the HMD 2, and a camera operation abnormality detection unit 40a detects abnormalities in the camera 40. The various sensors 41 are sensors such as gyros that measure the position and distance of objects captured by the camera 40, and measure the movement, acceleration, and rotation of the HMD 2. A sensor operation abnormality detection unit 41a detects abnormalities in the operation of the various sensors 41. A battery 42 supplies power to the HMD 2, and a battery capacity sensor 42a detects the remaining capacity of the battery 7.

[0024] The audio output unit 43 drives bone conduction type earphones, and the audio output abnormality detection unit 43 a detects an abnormality in the operation of the audio output unit 43 .

[0025] FIG. 5 is a diagram showing the types of data input and output between the portable terminal 1 and the HMD 2 via the external device IFs 16 and 36. As shown in FIG.

[0026] The data output from the portable terminal 1 shown in (a) to the HMD2 includes a unique ID for identifying the portable terminal 1, image data for displaying information on the HMD2, image data for displaying an AR (augmented reality) image that takes into account the front-to-back relationship in the depth direction using camera image data and ranging data input from the HMD2, and commands that respond to operational input from the HMD2.

[0027] The data output from the HMD2 shown in (b) to the mobile terminal 1 includes a unique ID for identifying the HMD2, image data captured by the camera 40, detection data from various sensors 41 (position, distance, acceleration, rotation, battery capacity, user authentication, etc.), and operation input data for the HMD2 (touch operation, gesture, voice input, etc.).

[0028] <Mobile device operation> Various applications operating on the mobile terminal 1 are stored in the memory 13, and a plurality of application icons that can be selected by the user are displayed on the home screen that appears at startup. Touching these displayed icons launches the corresponding application.

[0029] The mobile terminal 1 has an operation log display function that records when, where, which application, and how each device was used, and can display the log as needed. Hereinafter, an "operation log application" that displays the "operation log" operated using the mobile terminal 1 and the HMD 2 will be described as an example.

[0030] For this purpose, the "operation log application" automatically starts the operation of acquiring the operation log from the time the mobile terminal 1 is started. With this function, operation logs such as the startup time of the application for which an operation instruction was input, the application name, the operation content, the location information, and the use of the HMD 2 are stored in the data storage memory 14.

[0031] When displaying the stored operation logs, the operation log information stored in the data storage memory 14 is displayed as a list on the display 10. When an operation log displayed by the touch sensor 17 is selected, the details of the selected operation log are displayed.

[0032] Note that the operation log information stored in the data storage memory 14 is transmitted to, for example, an external application company 5a via the external communication unit 24 and the network 4 and is used for investigating the usage status of the mobile terminal 1. Regarding other applications stored in the mobile terminal 1, information such as the usage status is similarly transmitted to an external application company 5b.

[0033] <Operation of the HMD> Various data are sent from the portable terminal 1 to the HMD 2 via the external device IFs 16 and 36. This data includes operation log information, which is also displayed on the display 30 of the HMD 2. However, in this embodiment, the HMD 2 shares and displays detailed information (lower-level information and additional information in the hierarchical information) of an item selected by the user through a touch operation in the operation log displayed on the portable terminal 1. This makes it possible to realize an easy-to-view display format by taking advantage of the characteristics of each device (portable terminal 1 and HMD 2). Alternatively, as in Example 3 described below, if there is a risk that displaying the information on the portable terminal 1 may expose it to peeking by people nearby, the information is shared and displayed on the HMD 2. This has the effect of protecting the user's privacy.

[0034] FIG. 6 is a diagram showing an example of an operation log display on the mobile terminal 1. The mobile terminal 1 records user operations using an operation log app. FIG. 6 shows an example of a chronological display of operations for one day. For example, if the first operation on date: 2019 / 11 / 28 is to ring an alarm in the clock app on the mobile terminal 1, the operation log displays the following: time: 06:05, application (app): clock, operation: alarm, address, HMD2 unused (HMD icon not displayed). Next, the operation log displays the following: time: 08:30, app: SMS (Short Message Service), operation: 5 messages sent and received, HMD2 used (HMD icon displayed). The operation log displays the following sequentially in a similar manner, and the final operation of the day displays the following: time: 23:30, app: e-book app, operation: reading, HMD2 unused (HMD icon not displayed).

[0035] Here, the operation log is displayed based on the date: 2019 / 11 / 28, but you can also select and set the display of recorded information other than the date, such as the number of rows and columns to be displayed, and time.

[0036] Fig. 7 is a diagram showing another example of displaying the operation log on the mobile terminal 1. While Fig. 6 displays the operation details for one day, Fig. 7 displays the operation details for multiple days. That is, this example displays the operation logs in a format that allows past operation logs to be easily referenced by stacking multiple days' worth of operation logs, each day being one page, and turning the pages like a diary.

[0037] The operation logs shown in FIGS. 6 and 7 are information about the operation content, and can also display detailed information about the operation performed.

[0038] 8 is a diagram showing an example of hierarchical display of detailed information of the operation log. Here, an example is shown in which date, time, application, operation content, etc. are selected by touching with a finger, and detailed information about that item is displayed.

[0039] For example, suppose you touched "14 Photos" in the camera app at 17:30 on the full-screen display (a) (Touch 1). As a result, the recorded photos are displayed by type, as shown in (b) (Example display for Touch 1). Here, six categories are displayed, and the number of photos in each category is also displayed.

[0040] If you touch this displayed number information again (Touch 2), a list of photos in the touched category will be displayed, as shown in (c) (Touch 2 display example). In this way, by digging deeper into the hierarchy of the displayed content, you can get a concrete understanding of the operations you performed at that time. You can also look back on those operations later, like keeping a diary.

[0041] Up to this point, an example has been shown in which the operation log is displayed only on the mobile terminal 1, but it is also possible to transmit part of the information to the HMD 2 and share and display it on the HMD 2. The mobile terminal 1 is capable of displaying a large amount of information as shown in FIGS. 6 to 8. However, if, for example, information at a lower level is simultaneously displayed along with the operation log, the display area for each becomes smaller, making the displayed information difficult to see. Therefore, the HMD 2 is used to share and display, for example, detailed information at a lower level. Because the HMD 2 is a device mounted on the user's head, it can display smaller characters and less content than the mobile terminal 1. Therefore, of the operation log displayed on the mobile terminal 1 described in FIG. 8, the detailed information at a lower level due to the operation of Touch 1 is shared and displayed on the HMD 2.

[0042] FIG. 9 is a diagram showing an example of display shared between the mobile terminal 1 and the HMD 2. Here, the entire operation log screen is displayed on the mobile terminal 1 as shown in (a), and detailed information at a lower level is displayed on the display 30 of the HMD 2 as shown in (b). As detailed information, for example, as described above, categories of photo types are displayed as shown in FIG. 8(b), or multiple photos included in a category are displayed sequentially as thumbnails or photo images as shown in FIG. 8(c). This makes it easier for the user to view the information compared to when the information is displayed only on the mobile terminal 1.

[0043] Furthermore, the HMD 2 differs from the portable terminal 1 in the optical system, making it difficult for others in the vicinity to see the displayed content. By displaying on the HMD 2 information that should not be seen by others (for example, photos) among the information displayed on the portable terminal 1, only the user can see it, which is advantageous in terms of privacy protection. Furthermore, since the amount of information that the portable terminal 1 can display is also limited, by sharing the display information with the HMD 2, it can be used as a multiple display, improving usability.

[0044] As described above, in this embodiment, the mobile terminal 1 and the HMD 2 share the display of the operation log. The user can select display items, such as photos and SMS message contents, for the operation log displayed on the HMD 2 in the settings of the operation log app, or display detailed contents of the operation log displayed on the mobile terminal 1. Furthermore, when there are people nearby, it is also possible to have the HMD 2 display the selected items and display detailed contents of the operation log. Naturally, display on the HMD 2 is limited to when the HMD 2 is connected to the mobile terminal 1, and display data is sent from the mobile terminal 1 to the HMD 2. The operation when there are other people nearby will be described in detail in Example 3.

[0045] FIG. 10 is a diagram showing a modified example of the display on the mobile terminal 1. Compared to the display example shown in FIG. 6, only operations that used the HMD 2 are displayed in the operation log, and operations that did not use the HMD 2 are deleted. For example, the operation log at time: 06:05 (app: clock, operation: alarm), the operation log at time: 17:30 (app: camera, operation: 14 photos), and the operation log at time: 23:30 (app: e-book app, operation: reading) displayed in FIG. 6 did not use the HMD 2, and are therefore not shown in FIG. 10. Needless to say, by selecting an operation log using the HMD displayed here, the details can be displayed as shown in FIG. 8, or the details can be allocated to the HMD 2 and displayed as shown in FIG. 9.

[0046] Next, the operation flows of the mobile terminal 1 and the HMD 2 for implementing the shared display function of this embodiment will be described. 11 is a flowchart showing the operation procedure of the mobile terminal 1. The operation of the mobile terminal 1 starts (S100), an operation log application is launched, and a display function of an operation log of operations performed on the mobile terminal 1 is executed (S101).

[0047] A connection check is performed between the mobile terminal 1 and the HMD 2 (S102). To do this, the mobile terminal 1 confirms that data is being sent and received normally between the mobile terminal 1 and the HMD 2, and that the HMD 2 is being worn by the user, based on sensor information (such as acceleration information) from the HMD 2. This connection check is performed periodically or when an event occurs on the mobile terminal 1 (such as selection of a display item). If a connection error is determined in S102, communication with the HMD 2 is terminated and the connection is severed (S111).

[0048] If the connection is determined to be OK in S102, the user selects how to display the operation log: whether to display it solely on the mobile terminal 1 (examples of FIGS. 6 to 8), or whether to display it shared between the mobile terminal 1 and the HMD 2 (example of FIG. 9) (S103).

[0049] The mobile terminal 1 creates all display data of the operation log summary (the content display of the operation log explained in FIG. 6). Among the data, display data of the operation log summary to be displayed on the HMD 2 when the HMD 2 is used is also created (S104). The created all display data of the operation log summary is displayed on the display 10 of the mobile terminal 1 (S105).

[0050] When the user touches and selects an item of the operation log displayed on the display 10 for which the user wishes to display details (S106), detailed display data for that item is created (S107). For example, when the operation log is touched as shown in Fig. 8(a) (Touch 1), the data to be displayed by Touch 1 is extracted from the data storage memory 14 and created.

[0051] Next, the process branches depending on the display method of the operation log selected in S103 (S108). If the operation log is to be displayed solely on the mobile terminal 1, the process returns to S105, and the detailed display data created in S107 is displayed on the display 10 of the mobile terminal 1. Furthermore, if the user touches an item for which the user wants to display details (touch 2) in S106, detailed data to be displayed by touch 2 is created and displayed.

[0052] On the other hand, in the branch of S108, if the display is shared between the mobile terminal 1 and the HMD 2, the display destination is switched so that part of the display data is displayed on the HMD 2 (S109). Then, part of the display data is output to the HMD 2 via the external device IF 16 (S110). For example, as shown in FIG. 9, the detailed data selected by Touch 1 is output to the HMD 2. The HMD 2 receives this detailed data and displays it on the display 30.

[0053] Fig. 12 is a flowchart showing the operation procedure of the HMD 2. Here, a case where operation logs are shared and displayed on the HMD 2 as shown in Fig. 9 will be described. The operation of the HMD 2 starts (S200), and a connection check with the portable terminal 1 is performed (S201).

[0054] The camera operation abnormality detection unit 40a and the sensor operation abnormality detection unit 41a check the operation of various components (camera, sensor, display, etc.) mounted on the HMD 2 (S202). Also, the battery capacity sensor 42a checks the remaining capacity of the battery 42 of the HMD 2 (S203).

[0055] If the determination in S201 determines that there is a connection error, the occurrence of the error is displayed on the display 30 (S211). If the determination in S202 determines that there is an operation error in any of the components, or if the determination in S203 determines that the remaining battery power is low, error information is sent to the portable terminal 1 (S210), and the error information is displayed on the display 30 of the HMD 2 (S211). Note that by outputting the error information to the portable terminal 1 in S210, the HMD 2 may be instructed by the portable terminal 1 to display the error in S211.

[0056] If no abnormalities are found in the checks of S201 to S203, the check results of the HMD 2 are output to the portable terminal 1 (S204). The portable terminal 1 then confirms that the HMD 2 is worn by the user, that various operations are normal, and that the display data output from the portable terminal 1 can be displayed.

[0057] The HMD 2 acquires display data from the mobile terminal 1 via the external device IF 36 (S205). This corresponds to step S110 in FIG. 11. The acquired display data is displayed on the display 30 of the HMD 2 (S206). As a result, an image such as that shown in FIG. 9(b) (after Touch 1) is displayed, for example.

[0058] On the display screen of S206, the user selects the item (icon) they want to display next by touch operation (S207). For example, in FIG. 9(b), a touch operation of 2 is accepted. Alternatively, the user can select to end the display. The user's selection operation is performed by the touch sensor 37 of the HMD 2 or by gesture input.

[0059] Information on the item selected in S207 is output to the mobile terminal 1 via the external device IF 36 (S208). The mobile terminal 1 creates detailed data for the selected item, and the HMD 2 acquires the detailed data from the mobile terminal 1 (S209). Then, the process returns to S206, and the acquired detailed data is displayed on the display 30. As a result, a display responsive to the operation of Touch 2 in FIG. 9(b), for example, is performed.

[0060] The "operation log" used in the above explanation is hierarchical information, and is a suitable example for explaining the operation of reading information at a lower level. That is, the operation log is configured by recording instructions from a user performed on a mobile device in chronological order, along with information such as the location at the time and linked devices.

[0061] Sharing information displayed on a mobile device and an HMD is not limited to operation logs. Other examples include displaying the meaning of words you don't understand on the HMD while reading a book on a mobile device, or displaying detailed information about a specific location on a map while viewing it on a mobile device. Also, while watching a movie on a mobile device, you can use the HMD to look up blogs about characters in the movie or search for information about the geography of the background. In short, it becomes possible to share and display additional information on the HMD without interfering with the mobile device screen.

[0062] In this way, in Example 1, the mobile device and HMD cooperate to form a display screen. In this case, the HMD is treated as a child screen of the mobile device, and information that cannot be displayed on the mobile device screen or information that is hierarchically lower is displayed on the HMD. Therefore, the display content of the mobile device is maintained as is, and the content can be confirmed by looking at the mobile device, while more detailed information can be simultaneously viewed on the HMD. This makes it possible to realize user-friendly operation and an easy-to-view display format by taking advantage of the characteristics of each device. [Example]

[0063] In the first embodiment, a method was described in which the touch sensors 17, 37 of the mobile terminal 1 or the HMD 2 are used to select a corresponding item when the user wishes to display detailed content of the displayed operation log. Here, a camera is often installed in a typical mobile terminal 1 or HMD 2. In the second embodiment, a method (hereinafter referred to as "gaze touch") is described in which the camera is used to detect the user's line of sight and select an item.

[0064] 13 is a diagram illustrating gaze touch using the camera 20 of the mobile terminal 1. Gaze touch is a method of detecting which item the user's eyeball direction (gaze) is looking at and selecting that item. (a) shows a state in which the user U is looking at the display screen of the mobile terminal 1. The camera (in-camera) 20 mounted on the mobile terminal 1 is used to detect which position (which item) on the display screen the user U is looking at, that is, where the gaze is located.

[0065] (b) shows an example of the screen of the mobile terminal 1. To detect the line of sight of the user U, first, for example, five cross points (X1 to X5) are displayed one after another at predetermined intervals on the screen of the mobile terminal 1. The user looks at these displayed cross points in order according to instructions from the mobile terminal 1. The direction of the eyeballs at that time, i.e., the line of sight direction, is detected by the camera 20 of the mobile terminal 1, and the reference position and angle of the user's line of sight are found from the line of sight direction to these five cross points.

[0066] For example, the position and orientation of the face and the pupils and irises of the eyes when the user gazes at crosspoint X1 in the center of the screen are used as a reference, and the difference in the positions and orientations of the pupils and irises of the eyes when the user gazes at crosspoints X2 and X3 at the top of the screen is detected. Similarly, the difference in the positions and orientations of the pupils and irises of the eyes when the user gazes at X4 and X5 at the bottom of the screen is detected using X1 in the center of the screen as a reference. Furthermore, using the same method, the difference in position and orientation when the user gazes at the center and the left and right edges is detected, and these detected values ​​are stored in data storage memory 14.

[0067] (c) shows a state in which user U is actually gazing at a certain position on the screen. The user's gaze position (the position of the face and the positions and orientations of the pupils and irises of the eyes) captured by camera 20 is compared with the gaze position relative to the cross point, which is a reference position, to calculate the user's current gaze position. Then, a marker M indicating the gaze point of the user's gaze is displayed at the calculated position on the screen.

[0068] In gaze detection, the values ​​of the angle, position, and distance between the mobile terminal 1 and the face and eyeballs of the user U are important. When used for gaze touch, the above-mentioned reference direction of gaze needs to be corrected by checking the fluctuations (accuracy) of the angle, position, and distance between the mobile terminal 1 and the face, eyeballs, and even pupils and irises using the camera 20 and distance sensor 21 of the mobile terminal 1.

[0069] FIG. 14 is a diagram showing an example of a user's face image (user image) captured by the camera (in-camera) 20 of the mobile terminal 1. (a) shows a state in which the user U is gazing at a reference position on the screen (cross point X1 in FIG. 13). From this image, the gaze direction of the user U is determined and used as a reference value. (b) shows a state in which the user U is currently gazing at a desired position (selected position) on the screen. When the gaze position changes, the image of the user U, particularly the position and angle of the user's face and eyeballs, changes.

[0070] In this example, the face size is smaller and the distance is greater in the gaze image (b) at the selected position compared to the gaze image (a) at the reference position. The center position of the eyes is also shifted downward as indicated by the dashed line L1 → L2. The distance to the face can also be detected by the distance sensor 21, rather than by the difference between the camera images.

[0071] Here, the measurement values ​​of the positions and distances of the face and eyeballs obtained in the gaze image of the selected position (b) are used as reference values ​​for the measurement values ​​obtained when gazing at the five cross points (X1 to X5) in Fig. 13(a), and by calculating and correcting the difference between them, it is possible to calculate the position on the screen that the user is currently gazing at. In other words, it is possible to detect which position on the screen the user is gazing at using camera 20 of mobile terminal 1.

[0072] FIG. 15 is a flowchart showing the operation procedure of gaze touch using the camera 20 of the mobile terminal 1.

[0073] The operation of the mobile terminal 1 is started (S300), and the gaze touch application is started (S301). The initial setting of the gaze touch is started (S302). An image for the initial setting of the gaze touch is displayed on the display 10 of the mobile terminal 1 (S303). For example, cross points X1 to X5 as shown in FIG. 13(a) are displayed in sequence.

[0074] The user sequentially gazes at the images (cross points) displayed on the display 10 via the HMD 2 that the user is wearing (S304). The camera 20 mounted on the mobile terminal 1 captures an image of the user's eyeballs that are gazing at the screen (S305).

[0075] A reference value for the gaze direction is calculated from the information of the user image acquired when the cross point X1 is displayed, and is set as the initial position of the gaze touch (S306). The calculation results of the gaze position when gazing sequentially at each cross point X1 to X5 are saved together with information on the initial position of the gaze touch. The positional relationship (angle, position, distance) between the mobile terminal 1 and the eyeball at this time is acquired by the sensor 21 of the mobile terminal 1. Furthermore, if the HMD 2 is worn, information on the distance to the mobile terminal 1 and the position and direction of the pupil and iris of the eyeball are also acquired by the eye tracking camera (S307).

[0076] The initial position calculated in S306 and the positional relationship information between the mobile terminal 1 and the eyeball when gazing at each cross point acquired in S307 are linked to the device ID 35 of the HMD 2 and user information, and are stored in the data storage memory 14 of the mobile terminal 1 (S308).

[0077] This completes the initial setting (S309). Note that the average value of the gaze points of cross points X2 to X5 may be calculated, and if it is found to be in roughly the same position as the gaze point of cross point X1, it may be determined that each has been correctly detected.

[0078] Next, the user actually performs the gaze touch operation by gazing at a desired operation position on the screen of the portable terminal 1 (S310). The camera 20 mounted on the mobile terminal 1 captures an image of the eyeball of the user who is gazing at the image (S311).

[0079] The user's gaze direction is determined from the information on the acquired user image, and is compared with the positional relationship information at each point of gaze stored in the data storage memory 14 in S308 (to determine the difference), and the user's gaze point is calculated (S312).

[0080] The position of the gaze point, i.e., the gaze touch position, is displayed with a marker on the screen of the mobile terminal 1 (S313). If the gaze point remains at the position where the marker is displayed for a certain period of time or more, an operation similar to clicking or touching with a fingertip is performed on the position where the marker is located.

[0081] To accurately perform the above-mentioned gaze touch operation, it is necessary to constantly check the distance between the mobile terminal 1 and the face, the angle between the eyes, the position, the distance, etc., and, if any changes occur, to correct the change in gaze position due to the movement of the face or eyes from the reference point. For this reason, the angle, position, distance, etc., between the mobile terminal and the face or eyes are constantly monitored based on the initial setting position, and when a deviation of a certain amount or more occurs, the deviation is corrected based on a value calculated from the range of gaze movement obtained in the initial setting.

[0082] This eye-gaze correction involves displaying an icon that is frequently used in the application, such as the operation log end icon, in a fixed position as a landmark, and checking at a fixed frequency whether the icon is accurately understood, thereby correcting any deviations that can be corrected.

[0083] The gaze touch operation can also be performed using the camera 40 mounted on the HMD 2. This will be described below.

[0084] FIG. 16 is a diagram illustrating gaze touch using the camera 40 of the HMD 2. In order to identify the gaze position, for example, images of reference cross points X1 to X3 are displayed on the screen of the mobile terminal 1 as shown in FIG. 16. For example, point X1 is displayed in the center of the screen, point X2 in the upper left corner of the screen, and point X3 in the lower right corner of the screen. The user U gazes at these displayed cross points according to instructions from the mobile terminal 1, and takes a picture with the camera 40 of the HMD 2. The image taken by the camera 40 is then sent to the mobile terminal 1.

[0085] The mobile device 1 determines the reference center position from the acquired camera image of each cross point. For example, the position of cross point X1 in the center of the screen is compared with the average position of cross points X2 and X3 in the corners of the screen, and if they are roughly the same position, it is set as the reference position for gaze touch. If the HMD 2 is equipped with an eye-tracking camera to detect the position and angle of the pupil and iris of the eyeball, that information can be used. Note that when detecting gaze, the angle, position, and distance between the mobile device 1 and the HMD 2, as well as the position and angle of the HMD 2 itself relative to the eyeball (position and angle when mounted on the head) are important.

[0086] 17 is a diagram showing an example of an image of the mobile terminal 1 captured by the camera 40 of the HMD 2. From the image of the mobile terminal 1 captured by the camera 40 of the HMD 2 (herein referred to as the "terminal image"), the gaze point of the user U on the screen of the mobile terminal 1 is detected. In adjusting the initial position, if the image is captured as in the terminal image 1 of (a), the entire screen of the mobile terminal 1 is visible, and information for detecting which point the user is gazing at can be obtained, which is appropriate.

[0087] However, for example, if the distance between the mobile terminal 1 and the HMD 2 is too great, the entire mobile terminal 1 will be captured as a small image, as shown in (b) of the terminal image 2. This will cause an error in detecting the gaze point, making it difficult to accurately detect the gaze point. In this case, an instruction is issued for the HMD 2 to move closer to the mobile terminal 1.

[0088] In the case of terminal image 3 in (c), the center of the image has shifted to the upper left of mobile terminal 1, so the user is instructed to move it to the lower right. Conversely, in the case of terminal image 4 in (d), the center of the image has shifted to the lower right of mobile terminal 1, so the user is instructed to move it to the upper left.

[0089] The terminal image 5 in (e) captures the entire screen of the mobile terminal 1 to the maximum extent possible, indicating that it is in a position where it cannot be approached any closer. In this way, the mobile terminal 1 is adjusted so that it is photographed in the state of the terminal image 1 in (a).

[0090] As in the case of using the camera 20 of the mobile terminal 1 described above, it is necessary that the angle, position, distance, etc. between the mobile terminal 1 and the HMD 2 are within a certain range. Therefore, it is necessary to constantly check the changes in the angle and position between the mobile terminal 1 and the HMD 2 and make corrections to the reference position of the line of sight set above.

[0091] 18A and 18B are diagrams showing an example of determining the viewpoint position from a terminal image captured by the camera 40 of the HMD 2. (a) shows a case where the reference position of the screen of the mobile terminal 1 is determined, and (b) shows a case where the user's viewpoint position is determined from an actually acquired terminal image.

[0092] In the device image of (a), the distance and angle of the HMD 2 relative to the screen of the mobile device 1 are detected using three cross droppers X1 to X3 to determine the reference position. The difference between these values ​​and the device image of (b) is then detected. In this example, the center of the screen (cross point X1) of the device image of (b) is in the center of the range captured by the HMD 2, so it can be seen that the mobile device 1 is almost directly in front of the HMD 2. Furthermore, because the screen of the mobile device 1 is captured larger, it can be seen that the distance to the HMD 2 is closer than when the device image of (a) was captured.

[0093] Here, if the screen of the mobile device 1 is deformed into a trapezoid or diamond, it is because the mobile device 1 is being held at an angle. In such a case, the state of tilt can be detected by detecting the position information of the four corners of the screen of the mobile device 1. Then, it becomes possible to detect where on the screen of the mobile device 1 the gaze is being directed in (b) relative to the gaze point position determined by setting the reference position in (a). Note that if the HMD 2 is equipped with an eye-tracking camera and is capable of measuring the position and orientation of the pupil and iris of the eyeball, the accuracy of determining the gaze point can be improved by using these as parameters.

[0094] FIG. 19 is a flowchart showing the operation procedure of eye-gaze touch using the camera 40 of the HMD 2.

[0095] The operation of the HMD 2 is started (S400), the gaze touch application is started, and a command to start the application is sent to the mobile terminal 1 (S401). The initial setting of the gaze touch is started (S402). The mobile terminal 1 displays an image (cross points) for the initial setting of the gaze touch on its display 10 (S403). For example, cross points X1 to X3 as shown in FIG. 16 are displayed.

[0096] The user gazes at the image displayed on the mobile terminal 1 via the HMD 2 (S404). An image of the mobile terminal 1 that the user is gazing at is taken by the camera 40 mounted on the HMD 2 and sent to the mobile terminal 1 via the external device IF 36. The mobile terminal 1 acquires the image of the mobile terminal taken by the HMD 2 (terminal image) (S405).

[0097] The mobile terminal 1 calculates the position and angle of the HMD 2 from the position of the displayed image and the terminal image captured by the HMD 2, and sets the reference position for eye-gaze touch (S406). At this time, if the mobile terminal 1 and HMD 2 are too far apart or at too oblique an angle, the system instructs the user to correct the position of the mobile terminal 1. Then, the system acquires information on the angle and position between the mobile terminal 1 and HMD 2, as well as information on the angle of the pupil and iris of the eyeball using an eye-tracking camera mounted on the HMD 2 (S407).

[0098] The initial position calculated in S406, the angle between the mobile terminal 1 and the HMD 2 and the position information acquired in S407, are linked with the device ID 35 of the HMD 2 and user information, and are stored in the data storage memory 14 (S408). This completes the initial setting (S409).

[0099] Next, the user actually performs a gaze touch operation. The user gazes at a desired operation position on the screen of the portable terminal 1 via the HMD 2 (S410). The camera 40 mounted on the HMD 2 captures an image of the mobile terminal 1 being gazed at (terminal image) and sends it to the mobile terminal (S411).

[0100] The mobile terminal 1 compares the acquired terminal image with the angle and position information of the HMD 2 stored in the data storage memory 14 in S408, and calculates where on the screen of the mobile terminal 1 the gaze is being directed (S412).

[0101] The position of the gaze point, i.e., the gaze touch position, is displayed with a marker on the screen of the mobile terminal 1 (S413). If the gaze point remains at the position where the marker is displayed for a certain period of time or more, an operation similar to clicking or touching with a fingertip is performed on the position where the marker is located.

[0102] Based on the initial setting position, the system constantly monitors the distance and angle between the mobile device and the HMD, as well as the angle of the HMD itself when worn by the user, and corrects any deviations that exceed a certain level.

[0103] In the above example, the image captured by the camera 40 of the HMD2 is sent to the portable terminal 1, and the distance and angle are calculated on the portable terminal 1 side, but this is not limited to this, and these processes can also be performed on the HMD2 side or on an external server via a network.

[0104] According to the second embodiment, a user can select a desired display item and display detailed information about the selected item simply by gazing at the display screen of the mobile device, thereby improving the convenience of user operations for switching the display content. [Example]

[0105] In the first embodiment, a part of the detailed contents of the operation log displayed on the mobile terminal 1 is displayed on the HMD 2. In some cases, the display of the operation log on the mobile terminal 1 may not be desired to be seen by others from the viewpoint of privacy. In this respect, the HMD 2 has the advantage that, due to the difference in the optical system, the image is not focused on the lenses of the glasses, which are the display surface, making it difficult for people around to see what is being displayed. Therefore, in the third embodiment, a configuration will be described in which privacy is taken into consideration and the HMD 2 is actively used for display.

[0106] In this embodiment, the acquired operation log information is displayed by switching between the mobile terminal 1 and the HMD 2 as appropriate. In order to prevent the displayed operation log from being seen by people in the vicinity, an "anti-peeping function" is added. With the "anti-peeping function", in an environment where the operation log is likely to be peeked at or when it is not desired to show the operation log, the operation log information is hidden on the mobile terminal 1 and displayed on the HMD 2. The "anti-peeping function" may be implemented automatically or may be implemented by the user.

[0107] First, a case where the peeping prevention function is automatically implemented will be described. 20 is a diagram illustrating a configuration for automatically implementing the anti-peeping function. A user U is wearing an HMD 2 and looking at the display screen of a portable terminal 1. There is a possibility that a third person Un is present around the user U and is peeking at the screen of the portable terminal 1.

[0108] In such a case, the camera 20 (which is an internal camera that takes pictures of the user U) mounted on the mobile terminal 1 captures an image of the user's surroundings when the mobile terminal 1 is in use. If a person Un other than the user U appears in the captured camera image C1, it is determined that there is a possibility of peeping. In this case, the camera 40 of the HMD 2 may be used to capture an image in front of the user, and it may be detected whether a person Un who may be peeping is present in the captured camera image C2.

[0109] The key point here is that it is desirable for the photographing cameras 20 and 40 to be equipped with wide-angle lenses. The degree of wide-angle is required to be equal to or greater than the viewing angle that allows the display 10 of the mobile terminal 1 to be seen from all around.

[0110] In addition, the direction of the face of a person Un other than the photographed user U is detected, and the angle with the display 10 of the mobile terminal 1 is detected. If the angle is greater than the viewing angle of the display 10, it can be determined that there is little possibility of peeping.

[0111] It is possible to use facial recognition or the like to identify the user U appearing in the camera images C1 and C2. Also, when the peeping prevention function is activated, it is important to monitor by operating the cameras 20 and 40 constantly or at regular intervals.

[0112] Operating the cameras 20 and 40 constantly or continuously at regular intervals may result in unintended camera photography, and conversely, it is necessary to consider the privacy of others. Therefore, it is necessary to clearly indicate on the mobile terminal 1 that the camera is operating or that the anti-peeping function is operating so that those around you are aware of it. For example, it is desirable to clearly indicate this by turning on a lamp on the mobile terminal 1.

[0113] FIG. 21 shows a flowchart for automatically implementing the anti-peeping function. The operation of the mobile terminal 1 is started (S500), an operation log application is launched, and a display function of the operation log operated on the mobile terminal 1 is executed (S501). The user is authenticated (S502), and the connection with the HMD 2 is checked (S503). If a connection error is detected, communication with the HMD 2 is terminated and the connection is severed.

[0114] It is determined whether the user is moving or not based on acceleration information from the sensor 21 of the portable terminal 1 (S504). If the user is moving, the process proceeds to S509, and the image is displayed using the screen of the portable terminal 1 since there is little possibility of peeking. If it is determined in S504 that the user is stationary, there is a possibility that the user may be peeped at, and the "peeping prevention function" is executed (S505).

[0115] An image of the user's surroundings is captured by the camera 20 of the mobile terminal 1. Alternatively, an image of the surroundings of the mobile terminal 1 may be captured by the camera 40 of the HMD 2 (S506). The presence or absence of people other than the user is determined from the captured camera image, and the possibility of peeping is further determined from the angle between the captured person and the display 10 of the mobile terminal 1 (S507).

[0116] If it is determined in S507 that there is a possibility of peeping, the display of the operation log on the mobile terminal 1 is stopped and the display is performed only on the HMD 2 (S508). Also, if an operation (selection) is performed to display more detailed information on the information in the operation log displayed on the HMD 2, the more detailed information is displayed only on the HMD 2. On the other hand, if it is determined in S507 that there is no possibility of peeping, the operation log is displayed on the portable terminal 1 as is, or using both the portable terminal 1 and the HMD 2 (S509).

[0117] Then, the process returns to S504 to determine again whether the user is moving, and in S507, it is determined whether there is a possibility of peeping, and the display is switched. In this way, the display using only the HMD2 continues until there is no longer a possibility of peeping.

[0118] Next, a case where the user selects and executes the peeping prevention function will be described. FIG. 22 shows a flowchart when the user selects and executes the anti-peeping function. The operation of the mobile terminal 1 is started (S600), an operation log application is launched, and a display function of an operation log of operations performed on the mobile terminal 1 is executed (S601). A connection check is performed between the mobile terminal 1 and the HMD 2 (S602).

[0119] The user selects the setting (ON / OFF) of the "peeping prevention function" (S603). The sensor 21 (acceleration sensor) of the mobile terminal 1 determines whether the user is moving or stationary (S604). If the user is moving, the process proceeds to S607. If the user is stationary in S604, the setting status of the "peeping prevention function" of the mobile terminal 1 is determined (S605). If it is not set (S605, Off), the process proceeds to S607.

[0120] When the "peeping prevention function" is set (S605, On), the display of the operation log on the mobile terminal 1 is stopped and the display is performed only on the HMD 2 (S606). Also, if an operation (selection) is performed to display more detailed information about the operation log information displayed on the HMD 2, the more detailed information is displayed only on the HMD 2. In S607, since the user is on the move and there is little possibility that people around him / her will peek at the user, and the user has not selected the anti-peeping function, the operation log is displayed on the mobile terminal 1 as is, or using both the mobile terminal 1 and the HMD 2.

[0121] After that, the process returns to S604, where it is determined again whether the user is moving and the display is switched. In this way, the display using only the HMD2 continues until there is no longer any possibility of peeking.

[0122] It goes without saying that the above-described peeping prevention function is not limited to the case where an operation log is displayed, but can be applied to the display of various types of information.

[0123] According to the third embodiment, in the linked display of the mobile terminal and the HMD, information that the user does not want others to see is displayed only on the HMD, which prevents people around the mobile terminal from peeking at the information, thereby protecting the user's privacy. [Explanation of symbols]

[0124] 1: Mobile device, 2: Head-mounted display (HMD), 10,30:Display, 11, 31: Display control unit, 12,32: CPU (main control unit), 14: Data storage memory, 16,36: External device IF (interface), 17,37: Touch sensor, 20,40: camera, 21, 41: Various sensors.

Claims

1. A mobile terminal that displays information in cooperation with a head-mounted display connected thereto, The display and a memory for storing information to be displayed on the display; an external device interface that transmits and receives information to and from the head-mounted display; a sensor for detecting a state of the mobile terminal; A control unit; Equipped with The control unit determining whether the user of the mobile terminal is moving based on the detection result from the sensor; A mobile terminal characterized in that, when it is determined that the user is moving, the information stored in the memory is displayed on the display.

2. 2. The mobile terminal according to claim 1, A mobile terminal characterized in that, when the control unit determines that the user is not moving, it stops displaying the information on the display and transmits at least a portion of the information displayed on the display to the head-mounted display via the external device interface.

3. 2. The mobile terminal according to claim 1, Further equipped with a camera, The control unit determining whether or not a person other than the user is included in an image of the user's surroundings captured by the camera; A mobile terminal characterized in that, when it determines that the user is not moving and that the image contains a person other than the user, it stops displaying the information on the display and transmits at least a portion of the information displayed on the display to the head-mounted display via the external device interface.

4. 2. The mobile terminal according to claim 1, A mobile terminal characterized in that, when the control unit determines that the user is not moving, it stops displaying the information on the display and transmits additional information to the information displayed on the display to the head-mounted display via the external device interface.

5. 2. The mobile terminal according to claim 1, Further comprising a touch sensor, The control unit detecting, by the touch sensor, an item of information displayed on the display that is selected by a touch operation by the user; A mobile terminal characterized in that, when it is determined that the user is not moving, the display of the information on the display is stopped and detailed information of the detected item is transmitted to the head-mounted display via the external device interface.

6. 2. The mobile terminal according to claim 1, Further equipped with a camera, The control unit detecting an item of information displayed on the display in the direction of the user's line of sight using the camera; A mobile terminal characterized in that, when it is determined that the user is not moving, the display of the information on the display is stopped and detailed information of the detected item is transmitted to the head-mounted display via the external device interface.

Images