Display device and control method for display device

Abstract

To provide a technique for cooperation of a display unit and a portable information terminal that displays a screen of the portable information terminal on the display unit at a timing useful for a user and provides good operability.SOLUTION: A display unit and a portable information terminal comprise the same application software. A control unit of the display unit, when the portable information terminal is not included in a display area of a display of the display unit, performs control to display a display screen of the portable information terminal on the display, and controls the application software included in the display unit to be executed in synchronization with the application software of the portable information terminal. When the portable information terminal is made to be included in the display area, the control unit performs control to end the display of the display screen on the display and store an end position where the display screen has been displayed on the display. When the portable information terminal is no longer included in the display area, the control unit performs control to resume the display of the display screen at the stored end position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a display device and a control method for the display device, and relates to a display technology for a head mounted display (hereinafter abbreviated as "HMD"), as an example. [Background technology]

[0002] As a background art of this technical field, Patent Document 1 states, " A camera image and a gaze point of a person wearing a head mounted display with respect to the camera image. From the primary information, the position information of the wearer's gaze point relative to the primary information is calculated, and the position information related to this position information is Select and display secondary information, and when the wearer is viewing primary information, The system changes the way secondary information is displayed depending on whether or not the user is looking at the page (summary excerpt) It has been disclosed.

[0003] In addition, Patent Document 2 states that "a head-mounted display device that enables a user to view a virtual image and an external scene is A first image is a display image of an external device connected to the wearable display device, and a second image is a display image of the head-mounted display device. a generating unit for generating a list image including a second image held by the display device; and an image display unit that forms a virtual image representing the image (abstract excerpt)." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 104089 [Patent Document 2] JP 2015-222891 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Documents 1 and 2, when a head mounted display (hereinafter abbreviated as "HMD") and an external display device are connected for communication, a display image is shared with the HMD. Therefore, when the above prior art is applied to a portable information terminal and HMD having a communication function, when a link between the HMD and the portable information terminal is established, the display screen of the portable information terminal is displayed on the HMD, so that the screen of the portable information terminal may be displayed on the HMD even when the user does not expect it or does not need it, which is inconvenient in use.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and provides a method for a user to carry a portable device at a time when it is useful to the user. The present invention aims to provide a display device that displays the screen of a portable information terminal on the display device and has good operability. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention has the configurations described in the claims. As an example, a display device includes a display unit, a camera, a communication module for communicating with a mobile information terminal, and a control unit, the display device and the mobile information terminal are provided with the same application software, and the display unit: The device has a plurality of sides forming an outer edge and a plurality of regions adjacent to each of the plurality of sides, The control unit displays an image captured by the camera on the display unit, and when the image does not include the mobile information terminal, controls the display screen of the mobile information terminal to be displayed at a predetermined position on the display unit, controls the application software provided in the display device to be executed in synchronization with the application software of the mobile information terminal, and when the image includes the mobile information terminal, ends display of the display screen on the display unit, and when the image no longer includes the mobile information terminal and display of the display screen is resumed, the mobile information terminal The first edge of one of several edges If the subject is no longer included in the image due to being out of frame, One of the regions adjacent to the first side The display screen is displayed in a first area, and the mobile information terminal a second side that is one of the plurality of sides and is different from the first side If the subject is no longer included in the image due to being out of frame, One of the plurality of regions adjacent to the second sideThe display device is characterized in that it controls the display screen to be displayed in the second area. Effect of the Invention

[0008] According to the present invention, it is possible to provide a display device that displays a screen of a mobile information terminal on a display device at a timing useful to a user, and has good operability. Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief description of the drawings]

[0009] [Figure 1] Overview of the HMD linkage system [Diagram 2] HMD hardware configuration diagram [Diagram 3] HMD software configuration diagram [Figure 4] Hardware configuration of a mobile information terminal [Diagram 5] Software configuration diagram of mobile information terminal [Figure 6] FIG. 1 is a diagram showing an example of a sequence of an HMD-linked system according to a first embodiment; [Figure 7] A flowchart showing the flow of an application screen display control process according to the first embodiment. [Figure 8] A diagram showing the positional relationship between the visible area and the mobile information terminal (application screen not shown) [Figure 9] A diagram showing the scene through the display when the app screen is not displayed [Figure 10] A diagram showing the positional relationship between the visual recognition area and the mobile information terminal (app screen display) [Figure 11] A diagram showing the scene through the display when the app screen is displayed [Figure 12] FIG. 13 is a diagram showing variations of an application screen display start condition according to the second embodiment; [Figure 13] A diagram showing the redisplay position after framing out [Figure 14] FIG. 13 is a diagram showing a sequence example of an HMD-linked system according to the fourth embodiment. [Figure 15]Diagram showing the screen transitions of the HMD DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The same reference numerals are used to avoid repeated explanation.

[0011] First Embodiment In the first embodiment, the HMD and a smartphone as a portable information terminal are communicatively connected, and the HM When the mobile information device is not within the visual range of the user wearing the D, In this embodiment, the pre-screen is displayed on the HMD. This refers to the screen that displays the execution status of application software running on a terminal.

[0012] FIG. 1 is a schematic diagram of an HMD linkage system 100. The HMD linkage system 100 includes: The HMD 10 and the portable information terminal 80 are connected via wireless communication. Although a smartphone is shown, a tablet or smart watch may also be used. Any type of image display device may be used as long as it is possible.

[0013] The HMD 10 mainly includes a wearable body 1, a display 2, an outer camera 3, an inner camera 4, and an HM Equipped with D control device 5.

[0014] The wearing body 1 is a device in which the HMD 10 is worn on the head of a user M (see Figs. 8 and 10) of the HMD 10. The component that holds the other components of the HMD 10 and maintains the eye contact. Although a mirror shape is illustrated, the shape is not limited to this.

[0015] The display 2 is a transparent liquid crystal display, and the user of the wearing body 1 It is placed in front of M's eyes.

[0016] The outer camera 3 is configured so that the side of the attachment 1 facing the face of the user M is the inner side, and the opposite side is the outer side. When the front side is defined as the front side, the camera equipped with a CCD or CMOS mounted on the outer front side of the wearable body 1 is The outer camera 3 is attached to the wearable body 1 at a position and orientation that includes the user M's visual recognition area in its angle of view. It will be established.

[0017] The inner camera 4 is set inside the wearable body 1 at a position and orientation that allows the user M to capture the eye movement of the user M. These cameras are equipped with CCD or CMOS sensors.

[0018] The HMD control device 5 controls communication with the mobile information terminal 80 and displays on the display 2. In this embodiment, a general-purpose computer is used. A configuration for executing software that realizes the functions of the HMD linking method according to this embodiment using a computer. This will be explained using the composition as an example.

[0019] FIG. 2 is a diagram showing the hardware configuration of the HMD 10.

[0020] As shown in FIG. 2, the HMD 10 includes an HMD control device 5, an imaging unit 120, an audio unit 130, Operation unit 140, communication module 150, sensor unit 160, expansion I / F 170, display unit 18 0 are connected to each other via a system bus 102.

[0021] The HMD control device 5 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a Memory) 103, RAM (Random Access Memory) 104, HDD (Hard Disk Drive) A storage device consisting of a solid-state drive (SSD) equipped with semiconductor memory such as flash memory. The storage 110 is configured to include:

[0022] The imaging unit 120 converts the output signals of the outer camera 3 and the inner camera 4 into image signals. and an image signal processing unit 121 for converting the image signal.

[0023] The audio unit 130 includes a microphone 131 for inputting audio, and a noise filter 132 for filtering out noise from the audio signal from the microphone 131. A sound signal processor 132, which includes processing and amplification processing, and a speaker 13 for playing ringtones and sounds. Includes 3.

[0024] The operation unit 140 is made up of operation buttons and switches, and is used to turn the main power of the HMD 10 ON / OFF. It is composed of operation members such as FF buttons and volume dials. Each operation member is It is a component that receives input operations and detects input operations, so it can be used as an operation detection sensor. It has the following functions.

[0025] The communication module 150 is connected to a wireless LAN such as Wi-Fi (registered trademark) or infrared communication. and a wireless LAN communication unit 151 for communicating with a nearby and a wireless communication unit 152.

[0026] The sensor unit 160 includes a GPS (Global Positioning System) 161, which detects changes in rotation and orientation. A gyro sensor 162 for detecting, a geomagnetic sensor 163 for measuring the direction, and a tilt and movement , an acceleration sensor 164 for detecting data related to vibration and impact, and brightness around the HMD 10 A light sensor 165, a proximity sensor 166, a distance sensor 167, and a gaze detection sensor The gyro sensor 162, the acceleration sensor 164, and the line of sight detection sensor 168 are In each embodiment, it is used as a visual recognition area detection sensor that detects the visual recognition area of ​​the user M. .

[0027] The display unit 180 controls the display 2 and generates and outputs display data to the display 2. and a display control unit 182 that controls the display.

[0028] FIG. 3 is a diagram showing the software configuration of the HMD 10.

[0029] The ROM 103 stores a basic operation program 103a for the HMD 10 and other programs 1 The basic operation program 103a includes a startup program for the HMD 10 and a normal Display control of the top screen (see top screen 1501 in FIG. 15) that is displayed during the startup process The program also includes a program for communication with the portable information terminal 80.

[0030] The RAM 104 executes the programs stored in the ROM 103 and the storage 110. 01 is the area that is loaded at run time. During execution of each program, RAM 104 basically contains An action execution section 104a, an information display execution section 104b, and a temporary storage area 104c are generated. The information display execution unit 104b displays or hides the application screen of the portable information terminal 80 on the HMD 10. It includes a screen movement middle 104b1 for displaying, and an operation execution part 104b2.

[0031] The screen movement middle 104b1 controls the display of the application screen on the display 2 of the HMD 10. In this embodiment, it is configured as middleware, but it is also a program that runs on the OS. Alternatively, the screen change middleware 104b1 may be replaced by a mobile phone or a driver. Applications that run on the HMD 10 and correspond to the applications of the portable information terminal 80 It may also be configured by the above.

[0032] The storage 110 includes a storage area 110a for storing various information data, an information display program 110b, and other programs 110c. The screen movement flag (see FIG. 7) described later is The various information data may be stored in the various information data storage area 110a or in the temporary storage area 10 of the RAM 104. 4c. In addition, the display 2 may be divided into a number of small areas for management purposes. Small area management data is also stored in the various information data storage area 110a.

[0033] 4 is a diagram showing the hardware configuration of the mobile information terminal 80. The same components as those in the HMD 10 are represented by the same names and are illustrated using the reference numerals 201 to 282. The difference from the hardware configuration of the HMD 10 is that the gaze detection sensor 168 is not included. The terminal communication module 250 is a telephone network communication module for connecting to 3G and 4G lines. The inner camera 224 in the portable information terminal 80 is a The portable information terminal 80 is provided with a terminal display 281 on the same surface (front surface). The external camera 223 is a camera mounted on the back of the smartphone. It's Mera.

[0034] 5 is a diagram showing the software configuration of the portable information terminal 80. The same components are represented by the same names and are illustrated using the reference numerals 201 to 210c. The difference from the software configuration of is that it does not have the screen movement middleware 104b1 and the HMD1 Mobility-enabled app is software that works with HMD10 to display app screens on the HMD10. The mobile application 204d is installed in the mobile information terminal 80. Of the individual application software installed, display the application screen on the HMD10 It is an application software with functions such as map navigation applications and e-book applications. In this embodiment, the function of the mobile application 204 is not limited. d, but each app does not respond to screen movement, but the mobile information terminal 8 0, and a screen movement middleware is executed on the terminal display 281 of the portable information terminal 80. The screen displayed on the display screen may be displayed on the HMD 10.

[0035] In addition, the mobile application 204d is adapted to the display capabilities and usage schedule of the mobile information terminal 80 and the HMD 10. Depending on the application, high resolution and low resolution displays may be prepared, or detailed and simplified displays may be prepared. This is also fine.

[0036] The HMD 10 and the mobile information terminal 80 may be connected via close-proximity wireless communication or wireless LAN. Communication can also be peer-to-peer, one-to-one, or via the cloud.

[0037] As a mode of moving the application screen from the portable information terminal 80 to the HMD 10, The application screen displayed on the terminal display 281 of the terminal 80 may be captured and transmitted. In this case, the still image will move. If you are taking a screen capture, refresh the screen periodically. In addition, both the mobile information terminal 80 and the HMD 10 may use the mobile application 204d. The screen data is transferred by storing the screen data in the mobile application 204d and linking the operation of the mobile application 204d. Instead, the mobile application 204d stored in the HMD 10 and the mobile information terminal 80 The mobile application 204d is synchronized with the mobile application 204a, and the same content as that displayed on the terminal display 281 is displayed. The screen may be displayed on the display 2 of the HMD 10. The moving image display may be moved from the virtual reality display to the HMD 10 by means of streaming or the like.

[0038] An overview of the processing of the HMD linkage system 100 will be described with reference to FIGS. 7 is a flowchart showing a process flow of the HMD linkage system 100. FIG. 8 is a flowchart showing the flow of an application screen display control process according to the embodiment. 9 is a diagram showing the positional relationship with the mobile information terminal 80 (application screen not shown). The figure shows the scene through the display 2 when it is not displayed. 11 is a diagram showing the positional relationship with 0 (application screen display). The scene is shown through the display 2. The following will explain the steps in FIG.

[0039] The mobile information terminal 80 and the HMD 10 are both started up (S01, S02). Either the mobile information terminal 80 may be started first, or they may be started simultaneously. In the following description, it is assumed that the portable information terminal 80 is started up first.

[0040] A connection request is sent from the portable information terminal 80 to the HMD 10 (S03), and a link is established. (S04).

[0041] After the link is established, application software stored in the portable information terminal 80 is The mobility-enabled application 204d of the information terminal 80 is started (S05). An application start notification indicating that the mobility-enabled application 204d has started is transmitted to the HMD 10. (S06).

[0042] The HMD control device 5 that has received the application start notification Then, the device 104b1 is started (S07).

[0043] The screen movement middle 104b1 starts capturing images with the outer camera 3 of the HMD 10. The outer camera 3 starts outputting an image to the HMD control device 5 (S08). Since the HMD 10 includes the visual recognition area of ​​the user, in this description, the imaging area of ​​the outer camera 3 is HM It is assumed to be the visible range of user M of D10.

[0044] The HMD control device 5 performs a process of displaying an application screen on the display 2 (application screen display In the first embodiment, the portable information terminal is included in the image of the outer camera 3 (S09). Whether or not the end 80 is in the picture is used as a condition (trigger) for starting to display the app screen.

[0045] As shown in FIG. 7, the screen movement application 104d1 captures an image from the outer camera 3 (S 091), a display device that recognizes whether a subject (object) of an image is a mobile information terminal 80 or not Image recognition processing (S092) is performed. There are several specific algorithms for image recognition processing. However, for example, the front of the mobile information terminal 80 is imaged in advance by the outer camera 3, and the image obtained by this image is The front image is stored as training data. Then, the subject is compared with the image from the outer camera 3. Extraction processing is performed, and pattern matching processing is performed between the subject and the training data. If the number of pixels is equal to or larger than a threshold value set for the front screen, the mobile information terminal 80 may be determined to be a subject. By using the image, the mobile information terminal 80 may not be captured upside down or sideways in the image captured by the outer camera 3. Also, if the mobile information terminal 80 is placed in the shade and partially obscured, the subject is not considered to be the mobile information terminal 80. It is also possible to execute the process without recognition.

[0046] If it is determined that the subject is the mobile information terminal 80 (S093 / Yes), that is, as shown in FIG. When it is determined that the portable information terminal 80 is within the visual recognition area, the HMD control device 5 performs screen movement. Set the flag value to "0" (S094) to not display the application screen on display 2. In this case, as shown in FIG. 9, the user M can see through the display 2 of the HMD 10. The user M can see the mobile information terminal 80. An application displayed on the terminal display 281 directly via the display 2, not on a screen. Check the screen.

[0047] If it is determined that the subject is not the mobile information terminal 80 (S093 / No), that is, as shown in FIG. When it is determined that the portable information terminal 80 is not within the visual recognition area, the HMD control device 5 performs screen movement. Set the flag value to "1" (S095) and display the app screen on display 2. In this case, as shown in FIG. 11, the user M can select the application screen displayed on the display 2. Visually confirm.

[0048] While the application screen is being displayed on the display 2 of the HMD 10, The application screen of the ray 281 may be hidden or may continue to be displayed. The mobile information terminal 80 displays or hides the application screen, or switches from display to sleep mode (hidden mode). The process of transitioning to a screen display (operation of displaying the screen and dimming the screen) is performed by the display control unit of the mobile information terminal 80. The screen control function may be used to execute the screen control. Alternatively, the mobile application 204d may have a screen control function. This is also fine.

[0049] When the screen movement middle 104b1 satisfies the termination condition of the application screen control process (S096 / Yes) to end the application screen control process. If the conditions for ending the application screen control process are not met, If not (S096 / No), the process returns to S091 and the image capture from the outer camera 3 is repeated. This allows user M to continue using the HMD 10 unless the termination condition is met while the link is established. When the user puts on the mobile information terminal 80 and looks at it, the user can see the actual application screen of the mobile information terminal 80. When the user removes the information terminal 80 from the visual range, for example, when the user lowers the hand holding the portable information terminal 80 or When the portable information terminal 80 is placed in a pocket or a bag, the application screen is displayed on the HMD 10. can be.

[0050] When the end condition of the application screen control process is satisfied, the link of the HMD 10 and the mobile information terminal 80 Alternatively, the link may be maintained and only the screen movement middleware 104b1 may be terminated. As a result, the start notification of the mobility-enabled application 204d is sent to the HMD 10 again. You can immediately start moving between app screens.

[0051] The termination condition is, for example, to terminate the screen movement middleware 104b1 or the movement-compatible application 204d. It may be that an operation to turn off the mobile information terminal 80 or the HMD 10 is detected, or that the main power of the mobile information terminal 80 or the HMD 10 is turned off. It may also be that an operation of detecting a user's voice is detected.

[0052] In FIG. 6, the mobile application 204d of the mobile information terminal 80 is the master, and the screen migration middle 104 Although b1 was explained as a slave, the screen movement middleware 104b1 is the master, and the movement-enabled application 204d may be a slave. Or, a communication control method in which both are masters may be used. That is, the screen transfer middle 104b1 may send a connection request to the mobile information terminal 80. After the link is established, a notification of the start of the mobile compatible application 204d is sent from the mobile information terminal 80 to the HMD 1. You may send it to 0.

[0053] <Second embodiment> The second embodiment differs from the first embodiment in the display start condition of the application screen. So, as a display start condition, whether the mobile information terminal 80 is captured in the image of the outer camera 3 or not, First, whether or not the mobile information terminal 80 is included in the visual recognition area of ​​the user M is used as a condition. However, the display start condition is not limited to this. FIG. 12 shows the display start condition of the application screen according to the second embodiment. FIG. 13 is a diagram showing variations of conditions.

[0054] In the second embodiment, the following is used as the application screen display start condition.

[0055] Setting conditions: The app screen is set to be automatically displayed on display 2 of the HMD10. (S0911). If the setting is set to automatically display, is the app screen visible? If the application is not included in the area, the application screen may be displayed on the display 2.

[0056] Operation conditions: HMD10 main unit button operation, remote control operation, gestures, etc. Has the operation detection sensor detected that a user operation has been performed by the user? (S0912) When the operation is detected, the application screen may be displayed on the display 2.

[0057] Body movement conditions: outputs from the gyro sensor 162 and acceleration sensor 164 of the HMD 10, and The movement of the HMD 10 is detected using the images output from the side camera 3 and the inner camera 4. The movement of the user's neck (head) was detected by simulating the movement of the user's neck (head). (S0913). The body movement condition is that when neck movement is detected in the non-display state, the display transitions to the display state. The display start condition is explained as "the condition for starting the display." However, the neck movement while the app screen is displayed When detecting, the user may move their head to look at the mobile information terminal 80. The HMD control device 5 may use the gyro sensor as a condition for transitioning from display to non-display. The angular velocity included in the output from the acceleration sensor 162 and the acceleration included in the output from the acceleration sensor 164 If the degree of movement is greater than or equal to a certain threshold set to determine that the user has moved their neck, the app screen will be closed. The display 2 is switched from display to non-display, or from non-display to display. In this case, for example, if the difference between the images captured before and after the event is greater than or equal to a predetermined threshold, the image is not visible. The location of the device has changed, i.e. the user has moved their head to look at another location, and the app screen is cleared. The display 2 may be switched from visible to invisible, or from invisible to visible. The illustration is based on the assumption that the image is not displayed before motion detection.

[0058] Line of sight condition: Has it been detected that the line of sight has shifted from the mobile information terminal 80? (S0914) Image recognition processing is performed on the image from the side camera 3, and it is determined in which area of ​​the image the mobile information terminal 80 is located. The outer camera 3, the inner camera 4 and the display 2 are attached to the wearer. Since the image is fixed at 1, each image included in the image of the outer camera 3 on the display 2 The relationship between the position (pixel position) in the display and the display position on the display 2 is calibrated in advance. and generate and store outer camera alignment data.

[0059] Also, the gaze detection sensor 168 detects the eyes of the user from the image captured by the inner camera 4, The movement of the gaze is detected by tracking the change in the position of the eyes. The relationship with the display position of the display 2 can also be related, so Camera alignment data is generated and stored.

[0060] Then, the image of the outer camera 3 (the mobile information terminal 80 is captured) and the image of the inner camera 4 Each of the images (in which the user's eyes are imaged) is then matched with the outer camera alignment data and the inner camera alignment data. By comparing the camera positioning data with each other, the line of sight of the user M is Then, when the mobile information terminal 80 is included in the visual recognition area and the line of sight is not aligned with the terminal device, the mobile information terminal 80 is detected as being out of the visual recognition area. When the mobile information terminal 80 is on the display 2, the application screen is hidden. If your gaze is not on the device display 2, the app screen will be displayed on display 2. In this case, even if the mobile information terminal 80 is not included in the visible area, the display The app screen may be displayed on 2.

[0061] The screen movement middle 104b1 is a setting condition (S0911), an operation condition, etc., as shown in FIG. (S0912), body movement condition (S0913), or gaze condition (S0914) were satisfied. If it is determined that the application screen is displayed (S0911 to S0914 / any of them is Yes), the application screen display flag The value of is set to 1 (S0916) and the process ends. 11~S0914) are not satisfied (S0911~S0914 / all No ), the value of the application screen display flag is set to 0 (S0915), and the application screen control process ends. The process proceeds to a determination of whether the conditions are satisfied (S0917). The order of determining whether the display start conditions are satisfied is as shown in FIG. In addition, in FIG. 12, if any one of the above conditions is satisfied, step S09 is executed. Proceed to 16, but at least one of the conditions is met, i.e., one or more of the conditions are met If the above combination is satisfied, the process may proceed to step S0916.

[0062] According to this embodiment, the presence or absence of the portable information terminal 80 in the visible area described in the first embodiment Since the display start conditions can be used to control whether the app screen is displayed or not, By allowing the user M to set conditions based on the target and usage situation, the operation can be tailored to the user M's preferences. In the second embodiment, the availability of the mobile information terminal 80 in the visible area can be provided. It may also be combined with "none" as a display start condition.

[0063] <Third embodiment> The third embodiment shows variations regarding the display position of the application screen. There are various ways to determine where the app screen should be displayed on Display 2 of the HMD 10. The following are some of them:

[0064] (1) Display in preset position: Display the app screen in a preset position, such as the right edge of Display 2. The HMD control device 5 may display the application screen display control process (S0 9) Remember the position where the app screen was displayed the last time you closed the app, and then When the application screen display control process (S09) is resumed, the application screen is displayed at the previous end position. This is also fine.

[0065] (2) Display at a location according to the type of application: Display at a location according to the type of the mobile-enabled application 204d The position of the display 2 may be determined in advance. For example, If it is an app, the display 2 is off-center to ensure visibility in front of the eyes. The app screen may be displayed on the right or left edge. On the other hand, if it is an e-book app, For example, the app screen is displayed in the center of the display 2 so that the user can move around freely. The display position may be changed depending on the function of d.

[0066] (3) Gaze tracking: Based on the image from the inner camera 4, the gaze detection sensor 168 detects the user M The system will continue to detect your gaze and display the app screen where your gaze intersects with Display 2. For example, if the travel-enabled application 204d is a map navigation application, When you walk while wearing the MD10, a map will be displayed in your line of sight and the navigation will guide you along the way. This makes it easier to identify the actual roads and makes it easier to understand the navigation information.

[0067] (4) Position at which the mobile information terminal 80 is displayed again after it has moved out of the field of view of the HMD 10 Regarding the above, FIG. 13 is a diagram showing the redisplay position after the frame-out. As shown in FIG. When the outer camera 3 captures a video by generating a plurality of frames in a time series, The sides that form the outer edge of the frame and the sides a, b, c, and d of the display 2 and the display The small regions 1, 2, 3, and 4 of the ray 2 are associated with each other in advance. The screen movement middle 104b1 is an area adjacent to each side a, b, c, and d. The mobile information terminal 80 that was captured in the first frame was captured in the second frame captured later. The data corresponding to the edge is obtained by determining which edge (outer edge) the object crossed. The application screen may be displayed in a small area of ​​the display 2.

[0068] For example, as shown in FIG. 13, the outer camera 3 captures an image of a subject that is out of the frame at the top. In this case, the user M sees the edge a of the display 2 as being out of the frame. The preview screen is displayed in the small area 1. The frame captured by the outer camera 3 is When the frame is out of the frame, it appears to the user M as if the frame is out of the edge c of the display 2. Therefore, the app screen is displayed in small area 3.

[0069] In addition, the position displayed just before the mobile information terminal 80 was moved outside the display 2 is displayed. The application screen may be displayed at that position.

[0070] (5) Adjustment of the application screen position: Adjust the display position after moving the application screen to the HMD10. For example, a means for adjusting the display 281 of the mobile information terminal 80 may be provided. When you perform a swipe operation, the app screen on display 2 of the HMD10 will be linked to the swipe operation. The display position on the display 2 may be changed depending on the selected item.

[0071] According to this embodiment, the display position of the application screen can be adjusted according to the purpose of use and the user's preferences. This makes it possible to provide an HMD-linked system 100 with higher visibility.

[0072] <Fourth embodiment> In the fourth embodiment, the HMD 10 is started after the mobile application 204d is started. FIG. 14 shows the display control of the HMD linkage system in this embodiment. 15 is a diagram showing an example of a sequence of the HMD 10. .

[0073] When the HMD 10 starts up normally, the top screen 1501 of the HMD 10 is displayed on the display 2. is displayed, and then a screen regarding processing of the HMD 10 is displayed.

[0074] However, in the HMD linkage system 100, as shown in FIG. After the start-up (S11), the mobile-enabled application 204d starts up (S12), and a connection request is made (S13). When the HMD 10 is started (S1 5) The application screen is displayed from the link notification screen 1502 without displaying the top screen 1501. Alternatively, the screen may transition to a currently displayed screen 1503 or a standby screen 1504 while the application screen is not displayed.

[0075] That is, when the HMD is started (S15), the top screen 1501 is displayed in the normal start-up process. It is displayed on Play 2, but if the HMD10 starts up after receiving a notification that a mobile-enabled app is starting up, After that, the HMD 10 and the mobile information terminal 80 continue to perform a link process, so the top screen 1501 Without displaying the link, the link is established (S16) and the screen movement middleware 104b1 is started (S 17). The screen transfer middle 104b1 receives the link establishment and displays the link notification screen 1502. Then, imaging by the outer camera 3 is started (S18), and the application screen display control process (S19) is performed. The process may proceed to

[0076] In this embodiment, after the mobile information terminal 80 is operated to start using the mobile compatible application 204d, User M, who wanted to perform screen linking with the HMD10, started up the HMD10. In this case, the user M can quickly switch to the app screen on the HMD 10. I would like to display the link notification screen instead of the top screen 1501 as shown above. By displaying the top screen 1502, the user M does not want to see the top screen 1501 This avoids the discomfort of having to wait for the app screen to appear, and reduces the frustration of waiting for the app screen to appear. It is possible.

[0077] To further ease the sense of urgency, the link notification screen 1502 is also skipped and the app is However, if the application screen display start condition is not satisfied, the application screen display screen 1503 may be displayed. In this case, the app screen display screen 1503 is not displayed, so user M is not able to link. Therefore, if the application screen display start condition is met, the HM Immediately after the D10 is started, the application screen display screen 1503 is displayed, and the application screen display start condition is Only when the condition is not satisfied, the link notification screen 1502 may be displayed. In step S17, the dollar 104b1 does not display the link notification screen 1502, but displays the application screen. In the control process (S19), it is determined that the application screen should not be displayed on display 2. In this case, the link notification screen 1502 may be displayed.

[0078] According to this embodiment, the top screen of the HMD 10 in the HMD linkage system 100 is not displayed. When necessary, the display can be hidden, thereby reducing the user M's sense of impatience and improving convenience.

[0079] The above-described embodiments do not limit the present invention, and may be modified without departing from the spirit of the present invention. There are various modifications, which also fall within the technical scope of the present invention. For example, the first to fourth embodiments described above The embodiments may be combined in any manner. [Explanation of symbols]

[0080] 1: Wearable body 2: Display 3: Outer camera 4: Inside camera 5: HMD control device 10: HMD 80: Mobile information terminal 100: HMD linkage system

Claims

1. A display device, comprising: A display unit; Camera and A communication module for communicating with a portable information terminal; A control unit; Equipped with the display device and the mobile information terminal are equipped with the same application software; the display unit includes a plurality of sides forming an outer edge and a plurality of regions adjacent to each of the plurality of sides, The control unit is An image captured by the camera is displayed on the display unit; if the image does not include the mobile information terminal, controlling the display screen of the mobile information terminal to be displayed at a predetermined position on the display unit, and controlling the application software of the display device to be executed in synchronization with the application software of the mobile information terminal; When the mobile information terminal is included in the image, the display of the display screen on the display unit is terminated. When the portable information terminal is no longer included in the image and display of the display screen is resumed, if the portable information terminal has framed out from a first side which is one of the multiple sides and is no longer included in the image, the display screen is displayed in a first region which is one of the multiple regions and adjacent to the first side, and if the portable information terminal has framed out from a second side which is one of the multiple sides and is different from the first side and is no longer included in the image, the display screen is displayed in a second region which is one of the multiple regions and adjacent to the second side. A display device comprising:

2. The display device according to claim 1 , The control unit holds an image of the mobile information terminal captured by the camera as teacher data, and determines whether the mobile information terminal is included in the image captured by the camera by a pattern matching process using the teacher data. A display device characterized by:

3. The display device according to claim 1 , The control unit is When the process of displaying the display screen of the portable information terminal on the display unit is ended, and control the communication terminal to maintain a connection therewith. A display device comprising:

4. The display device according to claim 1 , The control unit is When the process of displaying the display screen of the portable information terminal on the display unit is ended, and control the device to disconnect from the information terminal. A display device comprising:

5. A method for controlling a display device, comprising: The display device includes a display unit, a camera, a communication module for communicating with a portable information terminal, and a control unit; the display device and the mobile information terminal are equipped with the same application software; the display unit has a plurality of sides forming an outer edge and a plurality of regions adjacent to each of the plurality of sides, The control unit is An image captured by the camera is displayed on the display unit; if the image does not include the mobile information terminal, controlling the display screen of the mobile information terminal to be displayed at a predetermined position on the display unit, and controlling the application software of the display device to be executed in synchronization with the application software of the mobile information terminal; When the mobile information terminal is included in the image, the display of the display screen on the display unit is terminated. When the portable information terminal is no longer included in the image and display of the display screen is resumed, if the portable information terminal has framed out from a first side which is one of the multiple sides and is no longer included in the image, the display screen is displayed in a first region which is one of the multiple regions and adjacent to the first side, and if the portable information terminal has framed out from a second side which is one of the multiple sides and is different from the first side and is no longer included in the image, the display screen is displayed in a second region which is one of the multiple regions and adjacent to the second side.

23. A method for controlling a display device comprising:

6. A method for controlling the display device according to claim 5, comprising: The control unit holds an image of the mobile information terminal captured by the camera as teacher data, and determines whether the mobile information terminal is included in the image captured by the camera by a pattern matching process using the teacher data.

23. A method for controlling a display device comprising:

7. A method for controlling the display device according to claim 5, comprising: The control unit is When the process of displaying the display screen of the portable information terminal on the display unit is ended, and control the communication terminal to maintain a connection therewith. A method for controlling a display device comprising:

8. A method for controlling the display device according to claim 5, comprising: The control unit is When the process of displaying the display screen of the portable information terminal on the display unit is ended, and control the communication terminal to disconnect the communication terminal. A method for controlling a display device comprising:

Images