Head-mounted display device

Abstract

To provide a HMD that allows display control of additional information of an information terminal to be performed on the display unit of the HMD when the information terminal displaying a main image with additional information superimposed thereon is viewed via the display unit of the HMD.SOLUTION: A head-mounted display device having a transmissive display unit includes: an image acquisition unit that acquires a field of view range image, which is an image corresponding to a user's field of view range; an additional information presence / absence analysis unit that analyzes the acquired field of view range image, checks whether an information terminal is included in the user's field of view range, and checks whether the display image of the information terminal included in the user's field of view range is a main image with additional information superimposed thereon; an additional information acquisition unit that requests the information terminal or a content server to transmit the additional information and receives the additional information; and a display control unit that detects an information terminal viewing area, which is the area in which the information terminal is viewed in the transmissive display unit, and displays the received additional information in a position that does not overlap with the information terminal viewing area of the transmissive display unit.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a head-mounted display device (hereinafter referred to as HMD: Head Mounted Display), and particularly to the display of video additional information.

Background Art

[0002] An HMD equipped with a display and worn on the head to view information is increasingly being used in recent years in fields such as gaming and work support.

[0003] Here, when a user views an information terminal (TV, smartphone, etc.) that displays additional information superimposed on a main image through the transmissive display unit of the HMD, the main image may be difficult to see due to the additional information overlapping the main image.

[0004] As background art in this technical field, there is Patent Document 1. Patent Document 1 discloses a video additional information display control device and its operation method in which video is displayed on a display device, additional information is displayed on an HMD, and the additional information does not overlap the display object in the video.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In Patent Document 1, a video additional information display control device controls the display position of additional information so that the additional information does not overlap the display object in the video displayed on the HMD, and the HMD simply displays the additional information based on the control of the video additional information display control device. Therefore, the subject of control is the video additional information display control device, and there is a problem that a video additional information display control device is required separately from the HMD.

[0007] In view of the above problems, an object of the present invention is to provide a head-mounted display (HMD) capable of controlling the display of additional information of an information terminal when viewing, via the display unit of the HMD, an information terminal that displays additional information superimposed on a main image.

Means for Solving the Problems

[0008] Taking an example, the present invention is a head-mounted display device having a transmissive display unit, including an image acquisition unit that acquires a field-of-view range image, which is an image corresponding to the user's field of view; an additional information presence / absence analysis unit that analyzes the acquired field-of-view range image to check whether an information terminal is included within the user's field of view and whether the display image of the information terminal included within the user's field of view has additional information superimposed on the main image; an additional information acquisition unit that requests the transmission of additional information to the information terminal or a content server and receives the additional information; and a display control unit that detects an information terminal viewing area, which is an area for visually recognizing the information terminal in the transmissive display unit, and displays the received additional information at a position that does not overlap with the information terminal viewing area of the transmissive display unit.

Effects of the Invention

[0009] According to the present invention, an HMD capable of controlling the display of additional information can be provided.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 11

Figure 12A

Figure 12B

Figure 12C

Figure 13A

Figure 13B

Figure 13C

Figure 14

Figure 15A

Figure 15B

Figure 15C

Figure 16

Figure 17A

Figure 17B

Figure 17C

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Examples

[0012] FIG. 1 is a block diagram showing the hardware configuration of the HMD 100 in this embodiment. In FIG. 1, 101 is a main control unit (CPU / MCU, etc.), 102 is a bus which is a transmission / reception path for commands and data, 103 is a RAM which serves as a work area when executing a basic operation program and other operation programs, and 110 is a storage unit which is a non-volatile storage medium such as FlashROM / EEPROM / SSD / HDD.

[0013] 120 is an operation input unit which is a user operation interface, and includes operation keys 121 such as SWs, a power key, and a volume key, and a touch sensor 122 such as a touch pad.

[0014] 130 is an image processing unit such as an image (video) processor, and includes a transmissive display unit (display) 131, an image signal processing unit (image (video) signal processor) 132, a first image input unit 133 which is an external camera for front shooting, and a second image input unit 134 which is an internal camera for gaze detection.

[0015] 140 is an audio processing unit (audio processor), and includes an audio output unit (speaker) 141, an audio signal processing unit (audio signal processor) 142, and an audio input unit (microphone) 143.

[0016] 151 is a position information acquisition unit (for receiving a position information signal such as a GPS signal). Also, 152 is a distance measurement unit and includes a distance sensor and a depth sensor. Also, 160 is a sensor unit and includes a gyro sensor 161, a geomagnetic sensor 162, and an acceleration sensor 163.

[0017] 170 is a communication unit (communication interface) including a LAN (Wi-Fi (registered trademark)) communication unit, a mobile communication unit, a Bluetooth (registered trademark) communication unit, etc., and 180 is an expansion interface unit, for example, a USB interface, etc., which is used for data transmission and reception, charging, etc.

[0018] Figure 2 is an external view and a mounting diagram of the HMD in this embodiment. In Figure 2, the upper figure is a top view of the state where the HMD 100 is mounted on the head of the user U1, the middle figure is a front view, and the lower figure is a side view. The HMD 100 in this embodiment is a goggle-type, optical transmissive HMD with a window on the front.

[0019] In Figure 2, 131 is a transmissive display unit, and the outside world can be directly visually recognized through (transmitting through) the transmissive display unit 131.

[0020] 133L is the first image input unit (L), which is the external camera on the left side, and 133R is the first image input unit (R), which is the external camera on the right side. Note that there may be only one first image input unit (for example, one first image input unit 133 in the central part). Also, distance measurement may be performed by stereo shooting with 133L and 133R. In this case, the distance measurement unit 152 may not be necessary. Alternatively, it may be equipped with one image input device having the functions of the first image input unit 133 and the distance measurement unit 152, and the camera can simultaneously acquire image information and distance information with the image input device.

[0021] Also, 122L is the touch sensor (L), and 122R is the touch sensor (R) (not shown in the figure). There may be only one touch sensor on either the left or the right. Also, 141L is the audio output unit (L) (stereo speaker L), and 141R is the audio output unit (R) (stereo speaker R) (not shown in the figure). Although not shown in the figure, it may have a monaural microphone as the audio input unit 143.

[0022] Figure 3 is a functional configuration diagram of the HMD in this embodiment. In Figure 3, the storage unit 110 is composed of a basic operation program 1001, which is a basic program such as an OS, an application program (hereinafter abbreviated as an app) that executes the functions of this embodiment, and other apps, and an application 1002, and various data storage areas 1009, which are areas for storing various operation setting values and various information (videos / still images / audio, etc.).

[0023] The programs / apps stored in the storage unit 110 are expanded (loaded) into the RAM 103, and the main control unit 101 executes the expanded programs / apps, whereby the functional units and other functional units of this embodiment are executed.

[0024] In FIG. 3, the RAM 103 stores the expanded execution program by function. That is, a basic operation function unit 1101 that executes the basic operation program of the HMD 100, an image acquisition unit 1111 that acquires an image (field-of-view range image) corresponding to the field-of-view range of the user U1 wearing the HMD 100, and an additional information presence / absence analysis unit 1112 that analyzes whether the display image of the information terminal 200 included in the field-of-view range image acquired by the image acquisition unit 1111 has additional information superimposed on the main image. An additional information acquisition unit 1113 that acquires data related to the additional information superimposed on the display image of the information terminal 200 included in the field-of-view range image acquired by the image acquisition unit 1111, and a display control unit 1114 that detects the area (information terminal visual recognition area) occupied in the transmissive display unit 131 of the information terminal 200 included in the field-of-view range image acquired by the image acquisition unit 1111 and controls the display position and display presence of the additional information based on the detection result. Further, the RAM 103 has a temporary storage area 1199 that is a temporary storage area for various information created / acquired by an application.

[0025] FIG. 4 is a system configuration diagram in this embodiment. In FIG. 4, the HMD 100 is connected to information terminals 200s such as smartphones and tablet terminals, information terminals 200t such as TVs and other display devices, and a server device 310 such as a content server via a network 300 such as the Internet.

[0026] FIG. 5 is a flowchart of the additional information display control process of the HMD in this embodiment. In FIG. 5, first, in step S101, the image acquisition unit 1111 acquires a field-of-view range image that is an image corresponding to the field-of-view range of the user U1 from the first image input unit 133. Next, in S102, the additional information presence / absence analysis unit 1112 analyzes the field-of-view range image of the user U1 acquired in the process of S101 and checks whether the information terminal 200 is included in the field-of-view range of the user U1. If the information terminal 200 is included in the field-of-view range of the user U1, the process proceeds to S103. If the information terminal 200 is not included in the field-of-view range of the user U1, the process ends.

[0027] In S103, the additional information presence / absence analysis unit 1112 analyzes the field-of-view image of user U1 obtained in the process of S101, and checks whether the display image of the information terminal 200 included in the field of view of user U1 has additional information superimposed on the main image. If the additional information is superimposed on the main image, the process proceeds to S104. If the additional information is not superimposed on the main image (only the main image), the process ends.

[0028] In S104, the additional information acquisition unit 1113 requests the information terminal 200 or the server device 310, which is a content server, to transmit additional information. Then, in S105, the additional information acquisition unit 1113 receives the additional information from the information terminal 200 or the server device 310. Note that if the additional information cannot be received within a predetermined time, the process may end. Then, in S106, the additional information acquisition unit 1113 transmits a display stop request for the additional information to the information terminal 200.

[0029] Then, in S107, the image acquisition unit 1111 acquires a field-of-view image, which is an image corresponding to the field of view of user U1, from the first image input unit 133. Note that the process of S107 may not be performed, and the field-of-view image obtained in the process of S101 may be used in subsequent processes.

[0030] Then, in S108, the display control unit 1114 analyzes the field-of-view image of user U1 obtained in the process of S107, and detects an information terminal viewing area, which is an area where the information terminal 200 is visible, within the transmissive display unit 131. Then, in S109, the display control unit 1114 displays the additional information received from the information terminal 200 or the server device 310 in the process of S105 at a predetermined position of the transmissive display unit 131 that does not overlap with the information terminal viewing area detected in the process of S108. Note that after the process of S106, if the additional information is still being displayed on the information terminal 200, the processes of S107 to S109 may not be performed.

[0031] FIG. 6 is a detailed flowchart of the additional information presence / absence analysis process of S103 in FIG. 5. In FIG. 6, first, at step S121, the additional information presence / absence analysis unit 1112 transmits, via the communication unit 170, a confirmation request to the information terminal 200 connected on the same network 300, asking whether or not the display image of the information terminal 200 has additional information superimposed on the main image. Note that the information terminal 200 cooperates with the HMD 100 within a local area network such as LAN or Bluetooth.

[0032] Then, at S122, the additional information presence / absence analysis unit 1112 checks whether or not it has received a reply from the information terminal 200 to the confirmation request transmitted in the process of S121. If a reply from the information terminal 200 is received within a predetermined time, the process proceeds to the process of S123. If a reply from the information terminal 200 is not received within a predetermined time, the process proceeds to the process of S125.

[0033] At S123, the additional information presence / absence analysis unit 1112 checks, based on the reply from the information terminal 200, whether or not the display image of the information terminal 200 has additional information superimposed on the main image. If the display image has additional information superimposed on the main image, the process proceeds to the process of S124. If the display image does not have additional information superimposed on the main image, the process proceeds to the process of S125.

[0034] At S124, the additional information presence / absence analysis unit 1112 sets the additional information superimposition flag to "1" and branches to "Yes" in the process of S103.

[0035] At S125, the additional information presence / absence analysis unit 1112 sets the additional information superimposition flag to "0" and branches to "No" in the process of S103.

[0036] FIG. 7 is a detailed flowchart of another additional information presence / absence analysis process of S103 in FIG. 5. FIG. 7 checks whether or not the display image of the information terminal 200 included within the visual field range has additional information superimposed on the main image, based on information from the server device 310, which is a content server via the network 300, rather than the information terminal 200.

[0037] In FIG. 7, first, in step S131, the additional information presence / absence analysis unit 1112 extracts the display image portion of the information terminal 200 from the field-of-view range image of the user U1 acquired in the process of S101, performs a search process on the network 300 based on the extracted display image, and acquires information regarding the server device 310 that supplies the main image (and / or additional information) to the information terminal 200. Then, in S132, the additional information presence / absence analysis unit 1112 checks whether information regarding the server device 310 that supplies the main image (and / or additional information) to the information terminal 200 has been acquired by the process of S131. If information regarding the server device 310 has been acquired, the process proceeds to the process of S133. If information regarding the server device 310 has not been acquired, the process proceeds to the process of S136.

[0038] In S133, the additional information presence / absence analysis unit 1112 transmits the display image portion of the information terminal 200 extracted in the process of S131 to the server device 310, and the server device 310 acquires information regarding the main image (and / or additional information) supplied to the information terminal 200.

[0039] Then, in S134, based on the image of the display image portion of the information terminal 200 extracted in the process of S131 and the information regarding the main image (and / or additional information) supplied to the information terminal 200 by the server device 310 acquired in the process of S133, the additional information presence / absence analysis unit 1112 checks whether the display image of the information terminal 200 has additional information superimposed on the main image. If the display image has additional information superimposed on the main image, the process proceeds to the process of S135. If the display image does not have additional information superimposed on the main image, the process proceeds to the process of S136.

[0040] In S135, similar to S124 in FIG. 6, the additional information presence / absence analysis unit 1112 sets the additional information superimposition flag to "1" and branches to "Yes" in the process of S103.

[0041] In S136, similar to S125 in FIG. 6, the additional information presence / absence analysis unit 1112 sets the additional information overlay flag to "0" and branches to "No" in the process of S103.

[0042] FIG. 8 is an explanatory diagram of the detection process of the information terminal visual recognition area in S108 in FIG. 5. The upper diagram in FIG. 8 shows the visual field of the user U1 through the transmissive display unit 131, and shows the real-world information terminal 200 that the user U1 visually recognizes through the transmissive display unit 131. The lower diagram in FIG. 8 is a diagram for explaining the information terminal visual recognition area in the transmissive display unit 131, and shows the information terminal visual recognition area 131R detected in the process of S108 with respect to the real-world information terminal 200 that the user U1 visually recognizes through the transmissive display unit 131.

[0043] Note that the information terminal visual recognition area 131R does not necessarily have a shape similar to the housing as shown in the figure, and may be, for example, a simple rectangle. Also, the information terminal visual recognition area 131R may be a shape that encloses only the display image portion of the information terminal 200. Further, the additional information received from the information terminal 200 may be displayed anywhere outside the information terminal visual recognition area 131R instead of being displayed inside the information terminal visual recognition area 131R.

[0044] FIG. 9 is an explanatory diagram of the additional information display control process of the HMD in this embodiment. As shown in the upper diagram of FIG. 9, when viewing the display image of the information terminal 200 through the transmissive display unit 131 of the HMD 100, if it is determined that the display image of the information terminal 200 has additional information (such as subtitle information) superimposed on the main image (program content, etc.), (1) the HMD 100 requests the information terminal 200 to transmit the additional information and stop displaying the additional information. (2) The HMD 100 receives the additional information from the information terminal 200. (3) Based on the user's field-of-view image acquired by the camera of the HMD 100, the information terminal viewing area in the transmissive display unit 131 of the HMD 100 is detected. Then, as shown in the lower diagram of FIG. 9, (4) the HMD 100 displays the additional information received in (2) at a position that does not overlap with the information terminal viewing area 131R detected in (3) of the transmissive display unit 131 of the HMD 100. Note that if it is determined that only the main image is displayed and no additional information is superimposed, the above processes (1) to (4) are not performed.

[0045] Also, to determine whether additional information is superimposed on the main image, the HMD 100 asks the information terminal 200 whether the display image has additional information superimposed on the main image. Or, the HMD 100 asks the content server that supplies the main image to the information terminal 200 about the main image and the additional information. Note that other methods may be used for the determination.

[0046] Also, the display unit of the HMD 100 may be non-transmissive. In this case, in the display control of (4) above, the user's field-of-view image acquired by the camera of the HMD 100 is displayed on the non-transmissive display unit, and the additional information received in (2) is superimposed and displayed at a position that does not overlap with the area where the information terminal is displayed.

[0047] Also, the HMD 100 may be of a glasses type or the like in addition to the illustrated goggle type. Also, the information terminal 200 may be a TV placed in the room, a handheld smartphone, or other information devices.

[0048] According to the present embodiment as described above, when viewing an information terminal that displays additional information superimposed on a main image through the display unit of an HMD, an HMD capable of controlling the display of additional information in the display unit of the HMD to be displayed at a position that does not overlap with the main image of the information terminal on the display unit of the HMD can be provided.

Embodiment

[0049] In this embodiment, details of the display position of additional information in the additional information display control process will be described.

[0050] FIGS. 10A, 10B, and 10C are diagrams for explaining the display control of additional information in this embodiment. In FIG. 10A, it is a case where the user U1 is facing the information terminal 200 directly, and shows a state where additional information is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 under the control in Embodiment 1. Note that U1S indicates the visual field range of the user U1.

[0051] FIG. 10B shows a first state in which the user U1 shakes the head to the left from the state of FIG. 10A, and the position of the information terminal 200 in the transmissive display unit 131 moves relatively to the right. In this case, the display position of the additional information in the transmissive display unit 131 is set to the same position as in FIG. 10A. That is, the display position of the additional information is fixed at a predetermined position in the transmissive display unit 131.

[0052] FIG. 10C shows a second state in which the user U1 shakes the head to the left from the state of FIG. 10A as in FIG. 10B, and the position of the information terminal 200 in the transmissive display unit 131 moves relatively to the right. In this case, the display position of the additional information in the transmissive display unit 131 may move to the right in accordance with the information terminal 200. That is, the display position of the additional information may be fixed at a relative position with respect to the information terminal 200.

[0053] Note that the display position of the additional information may be outside the visual recognition area of the information terminal and may be displayed in the background part where the additional information is most visible. For example, when the background part is composed of a patterned curtain and a white wall, control such as displaying the additional information so as to overlap the white wall is performed.

[0054] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the display unit of the HMD, the additional information of the information terminal is displayed at a position that does not overlap with the main image of the information terminal in the display unit of the HMD and at a position where the additional information is easy to view. It is possible to provide an HMD capable of controlling the display of additional information in the display unit of the HMD.

Example

[0055] In this embodiment, the display control process of the additional information selected by the user will be described.

[0056] FIG. 11 is a flowchart of the additional information display control process of the HMD in this embodiment. Note that the flowchart shown in FIG. 11 is a modified example of the additional information display control process described in the first embodiment.

[0057] In FIG. 11, first, in step S201, the image acquisition unit 1111 acquires an image (field-of-view range image) corresponding to the field-of-view range of the user U1 from the first image input unit 133. Then, in S202, the additional information presence / absence analysis unit 1112 receives a selection instruction from the user U1 for an arbitrary object on the display image of the information terminal 200 based on the field-of-view range image of the user U1 acquired in the process of S201. Then, in S203, the additional information presence / absence analysis unit 1112 confirms the position of the object (selected object) selected by the user U1 in the process of S202 within the display image of the information terminal 200.

[0058] Next, in S204, the additional information presence / absence analysis unit 1112 checks whether the selected object whose position in the display image of the information terminal 200 was confirmed in the process of S203 is additional information superimposed on the main image. If the selected object is additional information superimposed on the main image, the process proceeds to S205. If the selected object is not additional information superimposed on the main image, that is, if it is a part of the main image, the process ends.

[0059] In S205, the additional information acquisition unit 1113 requests the information terminal 200 or the server device 310, which is the content server, to transmit the selected object that is additional information. Then, in S206, the additional information acquisition unit 1113 receives the selected object (additional information) from the information terminal 200 or the server device 310.

[0060] In S207, the additional information acquisition unit 1113 transmits a display stop request for the selected object (additional information) to the information terminal 200. Then, in S208, the image acquisition unit 1111 acquires an image (field-of-view range image) corresponding to the field of view of the user U1 from the first image input unit 133.

[0061] In S209, the display control unit 1114 analyzes the field-of-view range image of the user U1 acquired in the process of S208 and detects a region (information terminal visual recognition region) for visually recognizing the information terminal 200 within the transmissive display unit 131. Then, in S210, the display control unit 1114 displays the selected object (additional information) received from the information terminal 200 or the server device 310 in the process of S206 at a predetermined position of the transmissive display unit 131 that does not overlap with the information terminal visual recognition region detected in the process of S209.

[0062] Note that specific examples of the selection instruction of the user U1 for an arbitrary object in S202 are as follows. While the user U1 is gazing at an arbitrary object on the display image displayed on the information terminal 200 via the transmissive display unit 131, (1) the touch sensor 122 detects a tap operation by the user U1, or (2) the second image input unit 134 detects a blinking operation of the user U1.

[0063] Also, the following are specific examples of the position confirmation process for the selection object in S203. (1) When there is a selection instruction by the user U1, the second image input unit 134 acquires the eye image of the user U1. (2) The acquired eye image is analyzed to detect the coordinates on the transmissive display unit 131 where the line of sight of the user U1 intersects. (3) Calculate which position of the user U1's visual field range image acquired in the process of S201 the detected coordinate position corresponds to. (4) Based on the position information within the calculated visual field range image, calculate which position the selection object at the line of sight position of the user U1 is in the display image of the information terminal 200.

[0064] FIG. 12A, FIG. 12B, and FIG. 12C are diagrams for explaining the display control of additional information in this embodiment. In FIG. 12A, FIG. 12B, and FIG. 12C, the left diagram and the right diagram respectively show the user's visual field through the transmissive display unit of the HMD before and after the additional information display control.

[0065] FIG. 12A shows a case where additional information 1 and additional information 2 exist and the user U1 makes a selection instruction while looking at additional information 1. That is, as a UI (User Interface) that can select additional information or a UI that serves as a trigger for selecting additional information, a cross cursor is displayed at the position where the line of sight is directed. When the cross cursor is displayed while looking at the additional information to be selected, a selection instruction is made by detecting the tap operation or blink operation of the touch sensor described above. In FIG. 12A, due to the additional information display control described in FIG. 11, only additional information 1 is displayed at a position that does not overlap with the information terminal viewing area within the transmissive display unit 131 and is deleted from the information terminal 200, and additional information 2 remains displayed on the information terminal 200, that is, remains displayed while being superimposed on the main image.

[0066] FIG. 12B shows a case where the user U1 gives a selection instruction while focusing on the additional information 2, which is different from FIG. 12A. In FIG. 12B, according to the additional information display control described in FIG. 11, only the additional information 2 is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 and is deleted from the information terminal 200, and the additional information 1 is displayed while being superimposed on the main image.

[0067] FIG. 12C shows another display control case where the additional information 1 and the additional information 2 exist and the user U1 gives a selection instruction while focusing on the additional information 1. In FIG. 12C, by selecting one piece of additional information, all the additional information superimposed on the main image of the information terminal 200 is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 and is deleted from the information terminal 200. In this way, all the additional information may be displayed at a position that does not overlap with the information terminal viewing area by using the selection of one piece of additional information as a trigger.

[0068] Also, according to the additional information display control, the additional information displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 may be displayed with its shape changed according to the shape of the transmissive display unit 131 and the position and size of the information terminal viewing area.

[0069] Also, even when there is one piece of additional information superimposed on the main image of the information terminal 200, the additional information display control process may be started by using the selection instruction by the user U1 as a trigger.

[0070] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image through the display unit of the HMD, an HMD can be provided that has a UI that enables selection of the additional information and displays the selected additional information at a position that does not overlap with the main image of the information terminal in the display unit of the HMD, and enables display control of the additional information in the display unit of the HMD.

Example

[0071] In this example, another example of the display process of the additional information will be described.

[0072] Figures 13A, 13B, and 13C are diagrams for explaining the display control of additional information in this embodiment. In Figures 13A, 13B, and 13C, similar to Figure 12A, the left diagram and the right diagram respectively show the user's field of view through the transmissive display unit of the HMD before and after the additional information display control.

[0073] Figure 13A shows the first processing of the additional information on the information terminal side after the additional information display control. That is, as shown in Figure 13A, after the additional information is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 by the additional information display processing control, on the information terminal 200 side, the display of the additional information may be stopped.

[0074] Figure 13B shows the second processing of the additional information on the information terminal side after the additional information display control. That is, as shown in Figure 13B, after the additional information is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 by the additional information display processing control, on the information terminal 200 side, the additional information may be reduced and displayed in a size that does not interfere with the viewing of the main image. In this case, instead of the additional information display stop request in the process of S106 in Figure 5, an additional information reduction display request is transmitted.

[0075] Figure 13C shows the third processing of the additional information on the information terminal side after the additional information display control. That is, as shown in Figure 13C, after the additional information is displayed at a position that does not overlap with the information terminal viewing area in the transmissive display unit 131 by the additional information display processing control, on the information terminal 200 side, the additional information may be moved and displayed at a position that does not interfere with the viewing of the main image. In this case, instead of the additional information display stop request in the process of S106 in Figure 5, an additional information movement request is transmitted. Note that the movement position of the additional information in the information terminal 200 may be a position where a part of the additional information protrudes from the display unit of the information terminal 200, as shown in Figure 13C.

[0076] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image through the display unit of an HMD, the additional information of the information terminal is displayed at a position that does not overlap with the main image of the information terminal on the display unit of the HMD, and an HMD capable of controlling the display of the additional information so as not to interfere with the viewing of the main image can be provided.

Embodiment

[0077] In this embodiment, an example of acquiring the additional information of an information terminal via a network and controlling the display of the additional information will be described.

[0078] FIG. 14 is a flowchart of the additional information display control process of the HMD in this embodiment. Note that the flowchart shown in FIG. 14 is a modified example of the additional information display control process described in Embodiment 1 and Embodiment 3.

[0079] In FIG. 14, first, in step S301, the additional information presence / absence analysis unit 1112 checks whether there is an information terminal 200 that has been coordinated on the same network 300. If there is a coordinated information terminal 200, the process proceeds to the process of S302. If there is no coordinated information terminal 200, the process ends.

[0080] Next, in S302, the additional information acquisition unit 1113 transmits a reservation for transmitting additional information to the coordinated information terminal 200. Note that regardless of whether additional information is being displayed on the information terminal side, a reservation for transmitting additional information may be transmitted to the information terminal in advance. Then, in S303, the image acquisition unit 1111 acquires an image (field-of-view range image) corresponding to the field-of-view range of the user U1 from the first image input unit 133.

[0081] Next, in S304, the additional information presence / absence analysis unit 1112 analyzes the field-of-view range image of the user U1 acquired in the process of S303, and checks whether a coordinated information terminal 200 is included in the field-of-view range of the user U1. If a coordinated information terminal 200 is included in the field-of-view range of the user U1, the process proceeds to the process of S305. If a coordinated information terminal 200 is not included in the field-of-view range of the user U1, the process returns to the process of S303.

[0082] In S305, the additional information presence / absence analysis unit 1112 transmits to the linked information terminal 200 that it is in a state (information terminal visible state) where the linked information terminal 200 is included within the visual field range of the user U1. Then, in S306, the additional information acquisition unit 1113 receives the additional information transmitted from the linked information terminal 200 corresponding to the information terminal visible state transmitted in the process of S305. If the display image of the linked information terminal 200 does not have additional information superimposed on the main image, the process may be terminated after receiving information to that effect. That is, in the information terminal 200, when receiving from the HMD100 that it is in the information terminal visible state, if the display image of the information terminal 200 has additional information superimposed on the main image, the additional information is transmitted to the HMD100, and if the display image of the information terminal 200 does not have additional information superimposed on the main image, information to that effect is transmitted to the HMD100.

[0083] Next, in S307, the additional information acquisition unit 1113 transmits a request to stop displaying the additional information to the linked information terminal 200. Then, in S308, the image acquisition unit 1111 acquires an image (visual field range image) corresponding to the visual field range of the user U1 from the first image input unit 133. Note that the process of S308 may not be performed, and the visual field range image acquired in the process of S303 may be used in subsequent processes.

[0084] Next, in S309, the display control unit 1114 analyzes the visual field range image of the user U1 acquired in the process of S308 and detects a region (information terminal visible region) for visually recognizing the linked information terminal 200 within the transmissive display unit 131. Then, in S310, the display control unit 1114 displays the additional information received from the linked information terminal 200 in the process of S306 at a predetermined position of the transmissive display unit 131 that does not overlap with the information terminal visible region detected in the process of S309.

[0085] Figs. 15A, 15B, and 15C are diagrams for explaining the display control of additional information in this embodiment. Fig. 15A is a diagram for explaining the first state, in which user U1 is facing the information terminal 200 in the left direction, and the entire information terminal 200 does not enter the visual field range U1S of user U1, showing a state where the entire information terminal 200 does not enter the transmissive display unit 131. And, in the state where the information terminal 200 that has been linked to the visual field range of user U1 does not enter in the process of S304 in Fig. 14, the information terminal visual recognition state has not been transmitted to the linked information terminal 200, and the additional information has not been received from the linked information terminal 200. Although not shown, the display image of the information terminal 200 is in a state where the additional information is superimposed on the main image.

[0086] Fig. 15B is a diagram for explaining the second state, showing the first state in which the entire information terminal 200 enters the visual field range U1S of user U1 as user U1 swings his head to the right from the state of Fig. 15A. In this case, it is the state immediately after detecting that the linked information terminal 200 has entered the visual field range of user U1 in the process of S304 in Fig. 14, and the additional information has not yet been received from the linked information terminal 200. Still, the display image of the information terminal 200 is in a state where the additional information is superimposed on the main image.

[0087] Fig. 15C is a diagram for explaining the third state, showing the state after performing the processes after S305 in Fig. 14 from the state of Fig. 15B. That is, Fig. 15C shows the state where the information terminal visual recognition state is transmitted to the linked information terminal 200 in the process of S305 in Fig. 14, the additional information is received from the linked information terminal 200 in the process of S306, and in the process of S310, the additional information received from the linked information terminal 200 in the process of S306 is displayed at a predetermined position of the transmissive display unit 131 that does not overlap with the information terminal visual recognition region detected in the process of S309.

[0088] As described above, according to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image through the display unit of the HMD, the additional information of the information terminal is acquired via a network, and the additional information is displayed at a position that does not overlap with the main image of the information terminal on the display unit of the HMD. Thus, it is possible to provide an HMD capable of controlling the display of additional information on the display unit of the HMD.

[0089] In the above description, in the process of S304 in FIG. 14, when the entire information terminal 200 does not enter the viewing range U1S of the user U1, it is determined that the information terminal 200 that has been linked to the viewing range of the user U1 is not present. When the entire information terminal 200 enters the viewing range U1S of the user U1, it is determined in the process of S304 in FIG. 14 that the information terminal 200 that has been linked to the viewing range of the user U1 is present. However, it may be determined that in the process of S304 in FIG. 14, when a part of the information terminal 200 does not enter the viewing range U1S of the user U1, the information terminal 200 that has been linked to the viewing range of the user U1 is not present, and when a part of the information terminal 200 enters the viewing range U1S of the user U1, it is determined in the process of S304 in FIG. 14 that the information terminal 200 that has been linked to the viewing range of the user U1 is present.

Embodiment

[0090] In this embodiment, another example of the display control of the additional information in Embodiment 5 will be described.

[0091] FIG. 16 is a flowchart of the additional information display control process of the HMD in this embodiment. FIG. 16 may be executed in parallel with FIG. 14 of Embodiment 5 or may be executed independently.

[0092] In FIG. 16, first, in step S321, similar to S303 in FIG. 14, the image acquisition unit 1111 acquires an image (viewing range image) corresponding to the viewing range of the user U1 from the first image input unit 133.

[0093] Next, in S322, the additional information presence / absence analysis unit 1112 analyzes the field-of-view image of the user U1 acquired in the process of S321, and checks whether there is an information terminal 200 that has been coordinated on the network within the field of view of the user U1. If an already coordinated information terminal 200 is included within the field of view of the user U1, the process returns to the process of S321. If an already coordinated information terminal 200 is not included within the field of view of the user U1, the process proceeds to the process of S323.

[0094] In S323, the additional information acquisition unit 1113 requests the transmission of the main image to the already coordinated information terminal 200. In addition, a request for redisplay of the additional information may also be transmitted. Next, in S324, the additional information acquisition unit 1113 receives the main image from the already coordinated information terminal 200. Then, in S325, the display control unit 1114 displays the main image received from the already coordinated information terminal 200 in the process of S324 at a predetermined position of the transmissive display unit 131.

[0095] FIGS. 17A, 17B, and 17C are diagrams for explaining the display control of the additional information in the present embodiment. FIG. 17A is the same as FIG. 15C and is a diagram for explaining the first state in the present embodiment, showing a state in which the entire information terminal 200 is within the field of view U1S of the user U1. In this case, a state is shown in which the additional information is displayed at a predetermined position that does not overlap with the information terminal visual recognition area of the transmissive display unit 131.

[0096] FIG. 17B is a diagram for explaining the second state, showing a state in which the user U1 swings the head to the left direction from the state of FIG. 17A, and the position of the information terminal 200 within the transmissive display unit 131 has moved relatively to the right direction. That is, in the process of S322, it is detected that a part of the information terminal 200 is missing from the field of view U1S of the user U1, but the additional information is shown in a state of being fixedly displayed at a predetermined position of the transmissive display unit 131.

[0097] FIG. 17C is a diagram for explaining a third state, showing a state where the user U1 further swings the head to the left from the state of FIG. 17B and the information terminal 200 is no longer included in the visual field range U1S of the user U1. That is, in the process of S322, it is detected that the information terminal 200 has deviated from the visual field range U1S of the user U1, and as an alternative to the information terminal 200 that was visually recognized through the transmissive display unit 131 for the main image of the information terminal 200 received in the process of S324, it shows the state of being displayed on the transmissive display unit 131.

[0098] Thus, in this embodiment, even if the position of the information terminal moves relatively in the visual field range, the additional information is fixedly displayed at a predetermined position on the transmissive display unit. Also, even when the information terminal is no longer included in the visual field range, the main image of the information terminal can be displayed on the transmissive display unit.

Embodiment

[0099] In this embodiment, another example of the display control of the additional information will be described.

[0100] FIG. 18 is an explanatory diagram of the additional information display control process of the HMD in this embodiment. FIG. 18 shows the visual field of the user U1 through the transmissive display unit 131. The upper diagram shows a state where the information terminal 200 with the additional information displayed is visually recognized through the transmissive display unit 131. The lower diagram in FIG. 18 shows a state where the additional information is displayed on the transmissive display unit 131 by the additional information display control.

[0101] In addition to being displayed in a state of being superimposed on the main image, the additional information may be displayed, for example, in a display shape such as data broadcasting in an L shape in the area vacated by reducing the main image, as shown in the upper diagram of FIG. 18.

[0102] In the case of the display shape of the additional information as shown in the upper diagram, in this embodiment, by the additional information display control, while requesting the information terminal 200 to transmit the additional information, a request for enlarged display of the main image is made, and as shown in the lower diagram of FIG. 18, the main image is enlarged and the additional information is displayed at a position that does not overlap with the information terminal visual recognition area of the transmissive display unit 131.

[0103] According to the present embodiment as described above, when viewing an information terminal that displays additional information superimposed on the main image through the display unit of the HMD, even if the additional information is superimposed in an L shape, it is possible to display the additional information of the information terminal at a position that does not overlap with the main image of the information terminal on the display unit of the HMD. An HMD having display control of additional information can be provided.

Example

[0104] In this embodiment, another example of the display control of additional information will be described.

[0105] FIG. 19 is an explanatory diagram of the additional information display control process of the HMD in this embodiment. In this embodiment, it is assumed that the distance between the user U1 and the information terminal 200 is short, and the information terminal 200 occupies most of the visual field of the user U1 through the transmissive display unit 131. That is, the information terminal viewing area 131R occupies most of the transmissive display unit 131, and in the additional information display control process of the HMD, it is an example of the additional information display control process when it is difficult to display the additional information at a position that does not overlap with the information terminal viewing area 131R within the display area of the transmissive display unit 131.

[0106] The upper diagram in FIG. 19 shows a state in which the information terminal 200 with the additional information displayed is viewed through the transmissive display unit 131, and shows the state before the additional information display control process is performed. On the information terminal 200, the additional information 200s is superimposed and displayed on the main video 200m. Also, as described above, the information terminal 200 occupies most of the visual field of the user U1 through the transmissive display unit 131.

[0107] The lower figure of FIG. 19 shows a state in which additional information is displayed on the transmissive display unit 131 by additional information display control, and shows the state after the additional information display control process is performed. As described above, when the information terminal 200 occupies most of the field of view of the user U1 through the transmissive display unit 131, before displaying the additional information 131s on the transmissive display unit 131, a main video reduction display request is transmitted to the information terminal 200. The information terminal 200 reduces the size of the main video 200m and displays a blank video 200b (monochromatic fixed image) in the empty part in response to the main video reduction display request.

[0108] By performing such control, although a part of the additional information 131s overlaps with the video part of the information terminal 200, since the overlapping part is the blank video 200b (monochromatic fixed image), the visibility of the additional information 131s is not impaired.

[0109] As described above, according to the present embodiment, when viewing an information terminal that displays additional information superimposed on a main image through the display unit of the HMD, even when the additional information can only be displayed at a position overlapping with the information terminal, it is possible to provide an HMD having additional information display control that can display the additional information without impairing its visibility.

[0110] Although the above embodiments have been described, the present invention is not limited to the above-described embodiments and includes various modifications. For example, the viewing target may be a PC screen or the like in addition to a TV or a smartphone, and the additional information may be subtitles, a sub-screen, or other superimposed data. All of these belong to the scope of the present invention. Also, the numerical values, messages, etc. appearing in the text and the drawings are merely examples, and using different ones will not impair the effects of the present invention.

[0111] Furthermore, the functions and the like of the present invention described above may be realized in hardware by designing part or all of them, for example, by means of an integrated circuit. Alternatively, they may be realized in software by a microprocessor unit, a CPU, etc. interpreting and executing an operation program for realizing each function and the like. Also, the scope of software implementation is not limited, and hardware and software may be used in combination. Also, part or all of each function may be realized by a server. Note that the server only needs to be able to cooperate with other components via communication and execute functions. For example, it may be a local server, a cloud server, an edge server, a net service, etc., and its form does not matter. Information such as programs, tables, and files for realizing each function may be stored in a memory, a recording device such as a hard disk or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD, or may be stored in a device on a communication network.

[0112] Also, the programs described in each processing example may be independent programs, or a plurality of programs may constitute one application program. Also, the order of performing each process may be changed for execution.

[0113] Also, the control lines and information lines shown in the figures show those considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. In fact, it may be considered that almost all components are interconnected.

[0114] Also, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Also, for part of the configuration of each embodiment, addition, deletion, and replacement with other configurations are possible.

Description of Reference Numerals

[0115] 100: HMD (Head-Mounted Display device), 101: Main control unit, 103: RAM, 110: Storage unit, 120: Operation input unit, 130: Image processing unit, 131: Transmissive display unit, 131R: Information terminal visual recognition area, 140: Audio processing unit, 151: Position information acquisition unit, 160: Sensor unit, 170: Communication unit, 180: Expansion interface unit, 200: Information terminal, 300: Network, 310: Server device, 1001: Basic operation program, 1002: Application, 1009: Various data storage areas, 1101: Basic operation function unit, 1111: Image acquisition unit, 1112: Additional information presence / absence analysis unit, 1113: Additional information acquisition unit, 1114: Display control unit, 1199: Temporary storage area, U1: User, U1S: Field of view

Claims

1. A head mounted display device having a transmissive display unit, an image acquisition unit that acquires a visual field range image that is an image corresponding to a visual field range of a user; an additional information presence / absence analysis unit that analyzes the acquired visual field range image, checks whether an information terminal is included in the user's visual field range, and checks whether the display image of the information terminal included in the user's visual field range is a main image with additional information superimposed thereon; an additional information acquisition unit that requests the information terminal or the content server to transmit the additional information and receives the additional information; a display control unit that detects an information terminal visible area in the transmissive display unit, which is an area where the information terminal is visible, and displays the received additional information in a position that does not overlap with the information terminal visible area of ​​the transmissive display unit; A head mounted display device comprising:

2. The head mounted display device according to claim 1, The head mounted display device, wherein the additional information acquisition unit transmits a request to the information terminal to stop displaying the additional information.

3. The head mounted display device according to claim 1, The head-mounted display device is characterized in that the additional information presence / absence analysis unit sends a confirmation request to the information terminal as to whether the display image of the information terminal is a main image with additional information superimposed thereon, and confirms whether the display image of the information terminal is a main image with additional information superimposed thereon based on a reply from the information terminal.

4. The head mounted display device according to claim 1, The additional information presence / absence analysis unit extracts a display image portion of the information terminal from an image of the user's field of view, performs a search process on a network based on the extracted display image, obtains information regarding a content server that supplies a main image to the information terminal, transmits the display image portion of the information terminal to the content server, receives information regarding the main image supplied by the content server to the information terminal from the content server, and confirms whether the display image of the information terminal is a main image with additional information superimposed thereon based on an image of the display image portion of the information terminal and the information regarding the main image supplied by the content server to the information terminal.

5. The head mounted display device according to claim 1, The head mounted display device according to claim 1, wherein the display control unit displays the additional information at a fixed position even when the position of the information terminal on the transmissive display unit is moved relatively.

6. The head mounted display device according to claim 1, A head-mounted display device characterized in that, when the position of the information terminal on the transparent display unit moves relatively, the display position of the additional information is displayed at a fixed position relative to the information terminal.

7. The head mounted display device according to claim 1, The head-mounted display device is characterized in that the additional information presence / absence analysis unit accepts a user's selection instruction for any object on the display image of the information terminal based on the acquired field of view range image, checks the position of the selected object within the display image of the information terminal, and checks whether the selected object is the additional information superimposed on the main image.

8. The head mounted display device according to claim 7, A head mounted display device characterized in that the user's selection instruction is made by detecting a tap operation or a blinking action of a touch sensor while the user's gaze is fixed on an object to be selected.

9. The head mounted display device according to claim 7, The head-mounted display device is characterized in that the display control unit displays all of the additional information in a position that does not overlap with the information terminal visual area, triggered by the user's selection of one piece of additional information by a selection instruction from the user.

10. The head mounted display device according to claim 1, The head mounted display device, wherein the additional information acquisition unit transmits a request for reducing the size of the additional information to the information terminal.

11. The head mounted display device according to claim 1, The head mounted display device, wherein the additional information acquisition unit transmits a request to move the additional information to the information terminal.

12. The head mounted display device according to claim 1, A head-mounted display device characterized in that the additional information presence / absence analysis unit checks whether or not there is a linked information terminal on the network, and if there is a linked information terminal, the additional information acquisition unit sends an additional information transmission reservation to the linked information terminal, the additional information presence / absence analysis unit sends to the linked information terminal a message indicating that the linked information terminal is included within the user's field of view, and the additional information acquisition unit receives the additional information sent from the linked information terminal via the network.

13. The head mounted display device according to claim 1, the additional information presence / absence analysis unit confirms whether or not the linked information terminal is present on the network, analyzes the visual field range image of the user, and confirms whether or not the linked information terminal is included in the visual field range of the user; the additional information acquisition unit requests the linked information terminal to transmit the main image and receives the main image; A head-mounted display device, characterized in that the display control unit displays the main image received from the linked information terminal at a predetermined position on the transmissive display unit.

14. The head mounted display device according to claim 1, The display image of the information terminal included within the user's visual field has additional information displayed in an L-shape in an empty area obtained by reducing a main image, the additional information presence / absence analysis unit confirms whether or not a display image of the information terminal included within a visual field range of the user has the L-shaped additional information; the additional information acquisition unit requests transmission of the additional information and at the same time requests enlarged display of the main image; The head mounted display device, wherein the display control unit displays the additional information at a position that does not overlap with the information terminal visible area of ​​the transmissive display unit.

15. A head mounted display device having a transmissive display unit, A first image input unit for acquiring an outside image; The Communications Department and A control unit is provided. The control unit is acquiring a visual field range image corresponding to a visual field range of a user from the image acquired by the first image input unit; Analyzing the acquired visual field range image, confirming whether or not an information terminal is included in the visual field range of the user, and confirming whether or not the display image of the information terminal included in the visual field range of the user is a main image with additional information superimposed thereon; requesting the information terminal or the content server to transmit the additional information via the communication unit, and receiving the additional information; A head-mounted display device characterized in that an information terminal visible area, which is an area in which the information terminal is viewed, is detected in the transparent display unit, and the received additional information is displayed in a position that does not overlap with the information terminal visible area of ​​the transparent display unit.

Images