Head-mounted display device

Abstract

To provide a HMD that allows display control of additional information of an information terminal to be performed on the display unit of the HMD when the information terminal displaying a main image with additional information superimposed thereon is viewed via the display unit of the HMD.SOLUTION: A head-mounted display device having a transmissive display unit includes: an image acquisition unit that acquires a field of view range image, which is an image corresponding to a user's field of view range; an additional information presence / absence analysis unit that analyzes the acquired field of view range image, checks whether an information terminal is included in the user's field of view range, and checks whether the display image of the information terminal included in the user's field of view range is a main image with additional information superimposed thereon; an additional information acquisition unit that requests the information terminal or a content server to transmit the additional information and receives the additional information; and a display control unit that detects an information terminal viewing area, which is the area in which the information terminal is viewed in the transmissive display unit, and displays the received additional information in a position that does not overlap with the information terminal viewing area of the transmissive display unit.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a head-mounted display device (hereinafter referred to as HMD: Head Mounted Display), and particularly relates to the display of video additional information.

Background Art

[0002] An HMD equipped with a display and worn on the head to view information has been increasingly used in recent years in fields such as gaming and work support.

[0003] Here, when a user views an information terminal (TV, smartphone, etc.) that displays additional information superimposed on a main image through the transmissive display unit of the HMD, the main image may be difficult to view due to the additional information overlapping the main image.

[0004] As the background art in this technical field, there is Patent Document 1. Patent Document 1 discloses a video additional information display control device and its operation method for displaying a video on a display device, displaying additional information on an HMD, and preventing the additional information from overlapping the display object in the video.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In Patent Document 1, the video additional information display control device controls the display position of the additional information so that the additional information does not overlap the display object in the video displayed on the HMD, and the HMD simply displays the additional information based on the control of the video additional information display control device. Therefore, the subject of control is the video additional information display control device, and there is a problem that a video additional information display control device is required separately from the HMD.

[0007] In view of the above problems, the present invention aims to provide an HMD that allows the display of additional information on an information terminal to be controlled on the HMD's display unit when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit. [Means for solving the problem]

[0008] One example of the present invention is a head-mounted display device having a transparent display unit, comprising: an image acquisition unit that acquires a field-of-view image, which is an image corresponding to the user's field of view; an additional information presence / absence analysis unit that analyzes the acquired field-of-view image to confirm whether or not an information terminal is included within the user's field of view and confirms whether or not the display image of the information terminal included within the user's field of view is a main image with additional information superimposed on it; an additional information acquisition unit that requests the transmission of additional information from an information terminal or content server and receives the additional information; and a display control unit that detects an information terminal viewing area, which is the area in the transparent display unit where the information terminal is viewed, and displays the received additional information in a position that does not overlap with the information terminal viewing area of ​​the transparent display unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an HMD that allows for the control of the display of additional information. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing the hardware configuration of the HMD in Example 1. [Figure 2] This shows the appearance and mounting diagram of the HMD in Example 1. [Figure 3] This is a functional configuration diagram of the HMD in Example 1. [Figure 4] This is a system configuration diagram for Example 1. [Figure 5] This is a flowchart of the HMD's additional information display control process in Example 1. [Figure 6]Figure 5 shows a detailed flowchart of the additional information presence / absence analysis process. [Figure 7] Figure 5 shows a detailed flowchart of the analysis process for the presence or absence of other additional information. [Figure 8] Figure 5 is an explanatory diagram of the detection process for the information terminal's viewing area. [Figure 9] This is an explanatory diagram of the HMD's additional information display control process in Example 1. [Figure 10A] This figure illustrates the display control of additional information on the HMD when the user is facing the information terminal directly in Example 2. [Figure 10B] This figure illustrates the display control of additional information on the HMD in Example 2 when the user turns their head to the left and the position of the information terminal within the transparent display unit moves relatively to the right. [Figure 10C] This figure illustrates other display controls for additional information on the HMD in Example 2, when the user turns their head to the left and the position of the information terminal within the transparent display unit moves relatively to the right. [Figure 11] This is a flowchart of the HMD's additional information display control process in Example 3. [Figure 12A] This diagram illustrates the display control of the HMD's additional information when the user makes a selection instruction while their gaze is fixed on additional information 1 in Example 3. [Figure 12B] This figure illustrates the display control of the HMD's additional information when the user makes a selection instruction while their gaze is fixed on additional information 2 in Example 3. [Figure 12C] This figure illustrates other display controls for the HMD's additional information when the user makes a selection instruction while their gaze is fixed on additional information 1 in Example 3. [Figure 13A] This figure illustrates the first process of controlling the display of additional information in Example 4. [Figure 13B] This figure illustrates the second process for controlling the display of additional information in Example 4. [Figure 13C] This figure illustrates the third process for controlling the display of additional information in Example 4. [Figure 14] It is a flowchart of the additional information display control process of the HMD in Example 5. [Figure 15A] It is a diagram for explaining the first state of the display control of additional information in Example 5. [Figure 15B] It is a diagram for explaining the second state of the display control of additional information in Example 5. [Figure 15C] It is a diagram for explaining the third state of the display control of additional information in Example 5. [Figure 16] It is a flowchart of the additional information display control process of the HMD in Example 6. [Figure 17A] It is a diagram for explaining the first state of the display control of additional information in Example 6. [Figure 17B] It is a diagram for explaining the second state of the display control of additional information in Example 6. [Figure 17C] It is a diagram for explaining the third state of the display control of additional information in Example 6. [Figure 18] It is an explanatory diagram of the additional information display control process of the HMD in Example 7. [Figure 19] It is an explanatory diagram of the additional information display control process of the HMD in Example 8.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

Examples

[0012] FIG. 1 is a block diagram showing the hardware configuration of the HMD 100 in the present embodiment. In FIG. 1, 101 is a main control unit (CPU / MCU, etc.), 102 is a bus which is a transmission / reception path of commands and data, 103 is a RAM which serves as a work area when executing a basic operation program and other operation programs, and 110 is a storage unit which is a non-volatile storage medium such as FlashROM / EEPROM / SSD / HDD.

[0013] 120 is an operation input section which is a user operation interface, and has operation keys 121 such as switches, power keys and volume keys, and touch sensors 122 such as a touchpad.

[0014] 130 is an image processing unit such as an image (video) processor, and includes a transparent display unit (display) 131, an image signal processing unit (image (video) signal processor) 132, a first image input unit 133 which is an external camera for forward shooting, and a second image input unit 134 which is an internal camera for gaze detection.

[0015] 140 is an audio processing unit (audio processor) and includes an audio output unit (speaker) 141, an audio signal processing unit (audio signal processor) 142, and an audio input unit (microphone) 143.

[0016] 151 is a position information acquisition unit (for receiving position information signals such as GPS signals). 152 is a distance measuring unit and includes a distance sensor and a depth sensor. 160 is a sensor unit and includes a gyro sensor 161, a geomagnetic sensor 162, and an acceleration sensor 163.

[0017] 170 is a communication unit (communication interface) consisting of a LAN (Wi-Fi®) communication unit, a mobile communication unit, a Bluetooth® communication unit, etc., and 180 is an expansion interface unit, such as a USB interface, which is used for data transmission and reception, charging, etc.

[0018] Figure 2 shows the external appearance and wearing diagram of the HMD in this embodiment. In Figure 2, the top figure is a top view of the HMD 100 worn on the head of user U1, the middle figure is a front view, and the bottom figure is a side view. As shown in Figure 2, the HMD 100 in this embodiment is a goggle-type, optically transmitted HMD with a window on the front.

[0019] In Figure 2, 131 is a transmissive display unit, and the outside world can be directly viewed through (through) the transmissive display unit 131.

[0020] 133L is the left external camera, which is the first image input unit (L), and 133R is the right external camera, which is the first image input unit (R). Note that there may be only one first image input unit (for example, one first image input unit 133 in the center). Alternatively, distance measurement may be performed by stereo imaging with 133L and 133R. In this case, the distance measurement unit 152 may not be necessary. Alternatively, a single image input device may be provided that combines the functions of the first image input unit 133 and the distance measurement unit 152, allowing the camera to acquire image information and distance information simultaneously with the image input device.

[0021] Furthermore, 122L is a touch sensor (L) and 122R is a touch sensor (R) (not shown), and there may be only one touch sensor on either the left or right side. Also, 141L is an audio output unit (L) (stereo speaker L) and 141R is an audio output unit (R) (stereo speaker R) (not shown). Although not shown, a monaural microphone may be included as an audio input unit 143.

[0022] Figure 3 is a functional configuration diagram of the HMD in this embodiment. In Figure 3, the storage unit 110 consists of a basic operation program 1001, which is a basic program such as the OS; an application 1002, which includes applications (hereinafter abbreviated as "apps") that execute the functions of this embodiment and other apps; and various data storage areas 1009, which are areas for storing various operation settings and various information (videos / still images / audio, etc.).

[0023] The programs / applications stored in the storage unit 110 are loaded into the RAM 103, and the main control unit 101 executes the loaded programs / applications, thereby executing the functional units and other functional units of this embodiment.

[0024] In Figure 3, the RAM 103 contains a functional description of the executable programs to be deployed. Specifically, it includes a basic operation function unit 1101 that executes the basic operation program of the HMD 100, an image acquisition unit 1111 that acquires an image (field of view image) corresponding to the field of view of user U1 wearing the HMD 100, an additional information presence / absence analysis unit 1112 that analyzes whether the display image of the information terminal 200 included in the field of view image acquired by the image acquisition unit 1111 is a main image with additional information superimposed on it, an additional information acquisition unit 1113 that acquires data related to the additional information superimposed on the display image of the information terminal 200 included in the field of view image acquired by the image acquisition unit 1111, and a display control unit 1114 that detects the area (information terminal viewing area) within the transparent display unit 131 of the information terminal 200 included in the field of view image acquired by the image acquisition unit 1111, and controls the display position and presence / absence of the additional information based on the detection result. The RAM 103 also has a temporary storage area 1199 which is a temporary storage area for various information created / acquired by the application.

[0025] Figure 4 is a system configuration diagram in this embodiment. In Figure 4, the HMD 100 is connected via a network 300 such as the Internet to information terminals 200s such as smartphones and tablet devices, information terminals 200t such as TVs and other display devices, and server devices 310 such as content servers.

[0026] Figure 5 is a flowchart of the HMD's additional information display control process in this embodiment. In Figure 5, first, in step S101, the image acquisition unit 1111 acquires a field of view image, which is an image corresponding to the user U1's field of view, from the first image input unit 133. Next, in S102, the additional information presence / absence analysis unit 1112 analyzes the user U1's field of view image acquired in the S101 process to check whether the information terminal 200 is included within the user U1's field of view. If the information terminal 200 is included within the user U1's field of view, the process proceeds to S103. If the information terminal 200 is not included within the user U1's field of view, the process ends.

[0027] In S103, the additional information presence / absence analysis unit 1112 analyzes the field of view image of user U1 acquired in the processing of S101 and checks whether the display image of the information terminal 200 included within the field of view of user U1 is a main image with additional information superimposed on it. If the additional information is superimposed on the main image, the process proceeds to S104. If the additional information is not superimposed on the main image (only the main image is present), the process ends.

[0028] In S104, the additional information acquisition unit 1113 requests the information terminal 200 or the server device 310, which is a content server, to send additional information. Then, in S105, the additional information acquisition unit 1113 receives the additional information from the information terminal 200 or the server device 310. If the additional information cannot be received within a predetermined time, the process may be terminated. Then, in S106, the additional information acquisition unit 1113 sends a request to the information terminal 200 to stop displaying the additional information.

[0029] Then, in S107, the image acquisition unit 1111 acquires a field of view image from the first image input unit 133, which is an image corresponding to the field of view of user U1. Alternatively, the process in S107 may be omitted, and the field of view image acquired in the process of S101 may be used in subsequent processes.

[0030] Then, in S108, the display control unit 1114 analyzes the field of view image of user U1 acquired in the S107 process and detects the information terminal viewing area, which is the area within the transparent display unit 131 where the information terminal 200 is visible. Then, in S109, the display control unit 1114 displays the additional information received from the information terminal 200 or server device 310 in the S105 process at a predetermined position on the transparent display unit 131 that does not overlap with the information terminal viewing area detected in the S108 process. Note that if the additional information remains displayed on the information terminal 200 after the S106 process, the S107-S109 processes do not need to be performed.

[0031] Figure 6 is a detailed flowchart of the additional information presence / absence analysis process in S103 in Figure 5. In Figure 6, first, in step S121, the additional information presence / absence analysis unit 1112 sends a confirmation request via the communication unit 170 to the information terminal 200 connected on the same network 300, asking whether the image displayed on the information terminal 200 is a main image with additional information superimposed on it. The information terminal 200 communicates with the HMD 100 within a narrow-area network such as a LAN or Bluetooth.

[0032] Then, in S122, the additional information presence / absence analysis unit 1112 checks whether it has received a reply from the information terminal 200 to the confirmation request sent in the processing of S121. If a reply from the information terminal 200 is received within a predetermined time, the process proceeds to S123. If a reply from the information terminal 200 is not received within a predetermined time, the process proceeds to S125.

[0033] In S123, the additional information presence / absence analysis unit 1112 checks, based on the reply from the information terminal 200, whether the image displayed on the information terminal 200 is a main image with additional information superimposed on it. If the displayed image is a main image with additional information superimposed on it, the process proceeds to S124. If the displayed image is not a main image with additional information superimposed on it, the process proceeds to S125.

[0034] In S124, the additional information presence / absence analysis unit 1112 sets the additional information superposition flag to "1", and the process in S103 branches to "Yes".

[0035] In S125, the additional information presence / absence analysis unit 1112 sets the additional information superposition flag to "0", and the process in S103 branches to "No".

[0036] Figure 7 is a detailed flowchart of the additional information presence / absence analysis process in S103 of Figure 5. Figure 7 shows how to determine whether the display image of the information terminal 200 within the field of view is a main image with additional information superimposed on it, based not on the information terminal 200, but on information from the server device 310, which is a content server via the network 300.

[0037] In Figure 7, first, in step S131, the additional information presence / absence analysis unit 1112 extracts the display image portion of the information terminal 200 from the field of view image of user U1 acquired in the processing of S101, performs a search process on the network 300 based on the extracted display image, and obtains information about the server device 310 that supplies the main image (and / or additional information) to the information terminal 200. Then, in S132, the additional information presence / absence analysis unit 1112 checks whether it was able to obtain information about the server device 310 that supplies the main image (and / or additional information) to the information terminal 200 through the processing of S131. If information about the server device 310 was obtained, the process proceeds to S133. If information about the server device 310 was not obtained, the process proceeds to S136.

[0038] In S133, the additional information presence / absence analysis unit 1112 transmits the display image portion of the information terminal 200 extracted in the processing of S131 to the server device 310, and obtains information regarding the main image (and / or additional information) that the server device 310 will supply to the information terminal 200.

[0039] Then, in S134, the additional information presence / absence analysis unit 1112 checks whether the display image of the information terminal 200 is a main image with additional information superimposed on it, based on the image of the display image portion of the information terminal 200 extracted in the processing of S131 and the information on the main image (and / or additional information) that the server device 310 supplies to the information terminal 200, which was acquired in the processing of S133. If the display image is a main image with additional information superimposed on it, the process proceeds to S135. If the display image is not a main image with additional information superimposed on it, the process proceeds to S136.

[0040] In S135, similar to S124 in Figure 6, the additional information presence / absence analysis unit 1112 sets the additional information superposition flag to "1", and the process in S103 branches to "Yes".

[0041] In S136, similar to S125 in Figure 6, the additional information presence / absence analysis unit 1112 sets the additional information superposition flag to "0", and branches to "No" in the processing of S103.

[0042] Figure 8 is an explanatory diagram of the detection process of the information terminal viewing area in S108 in Figure 5. The upper part of Figure 8 shows the field of view of user U1 through the transparent display unit 131, and shows the real-world information terminal 200 that user U1 is viewing through the transparent display unit 131. The lower part of Figure 8 is a diagram explaining the information terminal viewing area within the transparent display unit 131, and shows the information terminal viewing area 131R detected by the process in S108 with respect to the real-world information terminal 200 that user U1 is viewing through the transparent display unit 131.

[0043] The information terminal viewing area 131R does not need to be a shape similar to the enclosure as shown in the figure; for example, it may be a simple rectangle. Furthermore, the information terminal viewing area 131R may only enclose the display image portion of the information terminal 200. Additionally, any additional information received from the information terminal 200 may not be displayed inside the information terminal viewing area 131R, but may be displayed anywhere outside of it.

[0044] Figure 9 is an explanatory diagram of the HMD's additional information display control process in this embodiment. As shown in the upper diagram of Figure 9, when viewing the display image of the information terminal 200 via the transparent display unit 131 of the HMD 100, if it is determined that the display image of the information terminal 200 has additional information (subtitle information, etc.) superimposed on the main image (program content, etc.), then (1) the HMD 100 requests the information terminal 200 to transmit the additional information and stop displaying the additional information. (2) The HMD 100 receives the additional information from the information terminal 200. (3) Based on the user's field of view image acquired by the HMD 100's camera, the HMD 100 detects the information terminal viewing area in the transparent display unit 131 of the HMD 100. Then, as shown in the lower diagram of Figure 9, (4) the HMD 100 displays the additional information received in (2) in a position that does not overlap with the information terminal viewing area 131R detected in (3) in the transparent display unit 131 of the HMD 100. If it is determined that only the main image is displayed and no additional information is superimposed, the above processes (1) to (4) will not be performed.

[0045] Furthermore, to determine whether or not additional information is superimposed on the main image, the HMD100 queries the information terminal 200 to ask whether the displayed image is a main image with additional information superimposed on it. Alternatively, the HMD100 queries the content server supplying the main image to the information terminal 200 regarding the main image and the additional information. Other methods of determination may also be used.

[0046] Furthermore, the display unit of the HMD100 may be opaque. In this case, in the display control described in (4) above, the user's field of view image acquired by the HMD100's camera is displayed on the opaque display unit, and the additional information received in (2) is superimposed and displayed in a position that does not overlap with the area where the information terminal is displayed.

[0047] Furthermore, the HMD100 may be of the goggle type shown in the illustration, or it may be of the glasses type, etc. Also, the information terminal 200 may be a TV placed in the room, a smartphone held in the hand, or other information device, etc.

[0048] As described above, according to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit, the additional information of the information terminal is displayed in a position on the HMD's display unit that does not overlap with the main image of the information terminal, and the display unit of the HMD can control the display of the additional information. [Examples]

[0049] In this embodiment, the details of the display position of the additional information in the additional information display control process will be described.

[0050] Figures 10A, 10B, and 10C illustrate the display control of additional information in this embodiment. In Figure 10A, user U1 is facing directly towards the information terminal 200, and the control in Embodiment 1 shows the additional information being displayed in a position within the transparent display unit 131 that does not overlap with the viewing area of ​​the information terminal. U1S indicates the field of view of user U1.

[0051] Figure 10B shows a first state in which, starting from the state in Figure 10A, user U1 turns their head to the left, and the position of the information terminal 200 within the transparent display unit 131 moves relatively to the right. In this case, the display position of the additional information within the transparent display unit 131 is the same as in Figure 10A. That is, the display position of the additional information is fixed at a predetermined position within the transparent display unit 131.

[0052] Figure 10C, like Figure 10B, shows a second state in which user U1 turns their head to the left from the state in Figure 10A, causing the position of the information terminal 200 within the transparent display unit 131 to move relatively to the right. In this case, the display position of the additional information within the transparent display unit 131 may also move to the right in accordance with the information terminal 200. That is, the display position of the additional information may be fixed in a position relative to the information terminal 200.

[0053] Furthermore, the display position of the additional information may be outside the viewing area of ​​the information terminal, and may be set to display the additional information in the background area where it is most easily visible. For example, if the background area consists of patterned curtains and a white wall, control may be implemented to display the additional information so that it overlaps with the white wall.

[0054] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit, it is possible to provide an HMD that allows for display control of the additional information on the HMD's display unit, so that the additional information of the information terminal is displayed in a position on the HMD's display unit that does not overlap with the main image of the information terminal, and in a position that is easy to view. [Examples]

[0055] This embodiment describes the process for controlling the display of additional information selected by the user.

[0056] Figure 11 is a flowchart of the HMD's additional information display control process in this embodiment. Note that the flowchart shown in Figure 11 is a modified version of the additional information display control process described in Embodiment 1.

[0057] In Figure 11, first, in step S201, the image acquisition unit 1111 acquires an image (field of view image) corresponding to the user U1's field of view from the first image input unit 133. Then, in S202, the additional information presence / absence analysis unit 1112 receives a selection instruction from user U1 for any object on the display image of the information terminal 200, based on the field of view image of user U1 acquired in the process of S201. Then, in S203, the additional information presence / absence analysis unit 1112 confirms the position of the object (selected object) selected by user U1 in the process of S202 within the display image of the information terminal 200.

[0058] Next, in S204, the additional information presence / absence analysis unit 1112 checks whether the selected object, whose position in the display image of the information terminal 200 was confirmed in processing S203, is additional information superimposed on the main image. If the selected object is additional information superimposed on the main image, the process proceeds to S205. If the selected object is not additional information superimposed on the main image, i.e., if it is part of the main image, the process ends.

[0059] In S205, the additional information acquisition unit 1113 requests the information terminal 200 or the server device 310, which is a content server, to send the selected object, which is the additional information. Then, in S206, the additional information acquisition unit 1113 receives the selected object (additional information) from the information terminal 200 or the server device 310.

[0060] In S207, the additional information acquisition unit 1113 sends a request to the information terminal 200 to stop displaying the selected object (additional information). Then, in S208, the image acquisition unit 1111 acquires an image (field of view image) corresponding to the field of view of user U1 from the first image input unit 133.

[0061] In S209, the display control unit 1114 analyzes the user U1's field of view image acquired in S208 and detects the area within the transparent display unit 131 where the information terminal 200 is visible (information terminal viewing area). Then, in S210, the display control unit 1114 displays the selected object (additional information) received from the information terminal 200 or server device 310 in S206 at a predetermined position on the transparent display unit 131 that does not overlap with the information terminal viewing area detected in S209.

[0062] The following is a specific example of user U1's selection instruction for any object in S202: While user U1 is looking at any object on the display image shown on the information terminal 200 via the transparent display unit 131, (1) the touch sensor 122 detects a tap operation by user U1, or (2) the second image input unit 134 detects user U1 blinking.

[0063] Furthermore, a specific example of the position confirmation process for the selected object in S203 is as follows: (1) The second image input unit 134 acquires an image of the user U1's eyes when a selection instruction is given by the user U1. (2) The acquired eye image is analyzed to detect the coordinates on the transparent display unit 131 where the user U1's line of sight intersects. (3) The detected coordinate position is calculated to determine which position in the field of view image of the user U1 acquired in the process of S201 corresponds to. (4) Based on the calculated position information within the field of view image, the position of the selected object at the user U1's line of sight within the display image of the information terminal 200 is calculated.

[0064] Figures 12A, 12B, and 12C illustrate the display control of additional information in this embodiment. In Figures 12A, 12B, and 12C, the left and right diagrams show the user's field of view through the transparent display unit of the HMD before and after the additional information display control, respectively.

[0065] Figure 12A shows a scenario where additional information 1 and additional information 2 exist, and user U1 makes a selection instruction while focusing their gaze on additional information 1. Specifically, a cross cursor is displayed at the position where the user's gaze is focused, acting as a UI (User Interface) that allows the user to select additional information or as a trigger for selecting additional information. The user makes a selection instruction by focusing their gaze on the additional information they wish to select, displaying the cross cursor, and then detecting a tap operation on the touch sensor or a blinking motion as described above. In Figure 12A, due to the additional information display control described in Figure 11, only additional information 1 is displayed in a position within the transparent display unit 131 that does not overlap with the information terminal's viewing area and is removed from the information terminal 200, while additional information 2 remains displayed on the information terminal 200, i.e., superimposed on the main image.

[0066] Unlike Figure 12A, Figure 12B shows the case where user U1 makes a selection instruction while focusing their gaze on additional information 2. In Figure 12B, due to the additional information display control explained in Figure 11, only additional information 2 is displayed in a position within the transparent display unit 131 that does not overlap with the information terminal viewing area and is removed from the information terminal 200, while additional information 1 remains superimposed on the main image.

[0067] Figure 12C shows another display control case where additional information 1 and additional information 2 exist, and user U1 makes a selection instruction while looking at additional information 1. In Figure 12C, by selecting one piece of additional information, all additional information superimposed on the main image of the information terminal 200 is displayed in a position within the transparent display unit 131 that does not overlap with the viewing area of ​​the information terminal and is removed from the information terminal 200. In this way, selecting one piece of additional information may be used as a trigger to display all additional information in a position that does not overlap with the viewing area of ​​the information terminal.

[0068] Furthermore, additional information displayed in a position within the transparent display unit 131 that does not overlap with the information terminal viewing area may be displayed with a modified shape according to the shape of the transparent display unit 131 and the position and size of the information terminal viewing area, through additional information display control.

[0069] Furthermore, even if there is only one piece of additional information superimposed on the main image of the information terminal 200, the additional information display control process may be started in response to a selection instruction from user U1.

[0070] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit, it is possible to provide an HMD that has a UI that allows selection of additional information, and displays the selected additional information in a position on the HMD's display unit that does not overlap with the main image of the information terminal, thereby enabling display control of the additional information on the HMD's display unit. [Examples]

[0071] This embodiment describes another example of the process for displaying additional information.

[0072] Figures 13A, 13B, and 13C illustrate the display control of additional information in this embodiment. In Figures 13A, 13B, and 13C, similar to Figure 12A, the left and right diagrams show the user's field of view through the transparent display unit of the HMD before and after the additional information display control, respectively.

[0073] Figure 13A shows the first processing of additional information on the information terminal side after additional information display control. That is, as shown in Figure 13A, after the additional information is displayed in a position within the transparent display unit 131 that does not overlap with the viewing area of ​​the information terminal by the additional information display processing control, the information terminal 200 side may stop displaying the additional information.

[0074] Figure 13B shows the second processing of additional information on the information terminal side after the additional information display control. That is, as shown in Figure 13B, after the additional information display processing control displays the additional information in a position that does not overlap with the viewing area of ​​the information terminal within the transparent display unit 131, the information terminal 200 side may reduce the size of the additional information to a size that does not interfere with the viewing of the main image. In this case, instead of a request to stop the display of additional information in processing S106 of Figure 5, a request to reduce the display of additional information is sent.

[0075] Figure 13C shows a third processing of the additional information on the information terminal side after the additional information display control. That is, as shown in Figure 13C, after the additional information is displayed in a position within the transparent display unit 131 that does not overlap with the viewing area of ​​the information terminal by the additional information display processing control, the information terminal 200 may move the additional information to a position that does not interfere with the viewing of the main image. In this case, in the processing of S106 in Figure 5, an additional information movement request is sent instead of an additional information display stop request. Note that the position to which the additional information is moved on the information terminal 200 may be a position where part of the additional information extends beyond the display area of ​​the information terminal 200, as shown in Figure 13C.

[0076] According to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit, the additional information of the information terminal is displayed in a position on the HMD's display unit that does not overlap with the main image of the information terminal, and the information terminal can control the display of the additional information so as not to interfere with the viewing of the main image. [Examples]

[0077] This embodiment describes an example of acquiring additional information from an information terminal via a network and controlling the display of that additional information.

[0078] Figure 14 is a flowchart of the HMD's additional information display control process in this embodiment. Note that the flowchart shown in Figure 14 is a modified version of the additional information display control process described in Embodiments 1 and 3.

[0079] In Figure 14, first, in step S301, the additional information presence / absence analysis unit 1112 checks whether there is an information terminal 200 that is already linked on the same network 300. If there is an information terminal 200 that is already linked, the process proceeds to S302. If there is no information terminal 200 that is already linked, the process ends.

[0080] Next, in S302, the additional information acquisition unit 1113 sends an additional information transmission reservation to the linked information terminal 200. Note that the additional information transmission reservation may be sent to the information terminal in advance, regardless of whether the additional information is displayed on the information terminal. Then, in S303, the image acquisition unit 1111 acquires an image (field of view image) corresponding to the field of view of user U1 from the first image input unit 133.

[0081] Next, in S304, the additional information presence / absence analysis unit 1112 analyzes the field of view image of user U1 acquired in processing S303 to check whether the linked information terminal 200 is included within user U1's field of view. If the linked information terminal 200 is included within user U1's field of view, the process proceeds to S305. If the linked information terminal 200 is not included within user U1's field of view, the process returns to S303.

[0082] In S305, the additional information presence / absence analysis unit 1112 transmits to the linked information terminal 200 that the linked information terminal 200 is within the user U1's field of view (information terminal visible state). Then, in S306, the additional information acquisition unit 1113 receives the additional information transmitted from the linked information terminal 200 in accordance with the information terminal visible state transmitted in the processing of S305. If the display image of the linked information terminal 200 does not have additional information superimposed on the main image, the unit may receive information to that effect and terminate the process. That is, when the information terminal 200 receives from the HMD 100 that it is in an information terminal visible state, if the display image of the information terminal 200 has additional information superimposed on the main image, it transmits the additional information to the HMD 100, and if the display image of the information terminal 200 does not have additional information superimposed on the main image, it transmits information to the HMD 100 to that effect.

[0083] Next, in S307, the additional information acquisition unit 1113 sends a request to the linked information terminal 200 to stop displaying the additional information. Then, in S308, the image acquisition unit 1111 acquires an image (field of view image) corresponding to the field of view of user U1 from the first image input unit 133. Alternatively, the process in S308 may be omitted, and the field of view image acquired in the process in S303 may be used in subsequent processes.

[0084] Next, in S309, the display control unit 1114 analyzes the user U1's field of view image acquired in the S308 process and detects the area within the transparent display unit 131 where the linked information terminal 200 is visible (information terminal viewing area). Then, in S310, the display control unit 1114 displays the additional information received from the linked information terminal 200 in the S306 process at a predetermined position on the transparent display unit 131 that does not overlap with the information terminal viewing area detected in the S309 process.

[0085] Figures 15A, 15B, and 15C illustrate the display control of additional information in this embodiment. Figure 15A illustrates the first state, where user U1 is facing left relative to the information terminal 200, and the entire information terminal 200 does not fit within user U1's field of view U1S, and the entire information terminal 200 does not fit within the transparent display unit 131. Furthermore, in the process of S304 in Figure 14, the linked information terminal 200 is not within user U1's field of view, the information terminal viewing status has not been transmitted to the linked information terminal 200, and additional information has not been received from the linked information terminal 200. Although not shown, the display image of the information terminal 200 is the main image with additional information superimposed on it.

[0086] Figure 15B illustrates the second state, showing the first state where, from the state in Figure 15A, user U1 turns their head to the right, and the entire information terminal 200 enters user U1's field of view U1S. In this case, it is the state immediately after the detection in process S304 of Figure 14 that the linked information terminal 200 has entered user U1's field of view, and no additional information has yet been received from the linked information terminal 200. The display image of the information terminal 200 is still the main image with the additional information superimposed on it.

[0087] Figure 15C is a diagram illustrating the third state, showing the state after processing from S305 onwards in Figure 14, starting from the state in Figure 15B. Specifically, Figure 15C shows the state in which the information terminal viewing status is transmitted to the linked information terminal 200 in the S305 process of Figure 14, additional information is received from the linked information terminal 200 in the S306 process, and in the S310 process, the additional information received from the linked information terminal 200 in the S306 process is displayed at a predetermined position on the transparent display unit 131 that does not overlap with the information terminal viewing area detected in the S309 process.

[0088] As described above, according to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the HMD's display unit, the additional information from the information terminal is acquired via the network and displayed on the HMD's display unit in a position that does not overlap with the main image of the information terminal, thereby providing an HMD that allows for display control of the additional information on the HMD's display unit.

[0089] In the above explanation, the process at S304 in Figure 14 determines that the state in which the entire information terminal 200 does not fit within the field of view U1S of user U1 is a state in which the linked information terminal 200 is not within the field of view of user U1, and the process at S304 in Figure 14 determines that the state in which the linked information terminal 200 is within the field of view of user U1 is a state in which the linked information terminal 200 is within the field of view of user U1. However, it is also possible to determine in the process at S304 in Figure 14 that the state in which the linked information terminal 200 is not within the field of view of user U1 is a state in which the linked information terminal 200 is not within the field of view of user U1, and to determine in the process at S304 in Figure 14 that the state in which the linked information terminal 200 is within the field of view of user U1 is a state in which the linked information terminal 200 is within the field of view of user U1. [Examples]

[0090] In this embodiment, other examples of display control of additional information in Example 5 will be described.

[0091] Figure 16 is a flowchart of the HMD's additional information display control process in this embodiment. Figure 16 may be implemented in parallel with Figure 14 of Embodiment 5, or independently.

[0092] In Figure 16, first, in step S321, similar to S303 in Figure 14, the image acquisition unit 1111 acquires an image (field of view image) corresponding to the field of view of user U1 from the first image input unit 133.

[0093] Next, in S322, the additional information presence / absence analysis unit 1112 analyzes the field of view image of user U1 acquired in the processing of S321 to check whether or not there is an information terminal 200 that is already linked on the network within user U1's field of view. If the linked information terminal 200 is included within user U1's field of view, the process returns to S321. If the linked information terminal 200 is not included within user U1's field of view, the process proceeds to S323.

[0094] In S323, the additional information acquisition unit 1113 requests the linked information terminal 200 to transmit the main image. It may also send a request to redisplay the additional information at the same time. Next, in S324, the additional information acquisition unit 1113 receives the main image from the linked information terminal 200. Then, in S325, the display control unit 1114 displays the main image received from the linked information terminal 200 in the S324 process at a predetermined position on the transparent display unit 131.

[0095] Figures 17A, 17B, and 17C illustrate the display control of additional information in this embodiment. Figure 17A is the same as Figure 15C and illustrates the first state in this embodiment, showing the state in which the entire information terminal 200 is within the field of view U1S of user U1. In this case, the additional information is displayed in a predetermined position that does not overlap with the viewing area of ​​the information terminal of the transparent display unit 131.

[0096] Figure 17B illustrates the second state, showing that from the state in Figure 17A, user U1 turns their head to the left, and the position of the information terminal 200 within the transparent display unit 131 moves relatively to the right. In other words, in the process of S322, it is detected that a portion of the information terminal 200 is outside the field of view U1S of user U1, but the additional information remains fixed and displayed in a predetermined position on the transparent display unit 131.

[0097] Figure 17C illustrates the third state, showing that user U1 has turned its head further to the left from the state in Figure 17B, and the information terminal 200 is no longer included in user U1's field of view U1S. In other words, the process in S322 detects that the information terminal 200 has moved out of user U1's field of view U1S, and the process in S324 shows the state in which the main image of the information terminal 200 received is displayed on the transparent display unit 131 as a substitute for the information terminal 200 that was being viewed via the transparent display unit 131.

[0098] Thus, in this embodiment, even if the position of the information terminal moves relative to the field of view, the additional information remains fixed and displayed at a predetermined position on the transparent display unit. Furthermore, even when the information terminal is no longer included in the field of view, the main image of the information terminal can still be displayed on the transparent display unit. [Examples]

[0099] This embodiment describes other examples of display control for additional information.

[0100] Figure 18 is an explanatory diagram of the HMD's additional information display control process in this embodiment. Figure 18 shows the user U1's field of view through the transparent display unit 131. The upper part of the figure shows the user viewing the information terminal 200 displaying additional information through the transparent display unit 131. The lower part of Figure 18 shows the state in which additional information is displayed on the transparent display unit 131 by the additional information display control.

[0101] In addition to being displayed superimposed on the main image, the additional information may also be displayed in a format similar to data broadcasting, for example, in an L-shape, in the area left empty when the main image is reduced in size, as shown in the upper diagram of Figure 18.

[0102] In the case of the display shape of the additional information as shown in the upper figure, in this embodiment, the additional information display control requests the information terminal 200 to transmit the additional information and at the same time requests an enlarged display of the main image. As shown in the lower figure of Figure 18, the main image is enlarged and the additional information is displayed in a position that does not overlap with the viewing area of ​​the information terminal of the transparent display unit 131.

[0103] As described above, according to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the display unit of the HMD, even if the additional information is superimposed in an L-shape, it is possible to provide an HMD with display control for additional information that can display the additional information of the information terminal in a position on the HMD's display unit that does not overlap with the main image of the information terminal. [Examples]

[0104] This embodiment describes other examples of display control for additional information.

[0105] Figure 19 is an explanatory diagram of the HMD's additional information display control process in this embodiment. In this embodiment, it is assumed that the distance between user U1 and information terminal 200 is close, and that the information terminal 200 occupies most of user U1's field of view via the transparent display unit 131. That is, the information terminal viewing area 131R occupies most of the transparent display unit 131, and this is an example of additional information display control processing in a case where it is difficult to display additional information in a position within the display area of ​​the transparent display unit 131 that does not overlap with the information terminal viewing area 131R.

[0106] The upper part of Figure 19 shows the state in which the information terminal 200 displaying the additional information is viewed through the transparent display unit 131, and represents the state before the additional information display control processing is performed. The information terminal 200 displays the main video 200m with the additional information 200s superimposed on it. As mentioned above, the information terminal 200 occupies most of the field of view of the user U1 through the transparent display unit 131.

[0107] The lower part of Figure 19 shows the state after the additional information display control process has been performed, with additional information displayed on the transparent display unit 131 by the additional information display control process. As mentioned above, if the information terminal 200 occupies most of the field of view of user U1 via the transparent display unit 131, a request for main image reduction display is sent to the information terminal 200 before the additional information 131s is displayed on the transparent display unit 131. In response to the main image reduction display request, the information terminal 200 reduces the size of the main image 200m and displays a blank image 200b (monochromatic fixed image) in the empty space.

[0108] By performing this control, a portion of the additional information 131s overlaps with the video portion of the information terminal 200, but since the overlapping portion is a blank video 200b (monochromatic fixed image), the readability of the additional information 131s is not impaired.

[0109] As described above, according to this embodiment, when viewing an information terminal that displays additional information superimposed on a main image via the display unit of the HMD, even if the additional information can only be displayed in a position that overlaps with the information terminal, it is possible to provide an HMD with display control for additional information that can be displayed without impairing the readability of the additional information.

[0110] Although embodiments have been described above, the present invention is not limited to the embodiments described above and includes various modifications. For example, the viewing target may be a PC screen or the like, in addition to a TV or smartphone, and the additional information may be subtitles, sub-screens, or other superimposed data. All of these fall within the scope of the present invention. Furthermore, the numbers and messages that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0111] Furthermore, some or all of the functions of the present invention described above may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, they may be implemented in software by having a microprocessor unit, CPU, etc., interpret and execute operating programs that realize each function. Moreover, the scope of software implementation is not limited, and hardware and software may be used in combination. Furthermore, some or all of each function may be implemented on a server. The server only needs to be able to execute functions in cooperation with other components via communication, and its form is not limited, for example, a local server, cloud server, edge server, network service, etc. Information such as programs, tables, files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or it may be stored on a device on a communication network.

[0112] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is executed may also be changed.

[0113] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is reasonable to assume that almost all components are interconnected.

[0114] Furthermore, the embodiments described above are detailed explanations provided to facilitate understanding of the present invention, and are not necessarily limited to those comprising all the described configurations. It is also possible to replace parts of the configuration of one embodiment with those of another embodiment, and to add configurations from other embodiments to a given embodiment. Additionally, it is possible to add, delete, or replace parts of the configurations in each embodiment with those of other embodiments. [Explanation of Symbols]

[0115] 100: HMD (Head-Mounted Display Device), 101: Main Control Unit, 103: RAM, 110: Storage Unit, 120: Operation Input Unit, 130: Image Processing Unit, 131: Transparent Display Unit, 131R: Information Terminal Viewing Area, 140: Voice Processing Unit, 151: Location Information Acquisition Unit, 160: Sensor Unit, 170: Communication Unit, 180: Expansion Interface Unit, 200: Information Terminal, 300: Network, 310: Server Device, 1001: Basic Operation Program, 1002: Application, 1009: Various Data Storage Areas, 1101: Basic Operation Function Unit, 1111: Image Acquisition Unit, 1112: Additional Information Presence / Absence Analysis Unit, 1113: Additional Information Acquisition Unit, 1114: Display Control Unit, 1199: Temporary Storage Area, U1: User, U1S: Field of View

Claims

1. A method for displaying information on a head-mounted display device, The head-mounted display device's image acquisition unit acquires an external image, which is an image of the real world; An information terminal determination step that determines whether or not an information terminal is included in the acquired external image, If it is determined that an information terminal is included in the external image, the additional information determination step determines whether or not the information terminal is displaying an image in which additional information is superimposed on the main image. If the information terminal determines that it is displaying an image in which the additional information is superimposed on the main image, the additional information transmission request step involves requesting the information terminal to transmit the additional information. An additional information receiving step, which involves receiving the additional information from the information terminal, The head-mounted display device includes an information terminal viewing area detection step for detecting an information terminal viewing area, which is an area for viewing the information terminal in the display unit of the head-mounted display device, A display step in which the received additional information is displayed in a position that does not overlap with the viewing area of ​​the information terminal on the display unit, has A method for displaying information characterized by the following features.

2. The information display method according to claim 1, An information display method characterized by further comprising a step of transmitting a request to stop displaying the additional information to the information terminal.

3. The information display method according to claim 1, The system further includes an additional information superimposition confirmation request step, which involves sending a request to the information terminal to confirm whether or not it is displaying an image in which the additional information is superimposed on the main image. An information display method characterized by controlling whether or not to execute the additional information transmission request step, the additional information reception step, the information terminal viewing area detection step, and the display step based on the reply from the information terminal in response to the confirmation request.

4. The information display method according to claim 1, An information display method characterized in that, even when the position of the information terminal in the display unit moves relative to it, the display position of the additional information in the display step is fixed and displayed.

5. The information display method according to claim 1, An information display method characterized in that, when the position of the information terminal in the display unit moves relative to the information terminal, the display position of the additional information in the display step is fixed relative to the information terminal.

6. The information display method according to claim 1, An information display method characterized by further comprising a step of transmitting a request for reduced display of additional information to transmit a request for reduced display of the additional information to the information terminal.

7. The information display method according to claim 1, The information display method is characterized by further comprising an additional information transfer request transmission step of transmitting a request to transfer the additional information to the information terminal.

Images