Head-mounted display device and control method for head-mounted display device

Abstract

A head-mounted display device and a control method for the head-mounted display device appropriately prompt a user to pay attention to the possibility of danger posed by an object in the actual space or the like to the user. The HMD (100) includes: a display unit (20) that transmits an external scene; a line-of-sight detection unit that detects the line-of-sight direction of the user (U); a determination unit that performs the following processing: determining the attention level of a first object by calculating the risk level of the first object and correcting the risk level of the first object according to the line-of-sight direction, and determining the attention level of a second object by calculating the risk level of the second object and correcting the risk level of the second object according to the line-of-sight direction; and a display control unit that causes the display unit (20) to display a first image related to the first object when the attention level of the first object is higher than the attention level of the second object.

Description

Technical Field

[0001] The present invention relates to a head-mounted display device and a control method for a head-mounted display device. Background Art

[0002] Conventionally, a technique for notifying a user wearing a head-mounted display device (HMD: Head Mounted Display) of danger has been proposed. Patent Document 1 discloses the following structure: A moving object moving in the outdoor scene is identified, and a display mode of a display image is set according to the speed at which the moving object approaches the user.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-117175

[0004] In the structure disclosed in Patent Document 1, it is difficult to notify a user of an object moving at a low speed or a stationary object. Therefore, it is desired to more appropriately prompt the user to pay attention to the possibility of danger posed by an object in the actual space or the like to the user. Summary of the Invention

[0005] One aspect of the present disclosure is a head-mounted display device including: a display unit that transmits an outdoor scene including a first object and a second object; a line-of-sight detection unit that detects a line-of-sight direction of a user; a time calculation unit that calculates a first time taken for the first object to reach the user and a second time taken for the second object to reach the user; and a determination unit that acquires first information related to the first object and second information related to the second object, calculates a risk level of the first object based on the first information and the first time, calculates a risk level of the second object based on the second information and the second time, and when an attention level of the first object based on the line-of-sight direction and the risk level of the first object is higher than an attention level of the second object based on the line-of-sight direction and the risk level of the second object, the display unit displays a first image related to the first object.

[0006] Another aspect of the present disclosure is a control method for a head-mounted display device, comprising the following steps: obtaining first information related to a first object and second information related to a second object; detecting the line-of-sight direction of a user; calculating a first time taken for the first object to reach the user and a second time taken for the second object to reach the user; calculating a risk level of the first object based on the first information and the first time; calculating a risk level of the second object based on the second information and the second time; and causing a display unit to display a first image related to the first object when a attention level of the first object based on the line-of-sight direction and the risk level of the first object is higher than an attention level of the second object based on the line-of-sight direction and the risk level of the second object. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 FIG. is a diagram showing the structure of the HMD.

[0008] Figure 2 FIG. is an external view of the HMD.

[0009] Figure 3 FIG. is a block diagram of the HMD.

[0010] Figure 4 FIG. is a functional block diagram of the control unit.

[0011] Figure 5 FIG. is a schematic diagram showing a situation where the line-of-sight direction of the user is detected.

[0012] Figure 6 FIG. is a diagram showing a display example based on the HMD.

[0013] Figure 7 FIG. is a flowchart showing the operation of the HMD.

[0014] Figure 8 FIG. is a flowchart showing the operation of the HMD.

[0015] Figure 9 FIG. is a flowchart showing the operation of the HMD.

[0016] Figure 10 FIG. is an explanatory diagram showing an example of a transition of the display.

[0017] REFERENCE SIGNS LIST

[0018] 10: Controller; 11: Main body; 12: Dial operation unit; 13: Center key; 14: Operation panel; 15: Up and down keys; 17: LED display unit; 18: Power switch; 19: USB connector; 20: Display unit; 21: Right holding unit; 22: Right display unit; 23: Left holding unit; 24: Left display unit; 26: Right light guide plate; 27: Front frame; 28: Left light guide plate; 30: Headphones; 32: Right earphone; 34: Left earphone; 40: Connection cable; 42: Connector; 46: Audio connector; 61: External camera; 63: Microphone; 64: Distance sensor; 65: Illuminance sensor; 66: Thermal imager; 67: LED indicator; 68: 8: inner camera; 90, 90A, 90B: object; 100: HMD; 118: memory; 120: control unit; 121: non-volatile storage unit; 125: processor; 151: control program; 152: setting value storage unit; 153: threshold storage unit; 154: identification data storage unit; 155: damage database; 170: operation unit; 171: basic control unit; 173: distance detection unit; 174: temperature detection unit; 175: time calculation unit; 176: line of sight detection unit; 177: determination unit; 178: motion detection unit; 179: input detection unit; 301, 301A, 301B, 301C: attention image; U: user. DETAILED DESCRIPTION

[0019] [1. Overall structure of HMD]

[0020] Figure 1 is a diagram showing the structure of the HMD 100 . Figure 2 1 is an external view of the HMD 100 and is a perspective view of the side opposite to the face of the user U.

[0021] The HMD 100 is a head mounted display device which is worn on the head of the user U. HMD is an abbreviation of Head Mounted Display.

[0022] The HMD 100 is an optical transmission type display device that allows the user to directly see the external scene while seeing the virtual image. Here, the external scene is the scenery outside the user U wearing the HMD 100, and refers to the scene of the actual space that can be seen by the naked eye even when the user U does not wear the display unit 20. The user U is the user of the HMD 100 and can also be called an operator.

[0023] The HMD 100 includes a display unit 20 that is worn on the head of a user U and a controller 10 that controls the display unit 20 . The display unit 20 allows the user U to see a virtual image when worn on the head of the user U. The controller 10 functions as a control device for the user U to operate the HMD 100 .

[0024] In the following description, for convenience, the virtual image that the display unit 20 shows to the user U is called a display image. Expressing that the HMD 100 emits image light based on image data from the display unit 20 is referred to as displaying an image. The image is not limited to a static image and also includes a dynamic image or video. This structure is just an example. For example, the HMD 100 can also be light-impermeable. In this case, the HMD 100 can be, for example, a so-called video see-through type display device. Specifically, the HMD 100 can also have the following structure: it has a display unit 20 that does not have light transmissivity, and by displaying the captured image of the outer camera 61 described later on the display unit 20, the user U can indirectly see the outdoor scene.

[0025] The controller 10 has a box-shaped main body 11. The main body 11 has various switches and the like, serving as operators for accepting the operations of the user U. In the present embodiment, the display unit 20 has a glasses shape. The display unit 20 has a main body composed of a right holding part 21, a left holding part 23, and a front frame 27. A right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 are provided on the main body of the display unit 20.

[0026] The right holding part 21 and the left holding part 23 extend rearward from both end parts of the front frame 27, holding the display unit 20 on the head of the user U. Set the end part on the right side of the user U when wearing the display unit 20 among the two end parts of the front frame 27 as the end part ER, and the end part on the left side as the end part EL.

[0027] The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. In the worn state of the display unit 20, the right light guide plate 26 is located in front of the right eye of the user U. The left light guide plate 28 is located in front of the left eye of the user U.

[0028] The right display unit 22 and the left display unit 24 are modules obtained by unitizing an optical unit and peripheral circuits. The right display unit 22 is mounted on the right holding part 21, and the left display unit 24 is mounted on the left holding part 23. The right display unit 22 and the left display unit 24 emit image light based on image data.

[0029] The right light guide plate 26 and the left light guide plate 28 are optical components formed of a light-transmissive resin or the like. For example, the right light guide plate 26 and the left light guide plate 28 are prisms. The right light guide plate 26 guides the image light output by the right display unit 22 toward the right eye of the user U. The left light guide plate 28 guides the image light output by the left display unit 24 toward the left eye of the user. Thereby, the image light enters the two eyes of the user U, and the user U can see the image.

[0030] The image light guided by the right light guide plate 26 and the external light transmitted through the right light guide plate 26 are incident on the right eye of the user U. The image light guided by the left light guide plate 28 and the external light transmitted through the left light guide plate 28 are incident on the left eye of the user U. That is, the HMD 100 causes the image light corresponding to the image processed inside and the external light to be incident on the eyes of the user U overlappingly. The user U can see the outside scene through the right light guide plate 26 and the left light guide plate 28. Moreover, the user U can observe the image based on the image light overlappingly with the outside scene.

[0031] A light-shielding cover that attenuates the external light incident on the right and left eyes of the user U may also be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. This light-shielding cover may also be an electronic light-shielding cover capable of electrically adjusting the light transmittance.

[0032] An illuminance sensor 65 is arranged on the front frame 27. The illuminance sensor 65 receives the external light from the front of the user U wearing the display unit 20.

[0033] An outer camera 61 is arranged on the front frame 27. In the Figure 1 example, the outer camera 61 is located on the end ER side of the front frame 27. The outer camera 61 is a digital camera that captures a shooting range including the front of the user U. The outer camera 61 is arranged at a position that does not block the external light transmitted through the right light guide plate 26 and the left light guide plate 28. The position of the outer camera 61 is not limited to the Figure 1 example. The outer camera 61 may be arranged on the end EL side, or may be arranged at the connection portion between the right light guide plate 26 and the left light guide plate 28. The outer camera 61 corresponds to an example of the shooting unit.

[0034] The outer camera 61 is a digital camera having a shooting element such as a CCD or a CMOS and a shooting lens, etc. The outer camera 61 in this embodiment is a single-lens reflex camera, but may also be composed of a stereo camera. The outer camera 61 performs shooting according to the control of the control unit 120 described later and outputs the captured image data to the control unit 120. The outer camera 61 has a shooting lens. The shooting lens of the outer camera 61 may also be a so-called wide-angle lens. This wide-angle lens may also be a lens called an ultra-wide-angle lens or a semi-wide-angle lens. The shooting lens of the outer camera 61 may be a single-focus lens, a zoom lens, or a lens group composed of multiple lenses. CCD is the abbreviation of Charge Coupled Device (charge-coupled device). CMOS is the abbreviation of Complementary Metal-Oxide-Semiconductor (complementary metal oxide semiconductor).

[0035] An LED indicator 67 is arranged on the front frame 27. The LED indicator 67 is arranged near the outer camera 61 at the end ER and lights up during the operation of the outer camera 61 to notify that shooting is in progress. LED is the abbreviation of Light Emitting Diode (light emitting diode).

[0036] A distance sensor 64 is provided on the front frame 27. The distance sensor 64 detects the distance from the display unit 20 to the measurement object located in front of the user U. The measurement object is a real object or structure in the actual space. The distance sensor 64 can be, for example, a light reflection type distance sensor. Specifically, a sensor having a light source such as an LED or a laser diode and a light receiving unit for receiving the reflected light reflected by the measurement object from the light source can be cited. In addition, the distance sensor 64 can also be an ultrasonic type distance sensor. That is, the distance sensor 64 can also be a structure having a sound source for emitting ultrasonic waves and a detection unit for receiving the ultrasonic waves reflected by the measurement object. In addition, the distance sensor 64 can also be a laser rangefinder, which is also called a distance measuring sensor.

[0037] As described later, the HMD 100 performs SLAM processing by using at least one of the outer camera 61 and the distance sensor 64 through the control unit 120. The control unit 120 generates an environmental map of the objects around the user U wearing the display unit 20 and determines the own position of the user U in the environmental map. The distance sensor 64 is a sensor for performing SLAM processing. Specifically, it can also be a sensor constituting a LIDAR system. The distance sensor 64 is not limited to a sensor that detects the distance between the measurement object and the display unit 20. For example, it can also be a laser rangefinder. SLAM is the abbreviation of Simultaneous Localization and Mapping (simultaneous localization and mapping). LIDAR is the abbreviation of Light Detection and Ranging (light detection and ranging), Laser Imaging Detection and Ranging (laser imaging detection and ranging). When the processor 125 can perform SLAM only by using the outer camera 61, the HMD 100 can also be a structure without the distance sensor 64.

[0038] The HMD 100 can also have a gas sensor or other environmental sensor that detects the gas components around the user U together with or instead of the distance sensor 64.

[0039] In addition, although not shown, a thermal imager 66 is provided in the front frame 27. The thermal imager 66 is a camera that captures infrared light and outputs image data of an infrared image. The infrared image captured by the thermal imager 66 represents the temperature distribution within the imaging range of the thermal imager 66. The imaging range of the thermal imager 66 overlaps with the imaging range of the outer camera 61.

[0040] The controller 10 and the display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to the connector 42 of the main body 11.

[0041] The connection cable 40 has an audio connector 46. A headset 30 is connected to the audio connector 46. The headset 30 includes a right earphone 32 and a left earphone 34 that form a stereo earphone, and a microphone 63.

[0042] The right earphone 32 is worn on the right ear of the user U. The left earphone 34 is worn on the left ear of the user U. The right earphone 32 and the left earphone 34 are in-ear headphones or ear canal headphones. The right earphone 32 and the left earphone 34 may also be over-ear headphones that contact the auricle via ear cups. The right earphone 32 and the left earphone 34 output voice according to the voice signal output from the voice interface 181 described later.

[0043] The microphone 63 collects voice and outputs a voice signal to the voice interface 181. The microphone 63 can be a mono microphone or a stereo microphone. The microphone 63 can be a directional microphone or an omnidirectional microphone.

[0044] The controller 10 has a dial operation unit 12, a center key 13, an operation panel 14, up / down keys 15, an LED display unit 17, and a power switch 18. These can be referred to as operated parts operated by the user U. These operated parts are arranged on the surface of the main body 11. These operated parts are operated by the user's hand / finger, for example.

[0045] The LED display unit 17 is provided on the main body 11. The LED display unit 17 is an LED indicator that indicates the operation state of the HMD 100. The LED display unit 17 is covered by a transmissive part that allows light to pass through. The cover of the LED display unit 17 forms a part of the surface of the main body 11. When the LED display unit 17 emits light, the light passes through the transmissive part. Thus, characters, marks, patterns, etc. formed on the transmissive part can be seen. A touch sensor is arranged on the LED display unit 17 overlapping with the transmissive part, and the touch sensor detects the contact of the user U's hand / finger. The combination of the LED display unit 17 and the touch sensor functions as a software key.

[0046] The power switch 18 is a switch that toggles the on / off state of the power supply of the HMD 100.

[0047] The main body 11 has a USB connector 19 as an interface for connecting the controller 10 to an external device. USB is an abbreviation for Universal Serial Bus.

[0048] An inner camera 68 is provided in the front frame 27, and the inner camera 68 faces the face of the user U when the display unit 20 is worn by the user U. The HMD 100 of the present embodiment has a pair of inner cameras 68. Specifically, it has an inner camera 68 facing the right eye of the user U and an inner camera 68 facing the left eye of the user U. The inner camera 68 captures the right eye and left eye of the user U by infrared light and visible light respectively. The control unit 120 can determine the directions of the line of sight of the right eye and the line of sight of the left eye of the user U by using the captured images of the inner camera 68. In addition, the HMD 100 detects the sizes of the pupils of the right eye and the left eye by analyzing the captured images of the inner camera 68. The HMD 100 can also determine the constricted or dilated state of the pupils of the user U based on the change in the size of the pupils. In addition, the HMD 100 can also detect the open / closed state of the eyelids of the right eye and the left eye based on the captured images of the inner camera 68.

[0049] Figure 3 It is a block diagram showing the structure of each component constituting the HMD 100.

[0050] The controller 10 has a processor 125. The processor 125 is composed of a CPU, an MPU, etc. The memory 118 and the non-volatile storage unit 121 are connected to the processor 125. The memory 118 is, for example, a RAM, which forms a work area for temporarily storing data and programs. The non-volatile storage unit 121 is composed of semiconductor storage elements such as magnetic storage devices, flash ROMs, or other types of non-volatile storage devices. The non-volatile storage unit 121 stores non-volatilely the programs executed by the processor 125 and the data processed by the processor 125. CPU is an abbreviation for Central Processing Unit, MPU is an abbreviation for Micro Processing Unit, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.

[0051] An operation unit 170 is connected to the processor 125 as an input device. A 6-axis sensor 111, a magnetic sensor 113, and a GPS receiver 115 are connected to the processor 125 as sensors.

[0052] The processor 125 is connected to the communication unit 117, the voice interface 181, the external memory interface 191, the USB connector 19, the sensor hub 193, and the FPGA 194. These components function as interfaces with the outside. In the following descriptions and drawings, the interface is abbreviated as I / F. FPGA is an abbreviation for Field Programmable Gate Array.

[0053] The controller 10 has a control board. The processor 125 is mounted on the control board. The 6-axis sensor 111, the magnetic sensor 113, the GPS receiver 115, the communication unit 117, the memory 118, the non-volatile storage unit 121, etc. may also be mounted on the control board. The external memory interface 191, the USB connector 19, the sensor hub 193, the FPGA 194, and the interface 197 may also be mounted on the control board. The connector 42 and the USB connector 19 may also be mounted on the control board.

[0054] The memory 118 constitutes a work area for temporarily storing programs executed by the processor 125 and data processed by the processor 125, etc. The non-volatile storage unit 121 is composed of a semiconductor memory device such as a flash memory. The non-volatile storage unit 121 stores programs executed by the processor 125 and data processed by the processor 125.

[0055] The operation unit 170 detects operations on the touch sensor, the dial operation unit 12, the center key 13, the operation panel 14, the up / down keys 15, and the power switch 18 configured in the LED display unit 17. The operation unit 170 outputs an operation signal corresponding to the operation to the processor 125. The operation unit 170 lights, blinks, or turns off the LED display unit 17 under the control of the processor 125.

[0056] The 6-axis sensor 111 is an example of a motion sensor that detects the motion of the controller 10. The motion sensor may be replaced with an inertial sensor or a motion sensor. The 6-axis sensor 111 has a 3-axis acceleration sensor and a 3-axis gyro sensor. The magnetic sensor 113 is, for example, a 3-axis geomagnetic sensor. The 6-axis sensor 111 may also be an IMU obtained by modularizing the acceleration sensor and the gyro sensor. IMU is an abbreviation for Inertial Measurement Unit. In addition, it may also be a structure obtained by modularizing the 6-axis sensor 111 and the magnetic sensor 113.

[0057] The GPS receiver unit 115 receives GPS signals through a GPS antenna (not shown). The GPS receiver unit 115 detects or calculates the coordinates of the current position of the controller 10 based on the GPS signals. GPS is an abbreviation for Global Positioning System.

[0058] The 6-axis sensor 111, the magnetic sensor 113, and the GPS receiver unit 115 output their output values to the processor 125 at a pre-specified sampling period. In addition, the 6-axis sensor 111, the magnetic sensor 113, and the GPS receiver unit 115 may also output their detected values to the processor 125 according to the request of the processor 125.

[0059] The communication unit 117 is a communication device that performs wireless communication with external devices. The communication unit 117 includes an antenna (not shown), an RF circuit, a baseband circuit, a communication control circuit, etc. The communication unit 117 performs wireless communication according to standards such as wireless LANs including Bluetooth and Wi-Fi, for example. RF is an abbreviation for Radio Frequency. Bluetooth is a registered trademark. Wi-Fi is a registered trademark.

[0060] The voice interface 181 is connected to the right earphone 32, the left earphone 34, and the microphone 63 via the audio connector 46. The voice interface 181 outputs voice signals to the right earphone 32 and the left earphone 34 respectively under the control of the processor 125 to output voice. The voice interface 181 outputs the voice signal input from the microphone 63 to the processor 125. The voice interface 181 may also have a converter for converting analog voice signals and digital voice data. In this case, digital voice data is input and output between the voice interface 181 and the processor 125.

[0061] The HMD 100 is capable of processing stereo voice. Specifically, through the voice interface 181, it is possible to make the right earphone 32 and the left earphone 34 output 2-channel stereo voice including channels corresponding to the right ear and the left ear of the user U respectively.

[0062] The external memory interface 191 is an interface capable of connecting a removable memory device, and includes, for example, a memory card slot for mounting a cartridge recording medium and being able to read data, and an interface circuit.

[0063] The interface 197 connects the sensor hub 193 and the FPGA 194 to the display unit 20.

[0064] The sensor hub 193 obtains the detection values of various sensors included in the display unit 20 and outputs them to the processor 125. The FPGA 194 performs processing of data transmitted and received between the components of the processor 125 and the display unit 20 and transmission via the interface 197.

[0065] As described above, the display unit 20 is in the shape of glasses, and the right holding unit 21 and the left holding unit 23 are members similar to the temple arms of glasses. The right holding unit 21 and the left holding unit 23 can each rotate relative to the front frame 27. For example, the right holding unit 21 and the left holding unit 23 are each connected to the front frame 27 by a hinge structure. In a state where the user U wears the display unit 20, the right holding unit 21 and the left holding unit 23 are opened at an angle that conforms to the size of the user U's head.

[0066] Via the connection cable 40 and the internal wiring of the display unit 20 (not shown), the right display unit 22 and the left display unit 24 are respectively connected to the controller 10.

[0067] The right display unit 22 includes an OLED unit 221 that emits image light. The image light emitted by the OLED unit 221 is guided to the right light guide plate 26 by an optical system including a lens group and the like. The left display unit 24 includes an OLED unit 241 that emits image light. The image light emitted by the OLED unit 241 is guided to the left light guide plate 28 by an optical system including a lens group and the like. OLED is an abbreviation for Organic Light Emitting Diode.

[0068] The OLED units 221 and 241 each include an OLED panel and a drive circuit that drives the OLED panel. The OLED panel is a self-luminous display panel that emits light through organic electroluminescence. The OLED panel is formed, for example, by arranging light-emitting elements that respectively emit red, green, and blue light in a matrix. The drive circuit, under the control of the processor 125, performs selection of the light-emitting elements of the OLED panel and energization of the light-emitting elements, causing the light-emitting elements of the OLED panel to emit light. Thereby, the OLED units 221 and 241 form image light, which is incident on the right eye and left eye of the user U via the right light guide plate 26 and the left light guide plate 28.

[0069] The right display unit 22 includes a display unit substrate 210. An interface 211, a receiving unit 213, and an EEPROM 215 are mounted on the display unit substrate 210. The interface 211 is connected to the interface 197. The interface 211 connects the receiving unit 213, the EEPROM 215, the outer camera 61, the illuminance sensor 65, and the LED indicator 67 to the controller 10. The receiving unit 213 receives data input from the controller 10 via the interface 211. In the drawings, the receiving unit 213 is abbreviated as Rx.

[0070] The EEPROM 215 stores data. For example, the EEPROM 215 stores data related to the light-emitting characteristics or display characteristics of the OLED units 221 and 241, data related to the characteristics of the sensors included in the right display unit 22 or the left display unit 24, and the like. The data stored in the EEPROM 215 can be read by the processor 125. EEPROM is an abbreviation for Electrically Erasable Programmable ROM.

[0071] A signal representing captured image data or the capture result of the outer camera 61 is input from the outer camera 61 to the interface 211. A measurement result obtained by measuring the distance from the distance sensor 64 to an object within the detection range of the distance sensor 64 is input to the interface 211. A detection value corresponding to the amount of received light and / or the light-receiving intensity is input from the illuminance sensor 65 to the interface 211. Image data of an infrared image is input from the thermal imager 66 to the interface 211.

[0072] The LED indicator 67 lights up and goes out in accordance with a signal input via the interface 211. The inner camera 68 performs imaging and outputs a signal representing the captured image data or the capture result to the interface 211. The receiving unit 213 receives data transmitted by the processor 125 via the interface 211. The receiving unit 213 outputs the image data received via the interface 211 to the OLED unit 221.

[0073] The left display unit 24 includes a display unit substrate 230. The interface 231 and the receiving unit 233 are mounted on the display unit substrate 230. The 6-axis sensor 235 and the magnetic sensor 237 are mounted on the display unit substrate 230. The interface 231 connects the receiving unit 233, the 6-axis sensor 235, and the magnetic sensor 237 to the controller 10. The receiving unit 233 receives data input from the controller 10 via the interface 231.

[0074] The 6-axis sensor 235 is an example of a motion sensor that detects the motion of the display unit 20. The 6-axis sensor 235 includes a 3-axis acceleration sensor and a 3-axis gyro sensor. The 6-axis sensor 235 may also be an IMU obtained by modularizing the above-mentioned sensors. The magnetic sensor 237 is, for example, a 3-axis geomagnetic sensor. The 6-axis sensor 235 and the magnetic sensor 237 output detection values or detection data to the interface 231. These detection values or detection data are output to the processor 125 via the interface 231.

[0075] The outer camera 61, distance sensor 64, illuminance sensor 65, thermal imager 66, inner camera 68, 6-axis sensor 235, and magnetic sensor 237 are connected to the sensor hub 193 of the controller 10. Control signals are input to these respective sensors from the sensor hub 193. In addition, the LED indicator 67 is connected to the sensor hub 193.

[0076] The sensor hub 193 sets the sampling period and initializes each sensor under the control of the processor 125. The sensor hub 193 performs power-on of each sensor, transmission of control data, acquisition of detection values, etc., in accordance with the sampling period of each sensor. The sensor hub 193 outputs the detection values of each sensor to the processor 125 at a preset timing. The sensor hub 193 starts and stops the power-on of the LED indicator 67 under the control of the processor 125, and lights or blinks the LED indicator 67 in accordance with the start and end times of shooting by the outer camera 61.

[0077] [Structure of the control unit of the HMD]

[0078] Figure 4 is a functional block diagram of the control unit 120 of the HMD 100. The control unit 120 includes a memory 118, a non-volatile storage unit 121, and a processor 125. The control unit 120 may also include an EEPROM 215.

[0079] The non-volatile storage unit 121 stores a control program 151. The control program 151 is a program executed by the processor 125. The non-volatile storage unit 121 has a set value storage unit 152, a threshold storage unit 153, an identification data storage unit 154, and a damage database 155. In the following description and drawings, the database is abbreviated as DB. The set value storage unit 152, the threshold storage unit 153, the identification data storage unit 154, and the damage DB 155 are logical or virtual storage units provided in the storage area of the non-volatile storage unit 121.

[0080] The set value storage unit 152 stores various set values related to the operation of the HMD 100. When parameters, matrices, arithmetic expressions, LUTs, etc. are used in the control of the HMD 100 by the processor 125, the set value storage unit 152 stores them. LUT is an abbreviation for LookUp Table.

[0081] The threshold storage unit 153 stores thresholds used in the processing of the processor 125. For example, the threshold storage unit 153 stores the distance threshold, time threshold, display threshold, and disengagement time threshold described later. For example, as will be referred to later Figure 9As described above, a plurality of disengagement time thresholds are stored corresponding to the values of the damage information. The thresholds stored in the threshold storage unit 153 can be input by operating the operation unit 170, or can be written into the threshold storage unit 153 during the manufacturing process of the HMD 100. In addition, the control program 151 can be executed by the processor 125 to dynamically generate thresholds and store them in the threshold storage unit 153. In addition, the threshold storage unit 153 can also store candidates for a plurality of thresholds selected by the processor 125.

[0082] The recognition data storage unit 154 stores data for the processor 125 to recognize an object from the captured image of the external camera 61. The object includes a movable object that can move, a building, a device fixed to the road, the road surface, and the shape of the ground itself. Movable objects include, for example, moving bodies such as trams, cars, and motorcycles. In addition, movable objects are furniture, household appliances, tools, daily necessities, including other movable objects, and include animals and people other than the user U. Devices include walls, handrails, and installations such as shelves and ornaments installed on these. The shape of the road surface and the ground itself includes artificial structures such as steps, slopes, drainage grooves, drainage outlets, and inspection holes. In addition, the shape of the road surface and the ground itself can also include naturally occurring shapes such as unevenness, water accumulation, cracks, and depressions on the road surface.

[0083] As described later, the processor 125 performs a process of cropping an image of an object from the captured image of the external camera 61. In this process, the processor 125 executes a process of detecting an image of an object by pattern matching, a process of detecting an object using an image analysis algorithm, a process of detecting an object included in the captured image by SLAM processing, etc. The data required for these processes is stored in the recognition data storage unit 154.

[0084] The damage DB 155 is a database containing damage information related to damage caused by the object recognized by the processor 125. Damage caused by an object means damage that the user U may suffer due to the object. This can be replaced with damage to the user U caused by the object.

[0085] Damage that the user U may suffer due to an object means damage such as injury to the user U's body when the user U comes into contact with or encounters the object. For example, it can be cited as the fall / roll of the user U, the fall of the user U, the violent collision of the user U with the object, the flying / falling of the object, the damage / collapse of the object, the violent collision of the object with the user U. In addition to this, it can be cited as the user U's body being clamped by the object, the user U's body being cut by the object, the user U falling into the water, the user U's contact with a high-temperature or low-temperature object, radiation caused by radiation, damage caused by harmful light, gas poisoning, hypoxia, electric shock, traffic accidents, etc.

[0086] The damage DB 155 stores damage information in association with the type and name of the object, and this damage information indicates the degree of damage that the user U may suffer from the object. For example, the damage information stored in the damage DB 155 is an index that combines the occurrence probability of the damage that the user U may suffer from the object and the severity of the damage that the user U may suffer from the object. The damage information may also be a numerical value. Specifically, the damage information may also be an index obtained by quantifying the degree of damage. The damage information may also be a numerical value that represents the degree of damage step by step. In this case, the damage information may be replaced with a damage level.

[0087] The occurrence probability of the damage is determined according to statistical indicators for each combination of the type of the object and the type of the damage. The severity of the damage is the severity of the injury suffered by the user U when the above-mentioned damage occurs. The severity is determined in advance according to, for example, the size of the injury that may occur to the body of the user U, indicators during the treatment period, and the presence or absence of sequelae. When one object may be related to multiple damages, the damage DB 155 stores damage information that takes into account the occurrence probability and severity of all damages related to the object. The damage information stored in the damage DB 155 corresponds to an example of the first information and the second information. That is, the damage information corresponding to the first object detected by the distance detection unit 173 corresponds to an example of the first information, and the damage information corresponding to the second object corresponds to an example of the second information.

[0088] The control unit 120 includes a basic control unit 171, a display control unit 172, a distance detection unit 173, a temperature detection unit 174, a time calculation unit 175, a line-of-sight detection unit 176, a determination unit 177, an action detection unit 178, and an input detection unit 179. These respective functional units are configured by the cooperation of software and hardware by the processor 125 executing the control program 151.

[0089] The basic control unit 171 executes the basic function of controlling each component of the HMD 100. The basic control unit 171 executes startup processing when the power of the HMD 100 is turned on and initializes each component of the HMD 100. When the power of the controller 10 is turned off, the basic control unit 171 executes shutdown processing and stops the HMD 100.

[0090] The display control unit 172 controls the display unit 20 to display various screens including images and characters in a manner visible to the user U.

[0091] The distance detection unit 173 detects an object located around the user U based on at least any one of the captured image of the outer camera 61 and the detection result of the distance sensor 64. The distance detection unit 173 detects the distance between the detected object and the user U. In addition, the distance detection unit 173 may, for example, also perform SLAM processing, generate an environmental map of the objects around the user U, and determine the own position of the user U in the environmental map. Any one of the objects detected by the distance detection unit 173 corresponds to an example of the first object, and any object other than the first object corresponds to an example of the second object.

[0092] The temperature detection unit 174 detects the temperature of the object detected by the distance detection unit 173 based on the infrared image captured by the thermal imager 66. The temperature detection unit 174 determines the position of the object in the infrared image using at least any one of the detection result obtained by the distance detection unit 173 detecting the object and the captured image of the outer camera 61 used by the distance detection unit 173 in the detection of the object. The temperature detection unit 174 sets the temperature at the determined position as the temperature of the object. When the distance detection unit 173 detects a plurality of objects, the temperature detection unit 174 detects the temperature for each of the plurality of objects detected by the distance detection unit 173.

[0093] The time calculation unit 175 calculates the time until the object detected by the distance detection unit 173 comes into contact with the user U. Specifically, the time calculation unit 175 obtains the change in the distance between the object detected by the distance detection unit 173 and the user U, and calculates the relative speed between the object and the user U based on the obtained change. The time calculation unit 175 calculates the time until the distance between the object and the user U becomes zero based on the relative speed between the object and the user U and the distance between the object and the user U. Here, the distance detection unit 173 actually detects the distance between the object and the outer camera 61 as the distance between the object and the user U. This distance is the distance between the object and the display unit 20, but can actually be regarded as the distance between the object and the user U. Therefore, hereinafter, the distance detected by the distance detection unit 173 will be regarded as the distance between the object and the user U for explanation.

[0094] The distance between the object and the user U being zero means that the position of the object overlaps with the position of the user U or the position of the object is very close to the position of the user U. In the process of calculating the time until the distance between the object and the user U becomes zero, the time calculation unit 175 does not distinguish whether the object moves or the user U moves. That is, the time until the distance between the object and the user U becomes zero is calculated based on the relative speed between the object and the user U regardless of whether the object moves or the user U moves.

[0095] The time until the distance between the object and the user U becomes zero represents the time until the body or clothes of the user U come into contact with or meet the object. For example, when the object is a moving body, when the distance between the object and the user U becomes zero, the object comes into contact with the user U. When the object is a stationary device, the distance between the object and the user U being zero means that the user U comes into contact with the object. When the object is an unevenness on the ground, the distance between the object and the user U being zero means that the user reaches the position of the object.

[0096] When the distance detection unit 173 detects a plurality of objects, the time calculation unit 175 calculates the time until the distance between each of the plurality of objects and the user U becomes zero. Specifically, the time calculation unit 175 calculates the first time taken for the first object detected by the distance detection unit 173 to reach the user U and the second time taken for the second object detected by the distance detection unit 173 to reach the user U, respectively.

[0097] The line-of-sight detection unit 176 detects the direction of the line of sight of the user U by obtaining and analyzing the captured image of the inner camera 68.

[0098] Figure 5 It is a schematic diagram showing the situation of detecting the direction of the user's line of sight.

[0099] In Figure 5 it, the reference numeral OB represents the object, the reference numeral RE represents the right eye of the user U, and the reference numeral RD represents the direction of the line of sight of the right eye RE. The reference numeral LE represents the left eye of the user U, and the reference numeral LD represents the direction of the line of sight of the left eye LE.

[0100] The line-of-sight detection unit 176 detects the direction of the user U's line of sight. Specifically, the line-of-sight detection unit 176 analyzes the captured image obtained by the inner camera 68 capturing the right eye RE to detect the line-of-sight direction RD. Similarly, the line-of-sight detection unit 176 analyzes the captured image obtained by the inner camera 68 capturing the left eye LE to detect the line-of-sight direction LD. The position VP where the line-of-sight direction RD intersects with the line-of-sight direction LD corresponds to the position at which the user U is gazing. This position is referred to as the gaze position VP. In other words, the line-of-sight direction RD is the direction from the right eye RE of the user U towards the gaze position VP, and the line-of-sight direction LD is the direction from the left eye LEE of the user U towards the gaze position VP. The line-of-sight detection unit 176 detects the direction VD from the reference position of the display unit 20 towards the gaze position VP as the line-of-sight direction VD. The reference position is a position predetermined as a reference for the position of the display unit 20, for example, the center in the left-right direction of the front frame 27 or the center of the pair of inner cameras 68. The line-of-sight direction VD represents the direction of the user U's binocular-based line of sight obtained by synthesizing the line-of-sight direction RD and the line-of-sight direction LD. In other words, the line-of-sight direction VD is the direction from the center of the display unit 20 towards the gaze position VP.

[0101] When the user U gazes at the object OB, the line-of-sight direction RD and the line-of-sight direction LD of the user U intersect at the position of the object OB. That is, the gaze position VP is within the range overlapping with the object OB. In other words, when the gaze position VP is a position overlapping with the object OB, it can be said that the user U is gazing at the object OB.

[0102] The line-of-sight detection unit 176 can also detect the distance D2 from the right eye RE and the left eye LE of the user U to the gaze position VP. In this case, by comparing the distance to the object OB detected by the distance detection unit 173 with the distance D2 detected by the line-of-sight detection unit 176, it is possible to more accurately determine whether the gaze position VP is a position overlapping with the object OB. Here, as indicated by reference numeral D1 in the attached drawing, the distance detected by the distance detection unit 173 is the distance from the display unit 20 at the installation position of the outer camera 61 to the object OB. Therefore, even when the gaze position VP coincides with the object OB, as Figure 5 shown, there is a difference between the distance D1 and the distance D2. However, this difference is small compared to the size of the user U's body, etc., so accurate determination can be made by comparing the distance D1 and the distance D2 individually.

[0103] The determination unit 177 calculates the risk level of the object detected by the distance detection unit 173. The risk level is determined based on the damage information stored in the damage DB155 and the distance to the object detected by the distance detection unit 173. The risk level is an index indicating the degree of risk that the object potentially poses to the user U.

[0104] For example, the determination unit 177 calculates the risk level by multiplying the numerical value of the damage information by the distance detected by the distance detection unit 173. The determination unit 177 may also numerically convert the distance detected by the distance detection unit 173 in stages. In this case, as described above, the damage information can be set to a value obtained by numerically converting the size of the damage in stages. Therefore, by multiplying the value of the damage information by the value obtained by numerically converting the distance in stages, the risk level can be easily calculated.

[0105] The determination unit 177 determines the attention level of the object by correcting the risk level of the object. Specifically, the determination unit 177 corrects the risk level of the object according to whether the fixation position VP overlaps with the position of the object, and thereby determines the attention level. The situation where the fixation position VP overlaps with the position of the object means that the line-of-sight direction VD of the user U is the direction corresponding to the position of the object.

[0106] The attention level is an index for determining whether the user U should pay attention to the object. In the present embodiment, the attention level is a value numerically converted in stages. Here, when the risk level of the object is a value numerically converted in stages, the risk level may be directly set as the attention level.

[0107] For example, when the fixation position VP overlaps with the position of the object, the determination unit 177 corrects the risk level of the object to a lower value and sets the corrected risk level as the attention level. When the fixation position VP does not overlap with the position of the object, the determination unit 177 directly sets the risk level of the object as the attention level, for example.

[0108] In addition, when the distance detection unit 173 detects a plurality of objects, the determination unit 177 calculates the risk levels for the plurality of objects detected by the distance detection unit 173 respectively. Specifically, the determination unit 177 calculates the risk level of the first object detected by the distance detection unit 173 and corrects the risk level, thereby determining the attention level of the first object. In addition, the determination unit 177 calculates the risk level of the second object detected by the distance detection unit 173 and corrects the risk level, thereby determining the attention level of the second object. The determination unit 177 sets the object that overlaps with the fixation position VP among the plurality of objects as the object to be fixated on. The determination unit 177 corrects the risk level of the object to be fixated on to a value lower than the risk level of the object that is not the object to be fixated on, and sets the corrected risk level as the attention level. In addition, for example, the determination unit 177 sets the attention level of the object that is not the object to be fixated on to the same value as the risk level of the object. For example, when the fixation position VP is a position that overlaps with the first object, the determination unit 177 sets the risk level of the second object as the attention level of the second object, and corrects the attention level of the first object to be lower than the attention level of the second object.

[0109] The determination unit 177 corrects the attention level according to the time calculated by the time calculation unit 175 for the object. For example, when the first time calculated by the time calculation unit 175 is shorter than the second time and the damage information of the first object is a value higher than the damage information of the second object, the determination unit 177 may also correct the attention level of the first object to be higher than the attention level of the second object.

[0110] The determination unit 177 may also calculate the risk level of the object according to the temperature of the object detected by the temperature detection unit 174. For example, when the temperature of the object detected by the temperature detection unit 174 is higher than or lower than the reference temperature determined based on the possibility of causing damage to the user U, the determination unit 177 may also add the values of the damage information to calculate the risk level. The reference temperature determined based on the possibility of causing damage to the user U can be set to the temperature at which the user U may be scalded when contacting the object, the temperature at which the user U may be frostbitten when contacting the object, and may also include the reference temperature on the high-temperature side and the reference temperature on the low-temperature side.

[0111] The motion detection unit 178 detects the motion of the user U's body. The motion detection unit 178 uses the captured image of the outer camera 61 to detect the motion of the user U. For example, the motion detection unit 178 detects the motion of a part of the user U's body that does not coincide with the movement of the display unit 20. Here, a part of the user U's body is, for example, an arm, a hand, a finger, a leg or a foot. The motion detection unit 178 is not limited to the captured image of the outer camera 61, and may also use any one or more of the detection results of the 6-axis sensor 111 and the 6-axis sensor 235 to detect the motion of the user U's body. The motion detection unit 178 may also detect the motion of the entire body of the user U.

[0112] The time calculation unit 175 can also calculate the time until the distance between the object and the user U becomes zero based on the motion of the user U detected by the motion detection unit 178. As described above, the time calculation unit 175 calculates the time based on the relative speed between the object and the user U and the distance between the object and the user U. In this process, the time calculation unit 175 adds the motion of the user U detected by the motion detection unit 178 to the movement of the display unit 20 to obtain the relative speed between the object and the user U. Thereby, the time until the distance between the object and the user U becomes zero can be calculated more accurately. In addition, for example, when the motion detected by the motion detection unit 178 is a motion approaching the object, the time calculation unit 175 may also add the motion detected by the motion detection unit 178 to the movement of the display unit 20 to obtain the relative speed between the object and the user U.

[0113] The input detection unit 179 accepts the operation of the user U on the operation unit 170. The input detection unit 179 may also detect an input based on the gesture of the user U according to the captured image of the outer camera 61 and the detection results of the 6-axis sensor 111 and the 6-axis sensor 235.

[0114] Figure 6 FIG. is a diagram showing a display example based on the HMD 100.

[0115] When the attention level determined by the determination unit 177 is higher than a preset display reference, the display control unit 172 notifies the user U to pay attention to the object. As a notification method, a method of outputting voice from the right earphone 32 and the left earphone 34, and a method of operating a vibrator (not shown) can be cited. In the present embodiment, a method of displaying an image for prompting the user U to pay attention at a display position corresponding to the object as the object to be prompted by the display unit 20 is adopted.

[0116] The reference numeral VA is the image displayed by the display unit 20 and the range of the outdoor scene that the user U sees through the display unit 20. In other words, the field of view when the user U sees the outdoor scene through the display unit 20 corresponds to the range VA.

[0117] In the present embodiment, an example is shown in which the range in which the user U sees the images displayed on the right light guide plate 26 and the left light guide plate 28 coincides with the range in which the user U sees the outdoor scene through the display unit 20. This is merely an example. For example, the images displayed on the right light guide plate 26 and the left light guide plate 28 may also be smaller than the range in which the user U sees the outdoor scene through the display unit 20. In this case, the images displayed on the right light guide plate 26 and the left light guide plate 28 also overlap with the outdoor scene that the user U sees through the right light guide plate 26 and the left light guide plate 28.

[0118] Figure 6 An example is shown in which the user U is located inside a building and there are a plurality of objects 90A, 90B in the real space. Hereinafter, without distinguishing between the objects 90A, 90B, they are denoted as object 90. The same applies to the attention image 301 described later.

[0119] The object 90A is a staircase, and the object 90B is a self-propelled robotic vacuum cleaner. There is a risk that the user U may fall or collide with the object 90A. There is a risk that the user U may collide with the object 90B and a risk that the user U may fall due to the collision.

[0120] The attention image 301A is an image that prompts the user U to pay attention to the object 90A. The attention image 301A is displayed at a display position that overlaps with the position where the object 90A is seen in the range VA or at a display position near the position where the object 90A is seen. This position is referred to as the display position corresponding to the object 90A. The attention image 301B is an image that prompts the user U to pay attention to the object 90B and is displayed at the display position corresponding to the object 90B in the range VA. The attention image 301A and the attention image 301B may be the same image or different images. When the object 90A corresponds to the first object, the attention image 301A corresponds to an example of the first image. Similarly, when the object 90B corresponds to the second object, the attention image 301B corresponds to an example of the second image.

[0121] In addition, the display control unit 172 may also display the attention image 301 regarding an object that is not seen in the range VA. Figure 6 The shown attention image 301C is an image that prompts the user U to pay attention to an object that exists to the left of the range VA and is outside the range that the user U can see through the right light guide plate 26 and the left light guide plate 28. In order to indicate that there is an object to the left of the range VA, the attention image 301C includes an arrow pointing to the left.

[0122] The display control unit 172 displays the attention image 301 for one object out of the multiple objects detected by the distance detection unit 173, which is the object with the highest attention level determined by the determination unit 177. In Figure 6 Three attention images 301A, 301B, and 301C are shown in, but in the present embodiment, only one attention image 301 is simultaneously displayed by the display unit 20. In this case, there are the following advantages: The attention image 301 does not obstruct the user U's field of view, does not distract the user U's attention, and can effectively prompt attention.

[0123] The attention images 301A, 301B, and 301C can also be displayed by the display control unit 172 according to the image data pre-stored in the non-volatile storage unit 121. In addition, the display control unit 172 can also perform the process of generating the image data for displaying the attention images 301A, 301B, and 301C.

[0124] [3. Operation of the Display System]

[0125] Figure 7 , Figure 8 and Figure 9 are flowcharts showing the operation of the HMD 100. The operation of the control unit 120 related to the display of the attention image 301 is shown in each of these drawings.

[0126] In step S11, the distance detection unit 173 detects the objects captured in the captured image by acquiring and analyzing the captured image of the outer camera 61. Next, in step S12, the distance detection unit 173 detects the distance between the display unit 20 and the objects detected in step S11.

[0127] Next, in step S13, the determination unit 177 calculates the risk level for the objects detected by the distance detection unit 173. When the distance detection unit 173 detects multiple objects, the determination unit 177 calculates the risk level for each of the multiple objects separately.

[0128] In step S14, the determination unit 177 selects the object to be processed. In step S15, the gaze detection unit 176 detects the gaze direction VD of the user U. In step S16, the determination unit 177 determines the relationship between the gaze direction VD and the position of the object. For example, in step S16, the determination unit 177 determines whether the gaze direction VD is the direction corresponding to the position of the object selected as the object to be processed in step S14.

[0129] In step S17, the determination unit 177 determines whether the user U is looking at the object to be processed based on the result determined in step S16. If it is determined that the user U is not looking at the object to be processed (step S17; NO), the determination unit 177 proceeds to step S22 described later. If it is determined that the user U is looking at the object to be processed, the determination unit 177 proceeds to step S18.

[0130] In step S18, the determination unit 177 sets the object to be processed as the object being looked at. Next, in step S19, the determination unit 177 obtains the duration for which the user U has been looking at the object to be processed. This time is referred to as the fixation time. The fixation time is obtained, for example, by analyzing the history of the line-of-sight direction VD detected by the line-of-sight detection unit 176.

[0131] In step S20, the determination unit 177 determines whether the fixation time is longer than the time threshold. If it is determined that the fixation time is equal to or less than the time threshold (step S20; NO), the determination unit 177 proceeds to step S22. If it is determined that the fixation time is longer than the time threshold (step S20; YES), the determination unit 177 proceeds to step S21.

[0132] In step S21, the determination unit 177 corrects the risk level of the object to be processed to a lower value and determines the attention level based on the corrected risk level. For example, the determination unit 177 corrects the risk level of the object to be processed to a low specified level. Additionally, for example, the determination unit 177 may also correct the risk level of the object to be processed to be lower than the risk level of other objects. Further, for example, in step S21, the determination unit 177 performs processing so that the attention level of the object to be processed is lower than the attention level of the object when the line-of-sight direction is a direction not corresponding to the object to be processed. Then, the determination unit 177 proceeds to step S23.

[0133] In step S22, the determination unit 177 determines the attention level based on the risk level of the object to be processed and proceeds to step S23.

[0134] In step S23, the display control unit 172 determines whether the attention level of the object to be processed is higher than the display threshold. If it is determined that the attention level of the object to be processed is equal to or less than the display threshold (step S23; NO), the control unit 120 ends this processing. If it is determined that the attention level of the object to be processed is higher than the display threshold (step S23; YES), the display control unit 172 proceeds to step S24.

[0135] In step S24, the display control unit 172 determines whether an attention image 301 related to another object is being displayed. That is, it determines whether an attention image 301 of an object that is not the object to be processed is being displayed. When it is determined that the attention image 301 related to another object is not being displayed (step S24; NO), the display control unit 172 proceeds to step S26. When it is determined that the attention image 301 related to another object is being displayed (step S24; YES), the display control unit 172 proceeds to step S25.

[0136] In step S25, the display control unit 172 stops the display of the attention image 301 being displayed, that is, the attention image 301 of an object that is not the object to be processed, and proceeds to step S26. In step S26, the display control unit 172 causes the display unit 20 to display an attention image 301 prompting attention to the object to be processed, and ends this process.

[0137] The control unit 120, after Figure 7 completing the process shown, may also return to step S14, newly select another object as the object to be processed, and continue the process.

[0138] In addition, during the operation of the HMD 100, the control unit 120 repeatedly executes Figure 7 the process shown at a prescribed cycle. As a result, the attention level for the objects near the user U is periodically updated. Whenever the attention level is updated, the attention image 301 is displayed as needed, or the display of the attention image 301 is stopped.

[0139] Figure 8 Shows the actions performed by the control unit 120 during the period when the attention image 301 is being displayed.

[0140] In step S41, the display control unit 172 determines the object on which the attention image 301 is being displayed. In step S42, the distance detection unit 173 detects the distance between the object determined in step S41 and the display unit 20. In step S43, the display control unit 172 determines whether the distance detected in step S42 is greater than the distance threshold. When the detected distance is below the distance threshold (step S43; NO), the display control unit 172 ends this process. When the detected distance is greater than the distance threshold (step S43; YES), the display control unit 172 proceeds to step S44. In step S44, the display control unit 172 stops the display of the attention image 301 being displayed, and ends this process.

[0141] During the display of the attention image 301, the control unit 120 repeatedly executes Figure 8The processing shown. Thus, when the object moves away from the user U, the unnecessary attention image 301 can be quickly set to non-display.

[0142] Figure 9 Shows the actions performed by the control unit 120 on the object set as the object being gazed at in step S18 at a predetermined cycle.

[0143] In step S61, the determination unit 177 determines the object set as the object being gazed at. The object being gazed at is the object located at the position overlapping with the line-of-sight direction VD of the user U, and is basically one. In the case where there are multiple objects being gazed at, in step S61, the determination unit 177 selects any one object.

[0144] In step S62, the line-of-sight detection unit 176 detects the line-of-sight direction VD. In step S63, the determination unit 177 determines whether the line-of-sight direction VD has deviated from the object determined in step S61. In step S62, specifically, the determination unit 177 determines whether the line-of-sight direction VD has deviated from the object based on whether the line-of-sight direction VD detected in step S62 is in the direction overlapping with the position of the object determined in step S61. In other words, the determination unit 177 determines whether the line-of-sight direction VD is the direction corresponding to the object.

[0145] In the case where it is determined that the line-of-sight direction VD has not deviated from the object (step S63; No), the determination unit 177 ends this processing. In the case where it is determined that the line-of-sight direction VD has deviated from the object (step S63; Yes), the determination unit 177 transfers to step S64.

[0146] In step S64, the determination unit 177 obtains the time when the line-of-sight direction VD has deviated from the object. This time is called the line-of-sight departure time. The line-of-sight departure time is obtained, for example, by analyzing the history of the line-of-sight direction VD detected by the line-of-sight detection unit 176.

[0147] Next, in step S65, the determination unit 177 obtains the damage information of the object determined in step S61. Next, in step S66, the determination unit 177 obtains the departure time threshold corresponding to the value of the damage information. The damage information in this embodiment is numericalized as described above. The threshold storage unit 153 stores a plurality of departure time thresholds corresponding to the values of a plurality of damage information respectively.

[0148] In step S67, the determination unit 177 determines whether the line-of-sight departure time is longer than the departure time threshold. In the case where it is determined that the line-of-sight departure time is below the departure time threshold (step S67; No), the determination unit 177 ends this processing.

[0149] When it is determined that the line-of-sight departure time is longer than the departure time threshold (step S67; YES), the determination unit 177 proceeds to step S68. In step S68, the determination unit 177 cancels the setting of the object being gazed at for the object identified in step S61.

[0150] Next, in step S69, the determination unit 177 corrects the attention level of the object identified in step S61 to be higher than the attention levels of other objects. In step S70, the display control unit 172 determines whether the attention level corrected in step S69 is higher than the display threshold. When the display control unit 172 determines that the attention level is below the display threshold (step S70; NO), this process ends. When the display control unit 172 determines that the attention level is higher than the display threshold (step S70; YES), it proceeds to step S71.

[0151] In step S71, the display control unit 172 stops the display of the attention image 301 related to other objects. That is, the attention image 301 displayed for objects other than the object being processed is made non-displayed. In step S72, the display control unit 172 causes the display unit 20 to display the attention image 301 that prompts attention related to the object being processed, and this process ends.

[0152] Figure 10 FIG. is an explanatory diagram showing an example of the transition of the display of the display unit 20, schematically showing the state in which the display of the attention image 301 changes due to Figure 7 , Figure 8 and Figure 9 the actions shown.

[0153] Figure 10 FIG. shows an example in which two objects are detected by the distance detection unit 173. These two objects are set as object 1 and object 2. In addition, the distance between the display unit 20 and the object is represented by a value that is numerically quantified step by step. As Figure 10 shown, in the following example, the display threshold is set to attention level 3, the distance threshold is set to distance 10, and the time threshold is set to a gaze time of 5 seconds.

[0154] State ST1 is a state in which the user U is not gazing at either object 1 or object 2. The attention level of object 1 is a value higher than the attention level of object 2. And the attention level of object 1 is higher than the display threshold. Therefore, the attention image 301 is displayed for object 1.

[0155] When, in state ST1, user U gazes at object 1, control unit 120 transitions to state ST2. In state ST2, object 1 is set as the gazed-at object, and thus, the attention level of object 1 is corrected to be lower than that of object 2. As a result, the attention level of object 2 becomes higher than that of object 1. The attention level of object 2 is higher than the display threshold. Therefore, in state ST2, attention image 301 is displayed for object 2.

[0156] When, in state ST2, the line-of-sight direction VD of user U deviates from object 1, control unit 120 transitions to state ST1. In this case, by canceling the setting of the gazed-at object for object 1, the correction for reducing the attention level of object 1 is not performed, and thus, the attention level of object 1 is a value higher than that of object 2. Therefore, attention image 301 is displayed for object 1.

[0157] When, in state ST1, user U moves, the distance between object 1 and user U, and the distance between user U and object 2 change. The changed state is represented as state ST3. In state ST3, both the distance between object 1 and user U and the distance between user U and object 2 are longer than in state ST1. Therefore, both the attention level of object 1 and the attention level of object 2 are lower than in state ST1. After or during the movement of user U, control unit 120 re-executes Figure 7 the operation, thereby updating the attention levels of object 1 and object 2 to become state ST3.

[0158] In state ST3, the attention level of object 1 is a value higher than that of object 2. Also, the attention level of object 1 is higher than the display threshold. Therefore, attention image 301 is displayed for object 1.

[0159] When, in state ST3, user U moves further, the distance between object 1 and user U, and the distance between user U and object 2 change. The changed state is represented as state ST4. In state ST4, both the distance between object 1 and user U and the distance between user U and object 2 are longer than in state ST3. Therefore, both the attention level of object 1 and the attention level of object 2 are lower than in state ST3. After or during the movement of user U, control unit 120 re-executes Figure 7 the operation, thereby updating the attention levels of object 1 and object 2 to become state ST4.

[0160] In state ST4, the attention level of object 1 is a value higher than that of object 2. However, the attention level of object 1 is lower than the display threshold. Therefore, display unit 20 makes attention image 301 non-displayed.

[0161] [Function of the Embodiment]

[0162] As described above, the HMD 100 to which the embodiment of the present invention is applied includes: a display unit 20 that allows an external scene including a first object and a second object to pass through; and a gaze detection unit 176 that detects the gaze direction VD of the user U. The HMD 100 includes a time calculation unit 175 that calculates a first time taken for the first object to reach the user U and a second time taken for the second object to reach the user U. The HMD 100 includes a determination unit 177. The determination unit 177 acquires first information related to the first object and second information related to the second object, and calculates the risk level of the first object based on the first information and the first time. The determination unit 177 corrects the risk level of the first object according to the gaze direction VD, and determines the attention level of the first object based on the corrected risk level of the first object. The determination unit 177 calculates the risk level of the second object based on the second information and the second time, corrects the risk level of the second object according to the gaze direction VD, and determines the attention level of the second object based on the corrected risk level of the second object. The HMD 100 includes a display control unit 172 that causes the display unit 20 to display a first image related to the first object when the attention level of the first object is higher than the attention level of the second object.

[0163] The control method of the HMD 100 is a control method of the HMD 100 including a display unit 20 that allows an external scene including a first object and a second object to pass through and a gaze detection unit 176 that detects the gaze direction VD of the user U. Through the control unit 120, this control method acquires first information related to the first object and second information related to the second object. Further, through the control unit 120, it calculates a first time taken for the first object to reach the user U and a second time taken for the second object to reach the user U. Further, through the control unit 120, it calculates the risk level of the first object based on the first information and the first time. Further, through the control unit 120, it corrects the risk level of the first object according to the gaze direction VD, and determines the attention level of the first object based on the corrected risk level of the first object. Further, through the control unit 120, it calculates the risk level of the second object based on the second information and the second time, corrects the risk level of the second object according to the gaze direction VD, and determines the attention level of the second object based on the corrected risk level of the second object. Further, when the attention level of the first object is higher than the attention level of the second object, it causes the display unit 20 to display a first image related to the first object.

[0164] According to the HMD 100 and the control method of the HMD 100, regarding the objects existing around the user U, it is possible to accurately determine the attention level, which is an index indicating whether the user U should pay attention, considering the line-of-sight direction VD. Then, by displaying the attention image 301 according to the attention level, the HMD 100 can appropriately prompt the user U to pay attention to the possibility of danger to the user caused by objects in the actual space, etc.

[0165] When the attention level of the second object is higher than the attention level of the first object, the display control unit 172 causes the display unit 20 to display the second image related to the second object. Thus, when there are multiple objects around the user U, regarding the object with a higher attention level, the attention image 301 can be displayed. Therefore, it is possible to more appropriately prompt the user U to pay attention to the possibility of danger to the user caused by objects in the actual space, etc.

[0166] When the attention level of the second object is higher than the attention level of the first object, the display control unit 172 causes the display unit 20 to stop displaying the first image. Thus, when there are multiple objects around the user U, regarding the object with a lower attention level, the attention image 301 is not displayed, so it is possible to more appropriately prompt the user U to pay attention.

[0167] The HMD 100 has a distance detection unit 173 that detects the distance from the position of the user U to the first object. When the distance from the position of the user U to the first object is greater than the first threshold, the display control unit 172 causes the display unit 20 to stop displaying the first image. In this case, regarding the object that is far from the position of the user U and thus requires less attention from the user U, the attention image 301 can be set to non-display. Therefore, it is possible to avoid unnecessary display of the attention image 301 and improve the convenience of the user U regarding the use of the HMD 100.

[0168] When the attention level of the first object when the line-of-sight direction VD is in the direction corresponding to the position of the first object is lower than the attention level of the first object when the line-of-sight direction VD is not in the direction corresponding to the position of the first object, the determination unit 177 makes this determination. Thus, the frequency of displaying the attention image 301 for the object that the user U looks at is reduced. Therefore, it is possible to avoid unnecessary display of the attention image 301 and improve the convenience of the user U regarding the use of the HMD 100. And when the attention level of the object that the user U looks at is still higher than the display threshold after correction, the attention image 301 is displayed. Therefore, it is possible to appropriately display the attention image 301 corresponding to both the necessity for the user U to pay attention to the object and the line-of-sight direction VD of the user U.

[0169] The determination unit 177 obtains the fixation time in a state where the line-of-sight direction VD is a direction corresponding to the position of the first object, and when the fixation time exceeds the time threshold, makes the attention level of the first object lower than the attention level of the second object. Thereby, regarding the object that the user U has fixated on for a time exceeding the time threshold, the attention image 301 is set to non-display, so that unnecessary display of the attention image 301 can be avoided, and regarding the use of the HMD 100, improvement in the convenience of the user U can be achieved. In addition, it is possible to prevent the attention image 301 from interfering with the user U's fixation on the object.

[0170] When the line-of-sight direction VD is not a direction corresponding to the position of the first object, the determination unit 177 makes the attention level of the first object higher than the attention level of the second object. Thereby, it is possible to appropriately prompt the user U to pay attention to the possibility of danger to the user brought by objects in the actual space, etc., corresponding to the line-of-sight direction VD of the user U.

[0171] When the line-of-sight direction VD changes from a direction corresponding to the position of the first object to a direction not corresponding to the position of the first object, the display control unit 172 causes the display unit 20 to redisplay the first image related to the first object according to the damage information corresponding to the first object. Thereby, after the attention image 301 is set to non-display corresponding to the direction in which the line-of-sight direction VD of the user U faces the object, the attention image 301 is redisplayed as needed, which can prompt the user U to pay attention. In addition, by redisplaying according to the magnitude of the damage to the user U caused by the first object, unnecessary display of the attention image 301 can be avoided.

[0172] The first information is information related to the damage to the user U caused by the first object, and the second information is information related to the damage to the user U caused by the second object. For example, the first information is the value of the damage information of the first object, and the second information is the value of the damage information of the second object. Thereby, the magnitude of the damage that the user U may receive from the object can be reflected in the determination of the attention level. Therefore, by more accurately evaluating the possibility of danger to the user brought by objects in the actual space, etc., the user U can be appropriately prompted to pay attention.

[0173] When the first time is shorter than the second time and the damage to the user U caused by the first object is greater than the damage to the user U caused by the second object, the determination unit 177 may also correct the attention level of the first object to be higher than the attention level of the second object. In this case, it is possible to display the attention image 301 for an object that may reach the position of the user U first and cause greater damage.

[0174] The HMD 100 is provided with a temperature detection unit 174 which detects the temperature of a first object and the temperature of a second object. The determination unit 177 may also calculate the risk level of the first object based on the first information, the first time, and the temperature of the first object, and calculate the risk level of the second object based on the second information, the second time, and the temperature of the second object. In this case, the determination unit 177 can reflect the damage that the user U may suffer due to the temperature of the object in the determination of the attention level. Therefore, by more accurately evaluating the possibility that an object in the actual space poses a danger to the user, the user U can be appropriately prompted to pay attention.

[0175] The HMD 100 is provided with a motion detection unit 178 which detects the motion of the user U. When a motion of the user U approaching the first object is detected, the time calculation unit 175 calculates the first time based on the motion of the user U. When a motion of the user U approaching the second object is detected, the time calculation unit 175 calculates the second time based on the motion of the user U. Thus, in addition to the relative speed between the HMD 100 and the object, the motion of the user U is also reflected, and the first time and the second time can be calculated more accurately. Therefore, the user U can be appropriately prompted to pay attention to the possibility that an object in the actual space poses a danger to the user.

[0176] The HMD 100 is provided with an external camera 61 which captures the first object and the second object included in the external scene. The distance detection unit 173 detects the object based on the captured image of the external camera 61.

[0177] [5. Other Embodiments]

[0178] The present invention is not limited to the structure of the above-described embodiments and can be implemented in various ways without departing from its gist.

[0179] For example, the control unit 120 may also reflect the position of the display unit 20 in the body of the user U and perform detection of the distance between the user U and the object and calculation of the time until the object reaches the user U. In addition, for example, the control unit 120 may detect the line-of-sight direction VD using the detection results of the 6-axis sensor 235 and / or the magnetic sensor 237.

[0180] The device for processing the display image of the display unit 20 and / or the voice output from the right earphone 32 and the left earphone 34 is not limited to the controller 10. The HMD 100 may also use an external computer instead of the controller 10. That is, Figure 4Each functional unit constituted by the processor 125 shown is configured as a structure of a computer connected to the display unit 20, and various numerical values and information stored in the non-volatile storage unit 121 are configured as a structure stored in the computer connected to the display unit 20. In this case, the following processing is executed by the above computer. Figure 7 , Figure 8 and Figure 9 shown processing passes. The HMD 100 transmits the detection data of various sensors included in the display unit 20 and various sensors included in the controller 10 to the computer, and can perform display based on the display data input from the computer. Such a computer can be, for example, a smartphone, a PDA terminal, or a tablet personal computer.

[0181] In addition, in the above-described embodiment, a structure in which the controller 10 and the display unit 20 are connected in a wired manner is illustrated, but it is not limited thereto, and a structure in which the display unit 20 and the controller 10 are connected in a wireless manner may also be used. In addition, the controller 10 can also be implemented using multiple devices. In addition, a wearable device that can be attached to the user's body, clothing, or accessories worn by the user can be used instead of the controller 10. The wearable device in this case can be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, a virtual flight mouse, a game controller, a pen-type device, etc.

[0182] In addition, in the above-described embodiment, a structure in which the display unit 20 and the controller 10 are separated and connected via the connection cable 40 is illustrated as an example. However, it is not limited thereto, and a structure in which the controller 10 and the display unit 20 are integrally formed and worn on the user's head may also be used.

[0183] In addition, the display unit 20 is not limited to being directly worn on the head of the user U. For example, an image display unit in another form such as an image display unit worn like a hat may be used instead of the display unit 20.

[0184] The optical system that guides the image light to the user's eyes may have a structure in which the image light is incident on the user's eyes through the right light guide plate 26 and the left light guide plate 28, and various structures can be adopted. For example, a structure in which a half mirror is provided on a part of the right light guide plate 26 and the left light guide plate 28 and the image light generated by the right light guide plate 26 and the left light guide plate 28 is reflected to the user U's right eye RE and left eye LE through the half mirror can be cited. In addition, a structure in which an image is displayed on a display area that occupies the entire surface or most of the area of the right light guide plate 26 and the left light guide plate 28 may also be used. In this case, in the operation of changing the display position of the image, a process of reducing the image may also be included. In addition, a diffraction grating, a prism, or a holographic display unit can also be used as the right light guide plate 26 and the left light guide plate 28.

[0185] In addition, in each of the above-described embodiments, a structure in which the display unit 20 generates image light through the OLED units 221 and 241 has been described, but the present invention is not limited thereto. For example, the right display unit 22 and the left display unit 24 may use a transmissive liquid crystal panel, may use a reflective liquid crystal panel instead of the transmissive liquid crystal panel, or may use a digital micromirror device. In addition, a structure applying LCOS technology may be used instead of the LCD panel. LCOS is an abbreviation for Liquid crystal on silicon.

[0186] In addition, the display unit 20 may also have a structure using self-luminous display elements represented by an LED array, a laser array, a quantum dot light-emitting element, etc. In addition, for example, the display unit 20 may also be a laser scanning type display that combines a laser light source and a laser scanner.

[0187] In addition, Figure 3 , Figure 4 At least a part of the functional blocks shown, such as, can be implemented by hardware or can be a structure implemented by the cooperation of hardware and software, and is not limited to a structure in which independent hardware resources are configured as shown in the figure.

[0188] In addition, Figure 7 , Figure 8 and Figure 9 The processing units of the flowcharts shown are obtained by dividing according to the main processing contents for easy understanding of the processing of the first control unit 120. The embodiments are not limited by the division method and name of the processing units of each flowchart. In addition, the processing order of the above flowcharts is not limited to the illustrated examples.

[0189] In addition, the program executed by the processor 125 may also be stored in an external device or equipment and obtained via the communication unit 117 or the like. In addition, it can also be pre-recorded in a recording medium recorded in a computer-readable manner. As the recording medium, a magnetic, optical recording medium, or a semiconductor memory device can be used. Specifically, removable or fixed recording media such as a floppy disk, various optical discs, magneto-optical discs, flash memories, and cartridge recording media can be cited. In addition, the recording medium may also be a non-volatile storage device such as a RAM, a ROM, or an HDD which is an internal storage device provided in the image display device.

Claims

1. A head-mounted display device, comprising: A display unit that allows an external scene including a first object and a second object to pass through; A line-of-sight detection unit that detects the line-of-sight direction of a user; A time calculation unit that calculates a first time taken for the first object to reach the user and a second time taken for the second object to reach the user; and A determination unit that obtains information related to damage to the user caused by the first object, i.e., first information, and information related to damage to the user caused by the second object, i.e., second information, calculates the risk level of the first object based on the first information and the first time, corrects the risk level of the first object based on the line-of-sight direction, thereby determining the attention level of the first object, calculates the risk level of the second object based on the second information and the second time, corrects the risk level of the second object based on the line-of-sight direction, thereby determining the attention level of the second object. When the attention level of the first object is higher than the attention level of the second object, the display unit displays a first image related to the first object.

2. The head-mounted display device according to claim 1, wherein when the attention level of the second object is higher than the attention level of the first object, the display unit displays a second image related to the second object.

3. The head-mounted display device according to claim 2, wherein when the attention level of the second object is higher than the attention level of the first object, the display unit stops the display of the first image.

4. The head-mounted display device according to claim 1 or 2, wherein the head-mounted display device has a distance detection unit that detects the distance from the position of the user to the first object. When the distance from the position of the user to the first object is greater than a first threshold, the display unit stops the display of the first image.

5. The head-mounted display device according to claim 1 or 2, wherein the determination unit makes the attention level of the first object when the line-of-sight direction is a direction corresponding to the position of the first object lower than the attention level of the first object when the line-of-sight direction is not a direction corresponding to the position of the first object.

6. The head-mounted display device according to claim 5, wherein the determination unit obtains the fixation time when the line-of-sight direction is a direction corresponding to the position of the first object. When the fixation time exceeds a time threshold, the determination unit makes the attention level of the first object lower than the attention level of the second object.

7. The head-mounted display device according to claim 5, wherein when the line-of-sight direction is not a direction corresponding to the position of the first object, the determination unit makes the attention level of the first object higher than the attention level of the second object.

8. The head-mounted display device according to claim 6, wherein, when the line-of-sight direction changes from the direction corresponding to the position of the first object to a direction not corresponding to the position of the first object, the display unit redisplay the first image according to the first information.

9. The head-mounted display device according to claim 1, wherein, when the first time is shorter than the second time and the damage to the user caused by the first object is greater than the damage to the user caused by the second object, the determination unit corrects to make the attention level of the first object higher than the attention level of the second object.

10. The head-mounted display device according to claim 1 or 2, wherein, the head-mounted display device has a temperature detection unit that detects the temperature of the first object and the temperature of the second object, the determination unit calculates the risk level of the first object according to the first information, the first time, and the temperature of the first object, the determination unit calculates the risk level of the second object according to the second information, the second time, and the temperature of the second object.

11. The head-mounted display device according to claim 1 or 2, wherein, the head-mounted display device has a motion detection unit that detects the motion of the user, when the motion detection unit detects the user's motion approaching the first object, the time calculation unit calculates the first time according to the user's motion, when the motion detection unit detects the user's motion approaching the second object, the time calculation unit calculates the second time according to the user's motion.

12. A control method for a head-mounted display device, comprising the following steps: Obtaining information related to the damage to the user caused by the first object, i.e., the first information, and information related to the damage to the user caused by the second object, i.e., the second information; Detecting the line-of-sight direction of the user; Calculating the first time taken for the first object to reach the user and the second time taken for the second object to reach the user; Calculating the risk level of the first object according to the first information and the first time, and correcting the risk level of the first object according to the line-of-sight direction, thereby determining the attention level of the first object; Calculating the risk level of the second object according to the second information and the second time, and correcting the risk level of the second object according to the line-of-sight direction, thereby determining the attention level of the second object; and When the attention level of the first object is higher than the attention level of the second object, causing the display unit to display a first image related to the first object.

Images