Head-mounted gaze detection device and gaze image display system

Abstract

To provide a head-mounted gaze detection device and a gaze image display system that can capture an image from a wearer's gaze and obtain a wearer's gaze position in the image.SOLUTION: A head-mounted gaze detection device includes first and second prisms 33, 43 that pass wearer's left and right gazes S1, S2 forward respectively, a near-infrared camera 34 arranged to be adjacent to the first prism 33 on one side of the left and right sides, and an eye camera 44 arranged to be adjacent to the second prism 43 on the other side of the left and right sides. The first prism 33 on one side of the left and right sides refracts an optical axis P1 of the near-infrared camera 34 forward such that the optical axis P1 can coincide with the left or right gaze S1, and the second prism 43 on the other side of the left and right sides refracts an optical axis P2 of the eye camera 44 backward such that one of left and right eyes can be imaged. Information on the gaze position in the image captured by the near-infrared camera 34 is obtained based on image data captured by the eye camera 44.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a head-mounted gaze detection device that detects the gaze of a wearer, and a gaze image display system that includes the head-mounted gaze detection device. [Background technology]

[0002] For example, Patent Document 1 proposes a configuration for capturing an image of the wearer's eye movement and detecting the wearer's gaze point. This conventional configuration includes a field camera that captures an image in front of the wearer and an eye camera that captures the eye movement via a dichroic mirror, and the gaze position obtained by arithmetic processing of the data captured by the eye camera is superimposed on the image captured by the field camera and displayed on a monitor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-171586 A Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, in the conventional configuration of Patent Document 1 described above, the eyeball camera can accurately capture the movement of the eyeballs through a dichroic mirror placed in front of the eyes, but since the visual field camera is placed above both eyes, the optical axis of the visual field camera differs from the line of sight of the wearer, and the image captured by the visual field camera does not match the image seen by the wearer. Therefore, when superimposing the gaze position obtained from the data captured by the eyeball camera on the image captured by the visual field camera, it is difficult to accurately display the gaze position.

[0005] The present invention proposes a head-mounted gaze detection device and a gaze image display system that can accurately and stably acquire the gaze position of a wearer in an image captured in front of the wearer. [Means for solving the problem]

[0006] The first invention of the present invention is a head-mounted gaze detection device comprising a head mounting section to be mounted on the head of a wearer; left and right gaze passing members attached to the head mounting section and allowing the left and right gazes of the wearer to pass forward, respectively; a visual field imaging means arranged adjacent to one of the left and right gaze passing members and imaging the area in front of the wearer through the gaze passing member; an eyeball imaging means arranged adjacent to the other left and right gaze passing member and imaging the wearer's eyeball through the gaze passing member; and a gaze information acquisition means for acquiring information on the gaze position in the visual field image based on information on the visual field image captured by the visual field imaging means and information on the eyeball image captured by the eyeball imaging means, wherein one of the left and right gaze passing members is provided with a prism that allows the wearer's gaze to pass forward and refracts the optical axis of the visual field imaging means forward, and the other left and right gaze passing member is provided with a prism that allows the wearer's gaze to pass forward and refracts the optical axis of the eyeball imaging means backward, allowing the other left and right eyeball to be imaged by the eyeball imaging means.

[0007] In this configuration, the optical axis of the visual field imaging means can be aligned with one of the left and right lines of sight, and this alignment allows a visual field image captured at the wearer's line of sight to be obtained, and the eyeball imaging means can accurately obtain the direction of the line of sight from the information on the eyeball image. Then, the information on the visual field image and the information on the eyeball image (direction of the line of sight) can be accurately and stably obtained. According to the information on the visual field image thus obtained, if the visual field position is displayed on an image display device such as a monitor by superimposing it on the visual field image, the object being viewed by the wearer can be shown to the person viewing the image display device at the same line of sight and the wearer's gaze point can be accurately shown. As a result, for example, if a surgeon performing surgery uses the configuration of the present invention in a medical field, assistants and visitors can view information on the surgery from the same line of sight as the surgeon and can confirm the surgeon's gaze point during surgery, making it easier for visitors to accurately understand information on the surgery. This is not limited to medical fields, but is useful for education and technology transfer in educational fields, research and development fields, etc.

[0008] The second invention of the present invention is a gaze image display system comprising a head-mounted gaze detection device of the first invention, an image display device for displaying an image, and an image control means for controlling the display of an image on the image display device, wherein the image control means comprises a field of view image display means for displaying a field of view image on the image display device based on information of a field of view image captured by a field of view imaging means of the head-mounted gaze detection device, and a gaze image display means for displaying a gaze image indicating the gaze position on the field of view image displayed by the field of view image display means based on information of the gaze position acquired by the gaze information acquisition means of the head-mounted gaze detection device.

[0009] In this configuration, the visual field image displayed on the image display device can show the same thing as the wearer of the head-mounted gaze detection device at the same eye level as the wearer, and the gaze image displayed superimposed on the visual field image can accurately show the gaze point of the wearer. This makes it possible to accurately convey the position of the wearer's gaze to the person viewing the image display device. Therefore, by applying the device to, for example, medical settings, educational settings, and research and development settings, education and technology transfer can be efficiently carried out. Effect of the Invention

[0010] In the head-mounted gaze detection device of the present invention, as described above, the visual field imaging means can capture an image at the wearer's line of sight, so that the gaze position in the visual field image captured by the visual field imaging means can be accurately and stably shown. Therefore, if the visual field image and the gaze image showing the gaze position are displayed on an image display device or the like, the gaze point of the wearer can be accurately conveyed to a person viewing the image display device, which is useful for education and technical transfer in medical settings and the like.

[0011] As described above, in the gaze image display system of the present invention, a visual field image and a viewpoint image captured from the wearer's line of sight can be displayed on an image display device, so that the wearer's point of gaze can be accurately conveyed, enabling efficient education and technical transfer in medical settings, etc. [Brief description of the drawings]

[0012] [Figure 1] 1 is a perspective view showing a head-mounted gaze detection device 1 according to an embodiment of the present invention. [Diagram 2] 1A is a plan view and FIG. 1B is a front view of a head-mounted gaze detection device 1. [Diagram 3] 1 is a side view showing a state in which the head-mounted gaze detection device 1 is worn. [Figure 4] 1A is a plan view of the right-side line-of-sight imaging main body 22 with the right-side housing 31 cut away, and FIG. 1B is a front view of the right-side housing 31 cut away. [Diagram 5]1A is a plan view showing a cutaway left housing 41, FIG. 1B is a cutaway front view showing the left housing 41, and FIG. 1C is a front view of the left line of sight imaging main body 23. [Figure 6] (A) is an oblique view showing the first prism 33 of the right-side line-of-sight imaging body 22, (B) is an exploded oblique view of the first prism 33, (C) is an oblique view showing the second prisms 43, 43 of the left-side line-of-sight imaging body 23, and (D) is an exploded oblique view of the second prisms 43, 43. [Figure 7] 2 is an explanatory diagram showing the lines of sight S1, S2 of a wearer and an optical axis P1 of a near-infrared camera 34 in the head-mounted gaze detection device 1. FIG. [Figure 8] FIG. 2 is an explanatory diagram showing a line-of-sight image display system 61. [Figure 9] 1 is an explanatory diagram showing a display example on an image display device 64 of a line-of-sight image display system 61. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An embodiment of the present invention will be described with reference to the attached drawings. In the embodiment, the front-rear direction refers to the front-rear direction of a wearer wearing the head-mounted gaze detection device 1 of the embodiment, and left-right similarly refers to the left-right direction of the wearer. The left-right direction perpendicular to the front-rear direction and the up-down direction (vertical direction) is called the left-right lateral direction. In the left-right direction, the direction from the outside of the left and right toward the center of the wearer is called the inward direction. Furthermore, in the embodiment, the front side and the front side are used interchangeably.

[0014] 1 to 3, a head-mounted gaze detection device 1 of the embodiment includes protective glasses 2 that are integrally provided with a transparent shield part 4 that is disposed in front of and covers both eyes of a wearer 101, and a frame part 3 that is attached to the wearer's head by hanging it over both ears of the wearer. The head-mounted gaze detection device 1 includes a gaze imaging unit 6 and a lighting device 7 that are attached to the center of the frame part 3 via a connecting member 5, and the gaze imaging unit 6 and the lighting device 7 are integrally provided.

[0015] The connecting member 5 includes a base 11 fixed to the frame 3 of the protective glasses 2, a support rod 12 provided on the base 11 so as to be tiltable in the vertical direction, and a sliding part 13 provided with a vertical long hole through which the support rod 12 is inserted. A positioning part 15 is rotatably screwed to the tip of the support rod 12, and by fastening the positioning part 15, the vertical position of the sliding part 13 relative to the support rod 12 can be fixed, while by releasing the fastening of the positioning part 15, the sliding part 13 can be moved in the vertical direction relative to the support rod 12. The lighting device 7 is attached to the sliding part 13 so as to be tiltable in the vertical direction, and the line of sight imaging unit 6 is attached integrally to the lighting device 7. Such a connecting member 5 can tilt the support rod part 12, the sliding part 13, the lighting device 7, and the gaze imaging unit 6 in the front-rear direction together by tilting the support rod part 12 relative to the base part 11. This tilting of the gaze imaging unit 6 can change the distance to both eyes of the wearer 101. In addition, by sliding the sliding part 13 in the vertical direction relative to the support rod part 12, the sliding part 13, the lighting device 7, and the gaze imaging unit 6 can be slid in the vertical direction together. Furthermore, by tilting the lighting device 7 in the vertical direction relative to the sliding part 13, the lighting device 7 and the gaze imaging unit 6 can be tilted in the vertical direction together, and they can be inclined at a desired angle relative to the wearer.

[0016] The illumination device 7 includes an illumination support part 17 tiltably attached to the sliding part 13, a visible light 18 that irradiates visible light, and a near-infrared light 19 that irradiates near-infrared light (light in the near-infrared wavelength range), and the visible light 18 and the near-infrared light 19 are supported by the illumination support part 17. The visible light 18 and the near-infrared light 19 are arranged to irradiate light to an object (the viewpoint T of the wearer) seen through the first and second prisms 33 and 43 of the line-of-sight imaging unit 6. The visible light 18 irradiates visible light to improve the visibility of the object, while the near-infrared light 19 irradiates near-infrared light to enable a near-infrared camera 34 (described later) to capture a near-infrared image. Note that an ON / OFF switch and a power source (not shown) are connected to these lights 18 and 19 via cables.

[0017] Next, the line of sight imaging unit 6 according to the main part of the present invention will be described. As shown in FIGS. 1 to 3, the visual axis imaging unit 6 includes a guide support member 21 attached to the illumination support portion 17 of the illumination device 7, and left and right visual axis imaging bodies 22 and 23 supported by the guide support member 21, respectively.

[0018] The guide support member 21 is formed in an elongated shape in the left-right direction, and the left-right central portion is fixed to the illumination support portion 17. The guide support member 21 is formed symmetrically with respect to the central portion, and a first guide elongated hole portion 25 and a second guide elongated hole portion 26 are provided on both the left and right sides. The first guide elongated hole portion 25 is a long hole that opens linearly in the left-right lateral direction and penetrates the guide support member 21 in the up-down direction. The left and right first guide elongated hole portions 25 are provided in front of the left and right eyes of the wearer when the guide support member 21 is worn by the wearer. The second guide elongated hole portion 26 is a long hole that is inclined linearly forward with respect to the first guide elongated hole portion 25 on the outer side of the first guide elongated hole portion 25 and penetrates the guide support member 21 in the up-down direction.

[0019] The right line-of-sight imaging body 22 includes a right housing 31 supported on the guide support member 21 so as to be slidable in the left-right direction, a magnifying glass section 32 attached to the front of the right housing 31, and a first prism 33 (see FIG. 4) disposed inside the right housing 31. Furthermore, a near-infrared camera 34 is disposed inside the right housing 31 (see FIG. 4). The right housing 31 includes a first housing section 31a disposed on the inside (closer to the lighting device 7) and a second housing section 31b connected to the outside of the second housing section 31b via a connecting tube section 31c, and the inside of the first housing section 31a and the inside of the second housing section 31b are in communication with each other via the inside of the connecting tube section 31c.

[0020] Here, the first prism 33 disposed in the first housing part 31a is a cube in which two right-angle prisms 38, 38 are joined by overlapping their inclined faces 38a, 38a, as shown in Figs. 6(A) and (B), and is a so-called cube-type beam splitter. The inclined face 38a of the first prism 33 is coated with a mirror coating with a transmittance (transparency) of 0 to 50%. As shown in Figs. 4 and 7, the first prism 33 is disposed inside the first housing part 31a so that the inclined face 38a is inclined inward and forward in front of the wearer's eyes. The front face of the first prism 33 is disposed opposite the magnifying glass part 32 attached to the front part of the first housing part 31a. The front, rear and right side of the first prism 33 are coated with an anti-reflection coating, and the other three faces are coated with a black coating that suppresses the input and output of light.

[0021] 4, openings (not shown) that face the first prism 33 inside are formed in the front and rear parts of the first housing part 31a, and the magnifying glass part 32 is disposed in the front part. The magnifying glass part 32 is configured to include a lens barrel having a substantially truncated cone shape and two lenses arranged side by side at the front and rear ends of the lens barrel, and is attached integrally to the first housing part 31a so that the lens at the rear end faces the opening in the front part of the first housing part 31a.

[0022] 1 to 4, the first housing portion 31a is provided with a support rod portion 35 that protrudes upward from the upper surface and has a rod portion that is inserted into the first guide elongated hole portion 25 of the guide support member 21 and a fixing portion that is screwed to an upper portion of the rod portion on the first guide elongated hole portion 25 and cannot be inserted into the first guide elongated hole portion 25. By fastening the fixing portion of the support rod portion 35, the first housing portion 31a can be positioned and fixed to the guide support member 21, while by releasing the fastening of the fixing portion, the first housing portion 31a can be relatively moved along the first guide elongated hole portion 25. Similarly, the second housing portion 31b is provided with a support rod portion 36 that protrudes upward from the upper surface and has a rod portion that is inserted into the second guide elongated hole portion 26 of the guide support member 21 and a fixing portion that is screwed to an upper portion of the rod portion on the second guide elongated hole portion 26 and cannot be inserted into the second guide elongated hole portion 26. By tightening the fixing portion of this support rod portion 36, the second housing portion 31b can be positioned and fixed to the guide support member 21, while by releasing the tightening of the fixing portion, the second housing portion 31b can be moved relatively along the second guide long hole portion 26.

[0023] The right-side gaze imaging main body 22 is suspended by the support rods 35, 36 inserted into the first and second guide long hole portions 25, 26 of the guide support member 21. The gaze imaging main body 22 is movable in the left and right lateral directions when the fixing portions of the support rods 35, 36 are not fastened, and is positioned by fastening the fixing portion of at least one of the support rods 35, 36. When the gaze imaging main body 22 is moved left and right in this untightened state, the gaze imaging main body 22 moves left and right lateral directions while tilting in the front and rear directions around the support rods 35 inserted into the first guide long hole portions 25.

[0024] On the other hand, the left line of sight imaging body 23 comprises a left housing 41 supported on the guide support member 21 so as to be slidable in the left-right direction, a magnifying glass part 32 attached to the front part near the inside of the left housing 41, and second prisms 43, 43 (see FIG. 5) disposed inside the left housing 41. Furthermore, an eye camera 44 is disposed inside the left housing 41, and a monitor 51 and a lens 52 are disposed side by side in the front-rear direction (see FIG. 5).

[0025] Here, as shown in Fig. 5, two second prisms 43, 43 are arranged side by side in the left-right direction inside the left housing 41, one second prism 43 is arranged facing the rear of the magnifying glass part 32, and the other second prism 43 is arranged to the left (outside) of the one second prism 43. As shown in Figs. 6(C) and (D), these second prisms 43, 43 are cubes formed of two right-angle prisms 38, 38 like the first prism 33 of the first housing part 31a, and a mirror coating with a transmittance (transparency) of 0 to 50% is applied to the inclined surface 38a. The two second prisms 43, 43 are arranged inside the left housing 41 so that the inclined surface 38a is inclined inward and backward in front of the wearer's eyes, as shown in Fig. 4. The second prism 43 on one side (immediately behind the loupe portion 32) has an anti-reflective coating applied to the front, rear, and left sides, and the other three sides have a black coating applied to suppress light input and output. The other second prism 43 has an anti-reflective coating applied to the front, right, and left sides, and the other three sides have a black coating applied to suppress light input and output.

[0026] As shown in Fig. 5, the left housing 41 has openings (not shown) on the front and rear sides thereof facing the second prism 43 on one side (the inner side), and the magnifying glass part 32 is disposed on the front side. The magnifying glass part 32 has the same configuration as that disposed on the right-side line-of-sight imaging main body 22, and is attached integrally to the left-side housing 41 so that the lens at the rear end faces the opening on the front side of the left-side housing 41. Furthermore, the left housing 41 is provided with a support rod part 35 protruding upward from the upper surface near the inner side, and a support rod part 36 protruding upward from the upper surface near the outer side. These support rod parts 35, 36 have the same configuration as the support rod parts 35, 36 disposed on the right-side line-of-sight imaging main body 22.

[0027] The left-side line of sight imaging main body 23, like the right-side line of sight imaging main body 22 described above, is suspended by the support rods 35, 36 inserted into the first and second guide long hole portions 25, 26 of the guide support member 21. The line of sight imaging main body 23 is positioned by fastening at least one of the fixing portions of the support rods 35, 36. When the line of sight imaging main body 23 is moved left and right in a state where the support rods 35, 36 are not fastened, the line of sight imaging main body 23 moves left and right in the lateral direction and tilts in the front and rear direction around the support rod 35.

[0028] Furthermore, the positions, dimensions, shapes, etc. of the support rod portions 35, 36 of the line-of-sight imaging main bodies 22, 23 and the first and second guide long hole portions 25, 26 of the guide support member 21 are set so that the position T (hereinafter referred to as the viewpoint) at which the lines of sight S1, S2 passing through the first and second prisms 33, 43 intersect, as described below, does not change substantially when the line-of-sight imaging main bodies 22, 23 move.

[0029] In addition, as shown in FIG. 4, the right line-of-sight imaging body 22 described above has a near-infrared camera 34 disposed in the second housing portion 31b and the connecting tube portion 31c. The near-infrared camera 34 can receive light in the near-infrared wavelength range to capture an image, and includes a body portion having an imaging element and a lens barrel portion having a lens, and can adjust the focus by changing the relative distance between the imaging element and the lens. The near-infrared camera 34 is disposed so that the lens of the barrel portion faces the right side surface of the first prism 33. In this embodiment, a disk-shaped operation plate portion 37 for adjusting the focus is provided on the connecting tube portion 31c. The focal length and F-number (aperture) of the near-infrared camera 34 are determined so that the focus can be adjusted at the wearer's viewpoint T through the magnifying glass portion 32.

[0030] A cable 40 is connected to the near-infrared camera 34, and the near-infrared camera 34 is connected to the communication control device 39 via the cable 40. As shown in Figs. 1 and 4, the cable 40 is inserted from the second housing 31b, connected to the frame 3 of the protective glasses 2, and connected to the communication control device 39 held by the wearer. A signal instructing the near-infrared camera 34 to capture an image and image data captured by the near-infrared camera 34 are transmitted and received between the near-infrared camera 34 and the communication control device 39 via the cable 40, and power is supplied from the communication control device 39 to the near-infrared camera 34. The communication control device 39 includes a switch for causing the near-infrared camera 34 to capture an image, a power source, and a wireless communication function, and the wireless communication function allows the image data input from the near-infrared camera 34 to be communicated via a network.

[0031] 5, eye camera 44 is disposed outside the other (outer) second prism 43 in left housing 41 of left line-of-sight imaging body 23. Eye camera 44 can receive light in the visible light wavelength range to capture an image, and includes a main body portion having an imaging element and a mirror tube portion having a lens, with the lens of the mirror tube portion disposed to face the left side surface of outer second prism 43. Here, the focal length, F-number (aperture), etc. of eye camera 44 are determined so that the focus can be adjusted by the wearer's eyeball via second prisms 43,43.

[0032] Furthermore, inside the left housing 41, a monitor 51 is disposed immediately before the outer second prism 43, and a lens 52 is disposed between the second prism 43 and the monitor 51. Here, the lens 52 is disposed opposite to the front surface of the outer second prism 43, and the monitor 51 is disposed opposite to the front surface of the second prism 43 via the lens 52. Here, the monitor 51 is an extremely small monitor on which an image captured by the near-infrared camera 34 is displayed, as will be described later, and the lens 52 is used to enlarge and display the image displayed on the monitor 51.

[0033] As shown in Figs. 1 and 5, the eye camera 44 and the monitor 51 are connected to the communication control device 39 via cables 48 and 49, respectively. Similarly to the near-infrared camera 34, the eye camera 44 and the monitor 51 transmit and receive signals and data between them via the cables 48 and 49, and are supplied with power from the communication control device 39. Here, the image displayed by the near-infrared camera 34 is displayed on the monitor 51 by the communication control device 39. That is, the communication control device 39 is provided with a display control means for displaying an image on the monitor 51 based on image data transmitted from the near-infrared camera 34. This allows the monitor 51 to display an image captured by the near-infrared camera 34 (hereinafter referred to as a near-infrared image) in real time.

[0034] As described above, the left and right line-of-sight imaging bodies 22, 23 are configured with the first and second prisms 33, 43, so that an object located in front of the wearer's line of sight S1, S2 can be viewed in a magnified manner using the magnifying glass portions 32, 32, and can be imaged using the near-infrared camera 34 and the eye camera 44, and the wearer can view the near-infrared image displayed on the monitor 51.

[0035] More specifically, in the right-side line-of-sight imaging body 22, incident light entering from the front through the loupe portion 32 is split by the first prism 33 into transmitted light that passes through the inclined surface 38a of the first prism 33 and reflected light that is reflected outward (to the right) by the inclined surface 38a. As a result, as shown in Fig. 7, the line of sight S1 of the right eye passes through the first prism 33, and the optical axis P1 of the near-infrared camera 34 can coincide with the line of sight S1 of the right eye passing through the first prism 33, so that the near-infrared camera 34 can capture an image at the line of sight of the wearer.

[0036] In the left line-of-sight imaging body 23, the incident light incident from the front through the loupe portion 32 passes through the inner second prism 43, and the light emitted by the monitor 51 is reflected by the inclined surface 38a of the outer second prism 43, refracted inward (to the right), and reflected by the inclined surface 38a of the inner second prism 43, refracted backward. As a result, the line of sight S2 of the left eye passes through the inner second prism 43 and can see the near-infrared image displayed on the monitor 51, so that the left eye can see the object at the viewpoint T and the near-infrared image superimposed on each other. Here, as described above, the monitor 51 displays the near-infrared image captured by the near-infrared camera 34 in real time, so the wearer can see the images viewed with the left and right lines of sight S1 and S2 (images viewed with visible light) and the image captured by the near-infrared camera 34 (images viewed with light in the near-infrared wavelength range) superimposed on each other. The lens 52 disposed immediately behind the monitor 51 is used to adjust the size of the near-infrared image displayed on the monitor 51 to a size that can be viewed through the magnifying glass portion 32.

[0037] Furthermore, as described above, the outer second prism 43 reflects the light from the monitor 51 at the inclined surface 38a and transmits the light in the left and right directions at the inclined surface 38a. As a result, the optical axis P2 of the eye camera 44 passes through the outer second prism 43 and is refracted backward at the inner second prism 43, so that the left eye of the wearer can be imaged from approximately the front. The eye camera 44 focuses on the left eye, captures a moving image of the left eye, and transmits image data of the moving image to the communication control device 39.

[0038] The head-mounted gaze detection device 1 of this embodiment is used by a wearer by putting the protective glasses 2 on the wearer's head and positioning the gaze imaging unit 6 in front of the wearer's eyes (see FIG. 3). Then, the wearer operates the connecting member 5 to move the gaze imaging unit 6 up and down or tilt it in the forward and backward directions, thereby placing the gaze imaging unit 6 and the lighting device 7 at a desired position. Furthermore, the left and right gaze imaging bodies 22, 23 are operated to place the first and second prisms 33, 43 (and the magnifying glass parts 32, 32) of each of the gaze imaging bodies 22, 23 in front of the left and right eyes, respectively. By positioning the left and right gaze imaging bodies 22, 23 in this way to match both eyes of the wearer, the gazes S1, S2 pass through the first and second prisms 33, 43 and the magnifying glass parts 32, 32, as shown in FIG. 7, and the object can be viewed in a magnified view at a viewpoint T where the two gazes S1, S2 intersect.

[0039] By aligning the line-of-sight imaging bodies 22 and 23 in this way, when the wearer's line of sight S1 passes through the center or approximately the center of the first prism 33, the line of sight S1 and the optical axis P1 of the near-infrared camera 34 coincide with each other. This allows the near-infrared camera 34 to image the object at the line of sight of the wearer. That is, by imaging the object (point of view T) with the near-infrared light (light in the near-infrared wavelength range) irradiated by the near-infrared light light 19, image data of the near-infrared image of the object can be generated. Then, when the communication control device 39 receives the image data captured by the near-infrared camera 34, it performs control to display the near-infrared image based on the image data on the monitor 51. This allows the monitor 51 to display the near-infrared image captured by the near-infrared camera 34 in real time. When the wearer views the near-infrared image displayed on monitor 51 with his / her left eye in this manner, the near-infrared image captured by near-infrared camera 34 (an image that is not visible with visible light but is visible with light in the near-infrared wavelength range) is superimposed on the object at viewpoint T viewed with both eyes through the first and second prisms 33, 43 and the magnifying glass portions 32, 32, and can be viewed in real time.

[0040] Next, a gaze image display system 61 having the head-mounted gaze detection device 1 described above will be described. 8, a gaze image display system 61 includes a head-mounted gaze detection device 1, a computer 62 that receives image data transmitted from a communication control device 39 of the head-mounted gaze detection device 1 and performs a predetermined process, a storage device 63 that stores the image data via the computer 62, and an image display device 64 that displays moving images and still images based on the image data. Note that the image data transmitted from the communication control device 39 includes image data captured by the near-infrared camera 34 and image data captured by the eye camera 44.

[0041] The computer 62 includes a means for transmitting and receiving data, and an image control means 66 for displaying a moving image or a still image on the image display device 64. The image control means 66 includes a near-infrared image display means 67 for displaying a near-infrared image 72 (see FIG. 9) on the image display device 64 based on image data transmitted from the near-infrared camera 34, and a gaze image display means 68 for displaying a gaze image 73 (see FIG. 9) indicating a gaze position acquired based on image data transmitted from the eye camera 44 on the image display device 64. Here, the gaze image display means 68 acquires the gaze position of the eyeball on the near-infrared image by a predetermined calculation based on the image data transmitted from the eye camera 44 and the image data transmitted from the near-infrared camera 34, and generates a gaze image 73 indicating the gaze position. Then, the gaze image 73 is displayed on the image display device 64 by being superimposed on the near-infrared image 72. The image control means 66 allows the near-infrared image 72 and the gaze image 73 to be displayed on the image display device 64 in real time.

[0042] The predetermined calculation for obtaining the gaze position by the gaze image display means 68 is set in advance based on the magnification of the loupe units 32, 32, the position of the eyeball (pupil) displayed in the image data captured by the eyeball camera 44, etc., and it is possible to obtain the gaze position in the near-infrared image 72 displayed from the image data captured by the near-infrared camera 34. Here, as described above, the eyeball camera 44 can capture a moving image of the eyeball from approximately in front of the left eye, so that the gaze image display means 68 can accurately obtain the movement of the eyeball by analyzing the image data of the moving image, and can obtain position data (information) of the gaze position accurately and stably.

[0043] Furthermore, computer 62 performs a process of storing the image data received from near-infrared camera 34 and the image data received from ocular camera 44 in chronological correspondence in the storage device 63. Then, by being operated by an administrator or the like, the image data stored in storage device 63 is selectively read, and the near-infrared image and the line-of-sight image are superimposed and displayed on image display device 64 as described above.

[0044] 9, such a gaze image display system 61 can display a near-infrared image 72 on an image display device 64 based on image data captured by the near-infrared camera 34 of the head-mounted gaze detection device 1, and can also display a gaze image 73 acquired from image data captured by the ocular camera 44 superimposed on the near-infrared image 72 on the image display device 64. This allows a person viewing the image display device 64 to see an object 71 including an object visible with light in the near-infrared wavelength range, just like the wearer, and can visually recognize the gaze position of the wearer.

[0045] As described above, the head-mounted gaze detection device 1 of this embodiment allows the wearer to superimpose a near-infrared image seen with light in the near-infrared wavelength range on an object seen with the naked eye (visible light), and therefore the wearer can accurately view the near-infrared image. Here, the optical axis P1 of the near-infrared camera 34 is refracted by the first prism 33 and coincides with the wearer's gaze S1, so the wearer can see the near-infrared image superimposed on the object seen with the naked eye without feeling uncomfortable. Furthermore, the gaze image display system 61 of this embodiment superimposes a viewpoint image obtained from image data captured by the eyeball camera 44 on a near-infrared image captured by the near-infrared camera 34 of the head-mounted gaze detection device 1 and displays it on the image display device 64, so that a person viewing the image display device 64 can view the near-infrared image at the same line of sight as the wearer, and can visually recognize the gaze point of the wearer from the viewpoint image. The configuration of this embodiment can be applied in the medical field. For example, by wearing the head-mounted gaze detection device 1 by a medical worker who draws blood or administers an intravenous drip, the position of the blood vessel can be accurately and easily seen even by a patient who has difficulty in seeing the blood vessel with the naked eye. In addition, by wearing the head-mounted gaze detection device 1 by a surgeon performing surgery, cancers and the like that cannot be seen with the naked eye can be accurately and easily seen. Then, by displaying the near-infrared image captured by the near-infrared camera 34 and the viewpoint image based on the image captured by the ocular camera 44 during such surgery on the image display device 64, assistants and visitors can see the status of the surgery from the same perspective as the surgeon and can know the surgeon's gaze point. As a result, if the images are displayed in real time during surgery, the assistants and visitors can accurately know the status of the surgery, and therefore appropriate advice can be given according to the status of the surgery. Alternatively, the flow of the surgery and the gaze point by the surgeon can be accurately conveyed to visitors. Similarly, after surgery, the image data stored in the storage device 63 can be read and displayed on the image display device 64, which helps to disseminate knowledge about various surgical situations to students and other medical personnel.

[0046] In this embodiment, the protective glasses 2 correspond to the head-mounted part according to the present invention. The first prism 33 arranged in the right-side line-of-sight imaging main body 22 and the inner second prism 43 (located immediately behind the loupe part 32) arranged in the left-side line-of-sight imaging main body 23 correspond to the prism according to the present invention, and constitute the line-of-sight passing member according to the present invention. The near-infrared camera 34 corresponds to the visual field imaging means according to the present invention, and the eye camera 44 corresponds to the eyeball imaging means according to the present invention. The visual field image display means 68 of the computer 62 corresponds to the visual field information acquisition means according to the present invention. The near-infrared image display means 67 of the computer 62 corresponds to the visual field image display means according to the present invention, and the image control means 66 of the computer 62 corresponds to the image control means equipped with the visual field image display means and the visual field image display means according to the present invention. The near-infrared image corresponds to the visual field image according to the present invention, and the image data captured by the near-infrared camera 34 corresponds to the visual field image information according to the present invention. The image data captured by the eye camera 43 corresponds to the eye image information of the present invention, and the position data of the gaze position calculated by the image control means based on the image data corresponds to the gaze position information of the present invention.

[0047] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, the dimensions and shapes of the components in the above-described embodiment can be modified as appropriate.

[0048] In the above-mentioned embodiment, the right line of sight imaging body is provided with a near-infrared camera, but the present invention is not limited to this, and a camera that captures light in the visible light wavelength range may be provided instead of the near-infrared camera. In such a configuration, since the camera captures what can be seen with the naked eye, the configuration may not include the monitor, lens, and outer prism in the embodiment. Furthermore, the present invention may be provided with a camera (field of view imaging means of the present invention) that can receive and capture invisible light including the near-infrared wavelength range. In this configuration, the present invention may be provided with a camera that can receive and capture light (invisible light) in a wavelength range other than the near-infrared range, and is not limited to the near-infrared camera in the embodiment.

[0049] In the above-described embodiment, the image control means of the computer determines position data of the viewpoint position based on image data captured by the eye camera, but this is not limited to the above, and the communication control device may be configured to include a means for determining position data of the viewpoint position (gaze information acquisition means).

[0050] In the above-mentioned embodiment, the left line-of-sight imaging body is configured with a lens between the monitor and the prism, but this is not limited to this and may be configured without a lens. This lens is for adjusting the scale of the image displayed on the monitor to the object seen by the wearer through the magnifying glass, so if the size of the monitor or the display size of the image is adjusted, there is no need to install a lens.

[0051] In the above-described embodiment, the left and right visual axis imaging bodies are each provided with a magnifying glass portion, but the configuration may be such that no magnifying glass portion is provided. [Explanation of symbols]

[0052] 1 Head-mounted gaze detection device 2 Protective glasses (head-mounted part) 33 First Prism 43 Second Prism 34 Near-infrared camera (field of view imaging means) 44 Eye camera (eye imaging means) 64 Image display devices 66 Image Control Means 67 Near-infrared image display means (field image display means) 68 Gaze image display means P1,P2 optical axis S1,S2 line of sight

Claims

1. A head-mounted part to be mounted on the head of a wearer; Left and right line of sight passage members attached to the head mounting portion and allowing the wearer's left and right lines of sight to pass forward, respectively; a visual field imaging means arranged adjacent to one of the left and right line-of-sight passing members and configured to image a front view of the wearer through the line-of-sight passing member; an eyeball imaging means disposed adjacent to the other of the left and right line-of-sight passing members and configured to image an image of the wearer's eyeball through the line-of-sight passing member; a gaze information acquiring means for acquiring information on a gaze position in the visual field image based on information on the visual field image captured by the visual field imaging means and information on the eyeball image captured by the eyeball imaging means; Equipped with The left or right line of sight passage member is a prism that passes the wearer's line of sight forward and refracts the optical axis of the visual field imaging means forward, The other of the left and right line of sight passage members is a head-mounted gaze detection device comprising a prism that passes the wearer's gaze forward and refracts the optical axis of the eye imaging means backward, thereby enabling the eye imaging means to image the other of the left and right eyes.

2. The head-mounted gaze detection device according to claim 1 , an image display device for displaying an image; an image control means for controlling the display of an image on the image display device; Equipped with The image control means a visual field image display means for displaying a visual field image on the image display device based on information of a visual field image captured by the visual field imaging means of the head-mounted gaze detection device; a gaze image display means for displaying, in the image display device, a gaze image indicating the gaze position superimposed on a visual field image displayed by a visual field image display means, based on information on the gaze position acquired by a gaze information acquisition means of the head-mounted gaze detection device; A line-of-sight image display system comprising:

Images