Eye-gaze-detection-based smart glasses display device
The smart glasses display device addresses the issue of scattered lines of sight in medical equipment by using line-of-sight detection to project relevant information at the desired time and point, improving concentration and convenience in medical procedures.
Patent Information
- Application Number
- JP2022521335
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-10-10
- Filing Date
- 2020-10-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-10-12
AI Technical Summary
Conventional medical equipment that does not utilize VR, AR, or MR requires doctors to alternate their gaze between ultrasound monitors and patient information, leading to scattered lines of sight and reduced concentration during diagnosis or surgery.
A smart glasses display device that detects a user's line of sight using laser-irradiated distance measurement units to project an overlay image at the desired point and time, allowing intuitive interaction with virtual objects.
The device maintains the doctor's line of sight on the patient, enhancing concentration and convenience by providing necessary information through augmented or mixed reality without dispersion.
Smart Images

Figure 0007581340000001 
Figure 0007581340000002 
Figure 0007581340000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a smart glasses display device based on gaze detection, and more particularly, to a smart glasses display device based on gaze detection that detects a user's gaze and provides a natural overlay image according to the gaze direction through smart glasses. [Background technology]
[0002] Recently, medicine and healthcare have been in the spotlight as application fields of virtual reality (VR), augmented reality (AR), and mixed reality (MR). VR, AR, and MR technologies using smart glasses have emerged as an alternative to training medical professionals and psychological treatment to meet the increasing demand for medical services, including psychological disorders, due to the advent of an aging society and intensifying competition. In particular, VR, AR, and MR have expanded their scope of application to all areas of medicine, including surgery, treatment, and rehabilitation, moving away from their limited use in trauma treatment. VR, AR, and MR allow training education by embodying virtual patients and organs, and medical staff can make appropriate plans and test them in advance before surgery.
[0003] Conventional medical equipment that does not use VR, AR, and MR, such as ultrasound scanners, requires doctors to check the ultrasound image they want to display on a separate monitor screen, which is troublesome for doctors to alternate between checking the patient and the monitor for treatment. In addition, remote surgery equipment and robotic surgery equipment often cannot efficiently show various information required for surgery to medical staff. As a result, there are cases where doctors have to turn their heads to see the necessary information, or treat patients based on their own intuition without the necessary information during surgery.
[0004] This inconvenience and distraction of the line of sight have acted as factors that hinder doctors' concentration during diagnosis. In particular, if a situation arises where medical staff have to look at related information screens during diagnosis or surgery, and the medical staff's concentration is distracted, the patient may be at risk or may miss areas that need to be observed with concentration.
[0005] Therefore, there is a gradually increasing demand for display methods that can maximize doctors' concentration during diagnosis and maximize the convenience of medical staff by introducing VR, AR, and MR into conventional medical equipment.
[0006] The Background of the Invention is provided to facilitate a better understanding of the present invention and should not be construed as an admission that any matter described in the Background of the Invention is prior art. Summary of the Invention [Problem to be solved by the invention]
[0007] The present invention provides an innovative display method that displays information at a time and place desired by a doctor through AR, MR, VR, etc. in medical equipment, preventing the doctor from distracting his or her line of sight during diagnosis.
[0008] In particular, the problem to be solved by the present invention is to provide a gaze detection-based smart glass display device that detects a user's gaze and displays an image or video containing related information in a specific area that is intuitively comfortable for the user, allowing the user to more accurately grasp the affected area or inside of a patient's body through a virtual object.
[0009] The object of the present invention is not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]
[0010] In order to solve the above-mentioned problems, a smart glasses display device based on gaze detection according to an embodiment of the present invention includes a plurality of distance measurement units that irradiate a laser onto an eyeball including a fundus and measure a first distance from the fundus and a second distance from a surface of the eyeball, a gaze detection unit that detects a gaze direction based on the first distance and the second distance, an image acquisition unit that acquires an overlay image, and a gaze detection unit that detects a gaze within a certain range from the gaze direction. match Recognize the target object and follow the line of sight match The display unit may include a screen synthesis unit that calculates a position so that the overlay image can be displayed at a specified position according to an object, and a display unit in the form of smart glasses that displays the overlay image at the calculated position.
[0011] In this case, the gaze detection unit may sense the position of the pupil based on information indicating whether the distance measured by each of the distance measurement units is a first distance or a second distance.
[0012] In this case, the gaze detection unit may detect a gaze direction by sensing a direction from the eyeball to a pupil based on a first distance measured by a part of the distance measurement units.
[0013] At least some of the distance measuring units may be disposed along a support base that supports the display unit.
[0014] In addition, the display unit provides a screen including a circular guide area, and the screen synthesis unit detects a gaze direction of a user when the gaze moves along the provided guide area, and can obtain a gaze correction reference value based on the gaze direction detected according to the guide area.
[0015] The line of sight correction reference value may include eccentricity, flatness, or a focal position of an ellipse.
[0016] The screen synthesis unit also matchThe light irradiated by the target object is sensed, and a reference point for displaying the overlay image can be extracted.
[0017] The screen synthesis unit may also extract a point spaced a certain distance from the reference point as a screen boundary point.
[0018] In addition, the display unit may display the overlay image in a 3D stereoscopic shape.
[0019] The display unit may also include a microdisplay, a lens, a panel, and a splitter.
[0020] Meanwhile, a method for displaying in smart glasses based on gaze detection according to the present invention includes a distance measuring step of irradiating a laser to an eyeball including a fundus with a plurality of distance measuring units to measure a first distance from the fundus and a second distance from a surface of the eyeball, a gaze detection step of detecting a gaze direction based on the first distance and the second distance, an image acquisition step of acquiring an overlay image, and a gaze detection step of detecting a gaze within a certain range from the gaze direction. match Recognize the target object and follow the line of sight match The method may include a display position calculation step of calculating a position so that the overlay image can be displayed at a specified position according to an object, and a display step of displaying the overlay image at the calculated position via a display unit in the form of smart glasses.
[0021] In this case, the detecting of the line of sight may include sensing a position of a pupil based on information as to whether the distance measured by each of the distance measuring units is a first distance or a second distance.
[0022] The gaze detecting step may detect a gaze direction by sensing a direction from the eyeball to a pupil based on a first distance measured by some of the distance measuring units.
[0023] In addition, the smart glasses display method based on gaze detection may further include a step of: the display unit providing a screen including a circular guide area; detecting a gaze direction of a user when the user's gaze moves along the provided guide area; and determining a gaze correction reference value based on the gaze direction detected according to the guide area.
[0024] The line of sight correction reference value may include eccentricity, flatness, or a focal position of an ellipse.
[0025] The display position calculation step also includes: match The method may include sensing light irradiated by an object and extracting a reference point for displaying the overlay image.
[0026] The display position calculation step may further include extracting a point spaced a certain distance from the reference point as a screen boundary point.
[0027] The displaying step may also include displaying the overlay image in a 3D stereoscopic shape.
[0028] Meanwhile, a computer-readable recording medium according to an embodiment of the present invention includes a plurality of distance measuring units that irradiate a laser beam onto an eyeball including a fundus, measure a first distance from the fundus and a second distance from a surface of the eyeball, detect a gaze direction based on the first distance and the second distance, obtain an overlay image, and detect a gaze within a certain range from the gaze direction. match Recognize the target object and follow the line of sight match A position can be calculated so that the overlay image can be displayed at a specified position according to the object, and a command for displaying the overlay image at the calculated position via a display unit in the form of smart glasses can be stored.
[0029] The means for solving the problems according to the present invention are not limited to the above-mentioned examples, and various other effects are included in the present specification. Effect of the Invention
[0030] Basically, the present invention can provide a device that utilizes VR, AR and MR to display necessary information at a time and place desired by a doctor without dispersing the doctor's gaze.
[0031] The present invention has the effect of displaying an augmented reality image that moves based on the movement of the user's gaze in a specific area that the user feels intuitively comfortable with, allowing the user to grasp the patient's affected area or the inside of the body through a virtual object with high concentration without dispersing the gaze.
[0032] The effects of the present invention are not limited to the above-mentioned examples, and various other effects are included in the present specification. [Brief description of the drawings]
[0033] [Figure 1] 1 is an exemplary diagram illustrating a display process of smart glasses according to an embodiment of the present invention; [Figure 2a] 1 is a schematic diagram of a gaze detection-based smart glasses display system according to an embodiment of the present invention; FIG. [Figure 2b] 1 is a configuration diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention; [Figure 2c] 1 is a block diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention; [Figure 3a] 11A and 11B are diagrams illustrating a calibration mode for gaze detection according to an embodiment of the present invention. [Figure 3b] 11A and 11B are diagrams illustrating a calibration mode for gaze detection according to an embodiment of the present invention. [Figure 4] 4 is an exemplary diagram illustrating a process of generating an image on a display according to an embodiment of the present invention; [Diagram 5] FIG. 13 is a diagram illustrating a smart glass display method according to another embodiment of the present invention. [Figure 6] FIG. 13 is a diagram illustrating a smart glass display method according to another embodiment of the present invention. [Figure 7] FIG. 1 illustrates a smart glass display method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0034] The advantages of the present invention and the method of achieving the same will become clearer with reference to the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various different forms, and the present embodiments are provided merely to make the disclosure of the present invention complete and to fully inform those skilled in the art of the invention of the scope of the invention, and the present invention is only defined by the scope of the claims.
[0035] The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are illustrative only, and the present invention is not limited to the matters shown in the drawings. In addition, in explaining the present invention, if a detailed description of related prior art is determined to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. When "including," "having," "consisting of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it includes the plural unless otherwise explicitly stated.
[0036] In addition, when interpreting elements, they are interpreted as including an error range even if there is no explicit mention otherwise.
[0037] The features of the various embodiments of the present invention may be partially or fully combined or combined with each other, and various technical interlocking and driving mechanisms are possible, as will be fully understood by those skilled in the art, and each embodiment may be implemented independently of each other, or may be implemented together in a linked relationship.
[0038] FIG 1 is an exemplary diagram showing an overlay image display process according to an embodiment of the present invention. FIG 2a is a schematic diagram of an augmented reality display system based on gaze detection according to an embodiment of the present invention. FIG 2b is a configuration diagram of a smart glasses display device based on gaze detection according to an embodiment of the present invention. FIG 2c is a block diagram of a smart glasses display device according to an embodiment of the present invention. FIG 3a and FIG 3b are diagrams for explaining gaze direction correction related to gaze detection according to an embodiment of the present invention. FIG 4 is a diagram for explaining a process of displaying an overlay image on a display.
[0039] Referring to FIG. 1, the eye gaze detection based smart glass display system detects an image or video acquired from an image acquisition unit 205 by detecting the eye gaze. match The display device includes a gaze detection-based smart glass display device 100 and a gaze detection server 900 that display an overlay image or video of augmented reality, mixed reality, or virtual reality in a specific area on a display according to a target object 200. Although the gaze detection-based smart glass display device 100 and the gaze detection server 900 are shown as being separate in the embodiment of FIG. 1, the gaze detection-based smart glass display device 100 and the gaze detection server 900 may be mounted in a single housing. When mounted in a single housing, the gaze detection-based smart glass display device 100 and the gaze detection server 900 may be connected by wire.
[0040] In this embodiment, the line of sight match The object 200 and the image acquisition unit 205 are implemented in one device as an ultrasonic scanner. match The object 200 and the image acquisition unit 205 may be separate devices. For example, matchThe object 200 may be a scalpel or a surgical site of a surgical patient. For example, the image acquisition unit 205 may be a unit that acquires an image showing an area of the nerves of a surgical patient where the scalpel should not enter from an image stored in the gaze detection server 900. That is, the gaze detection server 900 may store an image showing the nerves of a surgical patient based on an image taken of the surgical patient before surgery.
[0041] The smart glasses display device 100 based on gaze detection is a device that detects the gaze of a user using a plurality of built-in distance measuring units 11 and displays an image acquired from an image acquisition unit 205 on the screen of a display unit 103. Here, the smart glasses display device 100 detects the gaze of a user using a plurality of built-in distance measuring units 11 and displays an image acquired from an image acquisition unit 205 on the screen of a display unit 103. match After recognizing the object 200, the recognized gaze is match The image acquired from the image acquisition unit 205 is displayed in a predetermined area related to the object 200. match The object 200 may be an electronic device or a general tool used in surgery, treatment, etc. match The object 200 may be an ultrasonic scanner, scissors, a clip, a scalpel, etc. In this embodiment, the line of sight is match The following description will be given on the basis that the object 200 is an ultrasonic scanner.
[0042] In the present invention, the smart glasses display device 100 based on gaze detection may be a type in which a screen 114 projected on a micro display 110 (see FIG. 2b) of a display unit 103 is reflected and visible through a bilateral lens 111, a panel 112, and a splitter 113. Alternatively, the smart glasses display device 100 may be in a form in which an external screen captured through an external camera 12 of the smart glasses display device 100 or a separate camera in a see-through mode while utilizing an HMD display is superimposed on an overlay screen through a VR screen.
[0043] 2a, 2b, and 2c, the gaze detection-based smart glass display device 100 may include a pupil sensing unit 101, a gaze detection unit 102, a display unit 103, a screen synthesis unit 104, and a control unit 105. The pupil sensing unit 101, the gaze detection unit 102, and the display unit 103, the screen synthesis unit 104, and the control unit 105 may be configured as software, hardware, or a combination of software and hardware, and those skilled in the art may implement such components using appropriate means.
[0044] The pupil sensing unit 101 is configured to sense the pupil of a user and includes a plurality of distance measuring units 11. In this case, the plurality of distance measuring units 11 may be arranged along the edge of the eyeglass frame of the smart glass display device 100 based on gaze detection.
[0045] The distance measuring unit 11 senses the position of the pupil 15 to grasp the movement of the user's line of sight, and is a sensor including a laser emitting unit and a light receiving unit, and preferably irradiates an infrared laser of a wavelength band that does not damage the pupil's vision. For the sake of convenience of explanation, the laser irradiation line (LL) irradiated from the distance measuring unit 11 is shown in FIG. 2a.
[0046] 4, the plurality of distance measuring units 11 can irradiate the eye 10 with infrared light and then measure the distance (d1+d2) reflected from the fundus of the eye or the distance (d2) directly reflected from the surface of the eye via the pupil 15. That is, the reflected distance can be divided into the distance (d1) from the fundus to the surface of the eye (or the pupil) and the distance (d2) from the surface of the eye to the sensor unit 114. In this case, the position of the pupil is sensed by measuring each distance (d1 and d2) obtained via the plurality of sensors.
[0047] Specifically, the laser light emitted from distance measurement unit 11 at a part of the user passes through pupil 15, is reflected by the fundus, which is the inside of the user's eyeball 10, and is sensed again by distance measurement unit 11 via pupil 15.
[0048] At this time, only a part of the laser light emitted from the distance measuring units 11 can pass through the pupil. The laser light that cannot pass through the pupil is reflected by the surface of the eyeball 10, not inside the eyeball 10, and enters the light receiving unit of the distance measuring unit 11.
[0049] The length (d1) from the fundus to the eyeball surface and the length (d2) from the eyeball surface 10 to the display unit 103 can be measured using lasers irradiated by the multiple distance measuring units 11. That is, since the distance (d1+d2) that passes through the pupil 15 is relatively shorter than the distance (d1+d2) that does not pass through the pupil 15, the position of the user's pupil 15 can be grasped by comparing the lengths. In addition, the direction from the eyeball to the pupil can be accurately determined based on whether the laser light irradiated from one distance measuring unit 11 passes through the pupil and / or the distance measured by the light that has passed through, using the distance information from the multiple distance measuring units 11.
[0050] Therefore, as shown in FIG. 4, the present invention measures the distance to the fundus from various positions where a plurality of distance measuring units 11 are positioned, and therefore it is possible to derive a very precise gaze direction.
[0051] In this case, the multiple distance measurement units 11 are arranged along the upper side (L1) and lower side (L2) of the rear surface (surface facing the face) of a support base (spectacles frame) that supports the display unit 103 of the smart glass display device 100. However, the position of the distance measurement units 11 is not limited to this, and they may be arranged on the side of the rear surface of the spectacle frame of the smart glass display device 100 based on the line of sight detection, or may be directly arranged on the lens 111 or panel 112 of the display unit 103 of the smart glass display device 100.
[0052] The gaze detection unit 102 is configured to sense the gaze direction of a user with respect to an augmented reality image to be displayed on the display unit 103, and can detect the gaze direction based on the distance (d1+d2) reflected from the fundus of the eyeball through the pupil, the distance (d2) directly reflected from the surface of the eyeball, and the position of the pupil. A more detailed description of the pupil sensing unit 101 and the gaze detection unit 102 will be given later with reference to Figures 3a and 3b.
[0053] The display unit 103 is match The display unit 103 displays an image acquired from the object 200 as an augmented reality, virtual reality, or mixed reality image on the display unit 103. For example, referring to FIG. 2b, the display unit 103 may include a microdisplay 10, a lens 111, a panel 112, and a splitter 13. In this case, when the microdisplay 10 irradiates an image through the lens 111 disposed in front, the irradiated image reaches an area of the panel 112 through the lens 111, and then the image is totally reflected by the splitter 13 attached to one surface of the display panel 12, and the image 114 can be seen by the user's eyes again through the splitter 13 in front of the eye.
[0054] In addition, the display unit 103 can display the image or video acquired by the image acquisition unit 205 as an augmented reality, virtual reality, or mixed reality image, as well as various information related to the acquired image. For example, when a medical staff member cuts a patient's blood vessel during surgery, the second image 220 can be displayed superimposed on the first image 210 to indicate that there is approximately 60.0 seconds remaining until the blood vessel can be safely reconnected.
[0055] The augmented reality, virtual reality or mixed image (hereinafter, overlay image) acquired by the image acquisition unit 205, for example, an image scanned by an ultrasonic scanner, is matchThe image is overlaid on a specific position designated by the object 200 via the display 103. For example, the image acquired by the image acquisition unit 205 may be an ultrasound image of a pregnant woman undergoing an ultrasound examination. In this case, the smart glass display device 100 directs the line of sight match It is recognized whether the object 200 (in the embodiment of FIG. 4, an ultrasonic scanner) is within a specific range 250 from the line of sight. match The overlay image acquired by the acquisition unit 205 is superimposed on a specific position of the object 200 (for example, the ultrasonic irradiation surface of the ultrasonic scanner in the case of an ultrasonic scanner).
[0056] The images are displayed on the left and right lenses of the gaze detection-based smart glass display device 100. At this time, the left and / or right lenses can display the same image. As shown in FIG. 1b, the images can be displayed as a left eye image (T1') and a right eye image (T1') for the same object, allowing the user to recognize the virtual object (T1) in a three-dimensional shape (3D).
[0057] Also, the image can move within the display unit 103 based on the movement of the user's line of sight, which will be described in detail later.
[0058] The screen synthesis unit 104 can arrange and synthesize the image or video acquired by the image acquisition unit 205 at an appropriate position. The screen synthesis unit 104 can adjust the line of sight according to the video acquired by the external camera 12. matchThe position of the object 200 is confirmed, and the position where the overlay image should be displayed is calculated in conjunction with the line of sight. In the embodiment of the ultrasound scanner of FIG. 4, the position where the overlay image is displayed is the ultrasound irradiation surface of the ultrasound scanner. The screen synthesis unit 104 can control the display 103 to display the ultrasound image at the calculated position. In this case, when scanning the abdomen of the parturient woman with ultrasound, the doctor can check that the scanned image is displayed well while looking at the ultrasound irradiation surface. Therefore, the doctor can perform ultrasound examination while adjusting the position of the fetus and the desired part so that they are well scanned without dispersing the line of sight.
[0059] When the smart glass display device 100 is an HMD showing virtual reality, the screen synthesis unit 104 may superimpose and display the image of the external camera 12 and the image of the image acquisition unit 205. However, even in this case, the image of the external camera 12 and the image of the image acquisition unit 205 are superimposed and displayed after calculating the position where the overlay image should be displayed in conjunction with the line of sight.
[0060] The control unit 105 can perform a function of controlling the overall operations of the pupil sensing unit 101, the gaze detection unit 102, the display unit 103, and the screen synthesis unit 104.
[0061] A method for correcting the distance estimated from the pupil position and the measured distances (d1 and d2) by the image synthesis unit 104 will be described in detail below with reference to Figs. 3a and 3b.
[0062] Specifically, as shown in Fig. 3a, the screen synthesis unit 104 provides the user with an arbitrary screen 300 displaying a circular guide area (TR) in a calibration mode via the display 103. Here, the calibration mode is a mode for measuring the user's line of sight, and the user is guided to move his / her line of sight from an initial starting point (SP) along the guide area (TR) displayed on the arbitrary screen 300 in a clockwise direction (user-based, which may be counterclockwise) along the edge of the guide area (TR).
[0063] At this time, when the user moves the line of sight along the guide region (TR), the line of sight detection unit 102 detects the line of sight of the user. In this case, as shown in FIG. 3b, the ideal line of sight of the user has an ideal circular orbit, but the actual line of sight of the user is formed to have an elliptical orbit that is narrower in width and wider in height than the ideal line of sight of the user. That is, the line of sight direction that the user can gaze comfortably usually has an elliptical shape rather than an accurate circular shape. In this case, the image synthesis unit 104 obtains a line of sight correction reference value based on the actual line of sight direction measured in the calibration mode. In this case, the line of sight correction reference value may include values that can represent an ellipse, such as eccentricity, flatness, and the focal position of an ellipse.
[0064] Therefore, the smart glasses display device 100 according to the present invention can correct the gaze direction detected by the gaze detection unit 102 based on the gaze correction reference value. Therefore, the gaze direction that the user actually intends to gaze can be accurately estimated, and the accuracy of the display position of the overlay image in screen processing is significantly increased.
[0065] In this embodiment, the line of sight is determined based on an image acquired via the external camera 12. match Although the position of the target object 200 is grasped, the present invention is not limited to this. For example, the image synthesis unit 104 may match The line of sight is determined by using beacons 260 mounted at at least three positions on the target object 200. match It is also possible to grasp the position of the target object 200. That is, the image synthesis unit 104 determines the line of sight match The beacon 260 mounted on the target 200 periodically detects the signal and adjusts the line of sight. match The position of the target object 200 can be ascertained.
[0066] Hereinafter, still another embodiment of the present invention will be described with reference to FIG.
[0067] In the embodiment of FIG. match The object 200 may be light irradiated by a light irradiating unit 260 mounted on one side of the ultrasound scanner. For example, when a doctor is examining an area of interest within an entire ultrasound image using the ultrasound scanner, the doctor turns on the light irradiating unit 260 mounted on the ultrasound scanner to indicate the desired location.
[0068] In this case, when the light irradiated by the light irradiating unit 260 is sensed through the external camera 12, as shown in FIG. 5, screen boundary points (P1-P4, P2-P3) are calculated with the point where the light is irradiated as the center. Here, if the point where the infrared laser is irradiated is considered as the reference point, four points (P1, P2, P3, P4) spaced apart at a certain distance in the diagonal direction from the reference point can be extracted as screen boundary points. In this way, the screen synthesis unit 104 calculates the line of sight. match The position of the overlay image 210 can be calculated based on the light emitted from the light emitting unit 260, which is the object 200.
[0069] 2a is an embodiment, and some components may be deleted or new components may be added as needed by those skilled in the art. For example, the smart glass display device 100 based on gaze detection may further include a controller connected via wired or wireless communication.
[0070] Hereinafter, still another embodiment of the present invention will be described with reference to FIG.
[0071] In the embodiment of FIG. match The object 200 may be a scalpel and the face of the person undergoing surgery. In this case, the overlay image 210 may be a planned surgical line. For example, when a doctor holds a scalpel and performs surgery to cut the patient's face, the surgical line image may be overlaid on the patient's face. In this case, when the patient's face and scalpel are detected within a certain range of the surgeon's line of sight in the image captured by the external camera 12, the surgical line, which is an overlay image, is displayed superimposed on the patient's face. In other words, the surgeon's line of sight is match There may be a plurality of objects 200, and when various conditions are satisfied (when at least a part of the scalpel and the face of the surgical subject are detected), the image synthesis unit 104 directs the gaze match The position of the overlay image 210 can be calculated based on the face of the surgical subject, which is the object 200. In this case, the overlay image 210 can be information on the reference situation (information on whether the depth of the scalpel inserted into the skin is appropriate). Alternatively, the location of nerves / blood vessels that must not be touched during surgery can be displayed as the overlay image 210. In this case, the overlay image 210 that is overlaid (superimposed) can be displayed by a known image processing method including an image block search or edge detection method.
[0072] Therefore, the present invention allows medical staff to focus solely on their primary tasks, such as surgery or examination, without distracting them by displaying an overlay image in a specific area of the smart glasses that moves based on the direction of gaze.
[0073] Hereinafter, a display method of smart glasses based on gaze detection according to an embodiment of the present invention will be described in detail with reference to FIG.
[0074] First, the display method of smart glasses according to an embodiment of the present invention includes a distance measurement step S110 in which a laser is irradiated onto an eyeball including a fundus by a plurality of distance measurement units 11 to measure a first distance from the fundus and a second distance from the surface of the eyeball.
[0075] In this case, the gaze detection unit 102 can detect the gaze direction based on whether the distance measured by each distance measurement unit 11 is reflected from the fundus or the surface of the eyeball, i.e., the first distance and the second distance (gaze detection step, S120).
[0076] Meanwhile, the display method of smart glasses acquires an overlay image. The overlay image can be acquired from an ultrasonic scanner, an external camera, a separately connected camera, or a server that stores an image / video, as described above (image acquisition step, S130). In this case, the gaze detection unit 102 can sense the position of the pupil based on information on whether the distance measured by each of the plurality of distance measurement units is a first distance or a second distance. Alternatively, the gaze detection unit 102 can detect the gaze direction sensing the direction from the eyeball to the pupil based on the first distance measured by some of the plurality of distance measurement units.
[0077] The image synthesis unit 104 detects a line of sight within a certain range from the line of sight direction. match Recognize the target object and follow the line of sight match A position can be calculated so that the overlay image can be displayed at a position designated according to the object (display position calculation step, S140). At this time, the screen synthesis unit 104 provides a screen including a circular guide area by the display unit 103, and detects the user's line of sight when the user moves his / her line of sight along the provided guide area, and can obtain a line of sight correction reference value based on the line of sight direction detected according to the guide area.
[0078] Meanwhile, the display position calculation step S140 calculates the line of sight matchThe display position calculation step may further include extracting a point spaced a certain distance from the reference point as a screen boundary point.
[0079] Finally, the display unit 103 in the form of smart glasses may include a display step of displaying the overlay image at the calculated position.
[0080] Therefore, according to the present invention, a device can be provided that utilizes VR, AR, and MR to display necessary information at a time and location desired by the doctor without dispersing the doctor's gaze.
[0081] In addition, the present invention has the effect of displaying an augmented reality image that moves based on the movement of the user's gaze in a specific area that the user feels intuitively comfortable with, allowing the user to grasp the patient's affected area or the inside of the body through a virtual object with high concentration without having to disperse their gaze.
[0082] Although the embodiments of the present specification have been described in detail above with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications can be made within the scope of the technical idea of the present invention. Therefore, the embodiments disclosed in the present invention are for illustration purposes, not for limiting the technical idea of the present invention, and the scope of the technical idea of the present invention is not limited by such embodiments. Therefore, it should be understood that the embodiments described above are illustrative and not limiting in all respects. The scope of protection of the present invention should be interpreted by the scope of the claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present specification. The inventions disclosed herein include the following: [Aspect 1] a plurality of distance measuring units that irradiate a laser beam onto an eyeball including a fundus to measure a first distance from the fundus and a second distance from a surface of the eyeball; a gaze detection unit that detects a gaze direction based on the first distance and the second distance; an image acquisition unit for acquiring an overlay image; Recognizing an object to which the line of sight is directed within a certain range from the line of sight direction; a screen synthesis unit that calculates a position so that the overlay image can be displayed at a designated position according to the target object to which the line of sight is directed; and The overlay image is displayed at the calculated position, and the smart glasses display device includes a display unit in the form of smart glasses. [Aspect 2] A smart glasses display device based on gaze detection as described in aspect 1, wherein the gaze detection unit senses the position of the pupil based on information on whether the distance measured by each of the multiple distance measurement units is a first distance or a second distance. [Aspect 3] The gaze detection unit detects the gaze direction by sensing the direction from the eyeball to the pupil based on a first distance measured by a portion of the plurality of distance measurement units. [Aspect 4] A smart glass display device with a gaze detection base as described in aspect 1, wherein at least some of the multiple distance measurement units are arranged along a support base that supports the display unit. [Aspect 5] The display unit provides a screen including a circular guide area, and the screen synthesis unit detects the user's gaze direction when the gaze moves along the provided guide area, and calculates a gaze correction reference value based on the gaze direction detected according to the guide area. [Aspect 6] A smart glass display device based on gaze detection as described in embodiment 5, wherein the gaze correction reference value includes eccentricity, flatness, or focal position of an ellipse. [Aspect 7] The smart glasses display device based on gaze detection according to aspect 1, wherein the screen synthesis unit senses light irradiated by the object to which the gaze is directed and extracts a reference point for displaying the overlay image. [Aspect 8] The smart glasses display device based on gaze detection according to aspect 7, wherein the screen synthesis unit extracts a point spaced a certain distance from the reference point as a screen boundary point. [Aspect 9] The display unit of the smart glasses display device based on gaze detection according to aspect 1 displays the overlay image in a 3D stereoscopic shape. [Aspect 10] The display unit of the smart glasses display device based on gaze detection described in embodiment 1 includes a microdisplay, a lens, a panel, and a splitter. [Aspect 11] a distance measuring step of measuring a first distance from the fundus and a second distance from a surface of the eyeball by irradiating a laser beam onto the eyeball including the fundus using a plurality of distance measuring units; a gaze detection step of detecting a gaze direction based on the first distance and the second distance; an image acquisition stage for acquiring an overlay image; a display position calculation step of calculating a position of a target object to be gazed at within a certain range from the gaze direction, and calculating a position of the target object to display the overlay image at a designated position according to the target object; A smart glass display method based on gaze detection, comprising: displaying the overlay image at the calculated position through a display unit in the form of smart glasses. [Aspect 12] The gaze detection-based smart glasses display method described in aspect 11, wherein the gaze detection step senses the position of the pupil based on information on whether the distance measured by each of the plurality of distance measurement units is a first distance or a second distance. [Aspect 13] The gaze detection step detects the gaze direction by sensing the direction from the eyeball to the pupil based on a first distance measured by a portion of the distance measurement units. [Aspect 14] providing a screen including a circular guide area on the display unit; The gaze detection-based smart glasses display method according to aspect 11 further includes a step of detecting the user's gaze direction when the user's gaze moves along the provided guide area, and calculating a gaze correction reference value based on the gaze direction detected according to the guide area. [Aspect 15] A smart glass display method based on gaze detection according to embodiment 14, wherein the gaze correction reference value includes eccentricity, flatness or the focal position of an ellipse. [Aspect 16] The gaze detection-based smart glasses display method according to aspect 11, wherein the display position calculation step includes a step of sensing light irradiated by the object to be gazed at and extracting a reference point for displaying the overlay image. [Aspect 17] The method of claim 16, wherein the display position calculation step further includes the step of extracting a point spaced a certain distance from the reference point as a screen boundary point. [Aspect 18] The gaze detection-based smart glasses display method according to claim 11, wherein the display step includes a step of displaying the overlay image in a 3D stereoscopic shape. [Aspect 19] irradiating a laser beam onto an eyeball including a fundus by a plurality of distance measuring units to measure a first distance from the fundus and a second distance from a surface of the eyeball; Detecting a line of sight direction based on the first distance and the second distance; Get the overlay image, Recognizing an object to which the line of sight is directed within a certain range from the line of sight direction; calculating a position so that the overlay image can be displayed at a designated position according to the target object; A computer-readable recording medium storing an instruction for displaying the overlay image at the calculated position via a display unit in the form of smart glasses.
Claims
1. a plurality of distance measuring units configured to irradiate a laser beam onto an eyeball including a fundus and measure a first distance from the fundus and a second distance from a surface of the eyeball; a gaze detection unit that detects a gaze direction based on the first distance and the second distance; an image acquisition unit for acquiring an overlay image; Recognizing an object to which the line of sight is directed within a certain range from the line of sight direction; a screen synthesis unit that calculates a position so that the overlay image can be displayed at a designated position according to the target object to which the line of sight is directed; and The overlay image is displayed at the calculated position, and the display device includes a display unit in the form of smart glasses.
2. The smart glasses display device based on gaze detection according to claim 1 , wherein the gaze detection unit senses a pupil position based on information on whether a distance measured by each of the distance measurement units is a first distance or a second distance.
3. The smart glasses display device based on gaze detection according to claim 1, wherein the gaze detection unit detects a gaze direction by sensing a direction from the eyeball to a pupil based on a first distance measured by a part of the distance measurement units.
4. The smart glasses display device according to claim 1 , wherein at least some of the distance measuring units are arranged along a support base that supports the display unit.
5. 2. The smart glasses display device based on gaze detection according to claim 1, wherein the display unit provides a screen including a circular guide area, and the screen synthesis unit detects a gaze direction of a user when the gaze moves along the provided guide area, and obtains a gaze correction reference value based on the gaze direction detected according to the guide area.
6. The smart glasses display device based on gaze detection according to claim 5 , wherein the gaze correction reference value includes an eccentricity, a flatness, or a focal position of an ellipse.
7. The smart glasses display device based on gaze detection of claim 1 , wherein the screen synthesis unit senses light irradiated by the target object to which the gaze is directed, and extracts a reference point for displaying the overlay image.
8. The smart glasses display device according to claim 7 , wherein the screen synthesis unit extracts a point spaced a certain distance from the reference point as a screen boundary point.
9. The smart glasses display device based on gaze detection of claim 1 , wherein the display unit displays the overlay image in a 3D shape.
10. The smart glasses display device of claim 1 , wherein the display unit includes a microdisplay, a lens, a panel, and a splitter.
11. a distance measuring step of measuring a first distance from the fundus and a second distance from a surface of the eyeball by irradiating a laser beam onto the eyeball including the fundus using a plurality of distance measuring units; a gaze detection step of detecting a gaze direction based on the first distance and the second distance; an image acquisition stage for acquiring an overlay image; a display position calculation step of calculating a position of a target object to be gazed at within a certain range from the gaze direction, and calculating a position of the target object to display the overlay image at a designated position according to the target object; A smart glass display method based on gaze detection, comprising: displaying the overlay image at the calculated position through a display unit in the form of smart glasses.
12. 12. The method of claim 11, wherein the gaze detection step comprises sensing a pupil position based on information on whether a distance measured by each of the distance measuring units is a first distance or a second distance.
13. 12. The method of claim 11, wherein the gaze detection step detects a gaze direction by sensing a direction from the eyeball to a pupil based on a first distance measured by a part of the distance measurement units.
14. providing a screen including a circular guide area on the display unit; The method of claim 11, further comprising: detecting a gaze direction of the user when the user's gaze moves along a provided guide area; and determining a gaze correction reference value based on the gaze direction detected according to the guide area.
15. The method of claim 14 , wherein the gaze correction reference value includes eccentricity, flatness, or a focus position of an ellipse.
16. 12. The method of claim 11, wherein the display position calculation step comprises the step of sensing light irradiated by the object to be gazed at and extracting a reference point for displaying the overlay image.
17. The method of claim 16, wherein the calculating the display position further comprises extracting a point spaced a certain distance from the reference point as a screen boundary point.
18. The method of claim 11 , wherein the displaying step comprises displaying the overlay image in a 3D stereoscopic shape.
19. irradiating a laser beam onto an eyeball including a fundus by a plurality of distance measuring units to measure a first distance from the fundus and a second distance from a surface of the eyeball; Detecting a line of sight direction based on the first distance and the second distance; Get the overlay image, Recognizing an object to which the line of sight is directed within a certain range from the line of sight direction; calculating a position so that the overlay image can be displayed at a designated position according to the target object; A computer-readable recording medium storing an instruction for displaying the overlay image at the calculated position via a display unit in the form of smart glasses.
Citation Information
Patent Citations
Medical display unit and system
JP2008018015A
Device for estimation of eye potential, method for calculation of eye potential, visual axis detector, wearable camera, head-mounted display, and electronic glasses
JP2011125693A
Spectacle type display device and server
JP2012008290A
Systems, devices, and methods for proximity-based eye tracking
US20170205876A1
