Multimodal recognition device for health screening based on eye imaging and eye tracking

Abstract

The present invention relates to a multimodal recognition device for realizing health screening based on eye imaging and eye movement tracking, comprising a desktop end and a remote end, wherein the desktop end comprises: an image acquisition unit, which is arranged in the first screen display unit and acquires image information and eye movement information of the patient's eye along a first direction, wherein the image information comprises: an iris image, an eye surface image, an eye fundus image, and an eye position image; a first control unit, which recognizes the identity of the user in real time according to the iris image, and determines the health status of the user according to the eye surface image, the eye fundus image, the eye position image, and the eye movement tracking information. By acquiring the patient's iris image, real-time iris recognition and identity authentication are performed to ensure that the patient is always the same person, thereby improving the effectiveness of health screening. By acquiring the patient's eye surface image, the eye fundus image, the eye position image, and the eye movement data, the patient's health status is analyzed from multiple modalities to ensure the diversity and accuracy of health screening.

Description

Technical Field

[0001] The present invention relates to the field of visual tracking technology, and in particular to a multimodal recognition device for realizing health screening based on eye imaging and eye movement tracking. Background Art

[0002] Eye imaging technology plays an important role in health screening and disease diagnosis. However, traditional eye imaging equipment usually requires professional doctors to operate and interpret, and there are problems such as complex use, high cost, and patients need to go to the hospital. At present, some eye imaging technologies have been widely used in the medical field. Iris imaging technology can quickly identify individuals by collecting iris images and extracting their features. Ocular surface images and fundus images provide important information about eye structure and lesions, which can be used for early detection and diagnosis of eye diseases. In addition, eye tracking and eye position images can evaluate the normality of eye movements and help determine diseases such as abnormal eye position and strabismus. However, there are some problems with traditional eye tracking and eye imaging equipment, such as high cost, large size, inconvenient use, and high professional knowledge requirements, which limit their application in health screening.

[0003] For example, fundus image acquisition technology is applied to diabetes screening, which usually focuses on retinal vascular abnormalities, leakage and bleeding, and changes in the macular area. During existing retinal photography, patients are required to keep their heads stable and not move around, and their eyes must be fixed on a specific target. When shooting from multiple angles, users are also required to follow the operator's guidance to change the target of gaze, which tests the operator's professionalism and cooperation with the patient. Eye tracking technology is used to screen for amblyopia and strabismus, and analysis is usually performed based on eye movement data.

[0004] Fundus image acquisition usually uses professional equipment, such as professional fundus cameras or optical coherence tomography (OCT) scanners, with a resolution of more than 10 megapixels. However, with the advancement of technology, ordinary SLR cameras can meet the resolution requirements of fundus image acquisition. Professional-grade SLR cameras are even above 30 megapixels. The cameras of eye tracking devices on the market also generally have a resolution of 10 megapixels or above. Combined with the visible light fill light system, fundus image acquisition has the possibility of moving away from professional scenarios and moving towards a more convenient, flexible and intelligent direction.

[0005] Therefore, it is necessary to provide a multimodal recognition device for health screening based on eye imaging and eye tracking to achieve convenient health screening and remote diagnosis. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal recognition device for health screening based on eye imaging and eye tracking, so as to realize convenient health screening and remote diagnosis.

[0007] According to one aspect of the present invention, a multimodal recognition device for health screening based on eye imaging and eye tracking is provided, comprising a desktop terminal for use by patients and a remote terminal that interacts with information of the desktop terminal and is used by doctors, wherein the desktop terminal comprises:

[0008] A first voice unit, establishing voice interaction with the remote end;

[0009] A first screen display unit, electrically connected to the first voice unit and facing the patient;

[0010] An image acquisition unit is provided in the first display unit and acquires image information of the patient's eyes along a first direction, wherein the image information includes: an iris image, an ocular surface image, an ocular fundus image, and an eye position image;

[0011] The first control unit recognizes the identity of the user in real time according to the iris image, and determines the health status of the user according to the ocular surface image, the fundus image, and the eye position image.

[0012] More preferably, the fundus image includes: macular features;

[0013] The eye position image includes: pupil features;

[0014] The first control unit marks a first tracking point on the pupil of the eye position image, marks a second tracking point on the macula of the fundus image, tracks the rotation of the patient's eyeball according to the first tracking point, and controls the image acquisition unit to automatically focus with the second tracking point as the focus point.

[0015] More preferably, the first tracking point is the geometric center point of the pupil, and the second tracking point is the geometric center point of the macula.

[0016] The first control unit locates the position of the pupil in the eyeball image according to the first tracking point to track the angle of eyeball rotation, and uses the second tracking point as a focus point to control the image acquisition unit to automatically focus.

[0017] Preferably, the desktop terminal further includes:

[0018] A fill light unit, including an infrared light and a visible light;

[0019] The first screen display unit includes: a first display screen, wherein an extended plane on which a side of the first display screen facing the user is located is recorded as a first reference plane,

[0020] Observing along the first direction, the first tracking point is the projection of the geometric center of the patient's pupil on the first reference plane when the patient is looking directly at the first display screen, the projection of the midpoint of the infrared lamp on the first reference plane is marked as the first marking point, and the projection of the visible light lamp on the first reference plane is marked as the second marking point. The first marking point is located above the first tracking point, and the second marking point is located on the same horizontal line as the first tracking point.

[0021] More preferably, the first display screen includes:

[0022] A normal display unit, configured in a portion of the display area of ​​the first display screen;

[0023] An under-screen display unit, configured in another display area of ​​the first display screen;

[0024] The image acquisition unit comprises:

[0025] The under-screen camera is arranged on the side of the under-screen display unit facing away from the patient, and collects eyeball images and fundus images along a first direction.

[0026] More preferably, the direction perpendicular to the first direction is recorded as the second direction, and the surface perpendicular to the first reference plane and perpendicular to the second direction is recorded as the second reference plane;

[0027] Observing along the second direction, the first tracking point is the projection of the geometric center of the pupil on the second reference plane when the patient is looking straight at the first display screen, and the second tracking point is the projection of the geometric center of the macula on the second reference plane when the patient is looking straight at the first display screen;

[0028] The first control unit establishes an eyeball model based on the eyeball image, obtains the position of the eyeball center in the second reference plane through edge detection method, and the projection of the midpoint of the under-screen camera on the second reference plane is marked as a third marking point, and the eyeball center and the third marking point are located on the same horizontal line.

[0029] Preferably, the desktop terminal further includes:

[0030] A lens assembly is located on a side of the first display screen away from the under-screen camera, and along a direction away from the display screen, the lens assembly includes:

[0031] A linear polarizer, attached to a side of the first display screen facing away from the under-screen camera;

[0032] A first lens is spaced apart and arranged on a side of the linear polarizer away from the first display screen;

[0033] a second lens, spaced apart and arranged on a side of the first lens away from the first display screen;

[0034] an infrared anti-reflection reflective polarizer, used for reflecting and passing polarized light of visible light and passing infrared light, and located on a side of the second lens close to the first display screen;

[0035] The infrared light emitted by the infrared lamp irradiates the pupil and is reflected to the screen display camera through the lens assembly; the visible light emitted by the visible light lamp irradiates the eyeball and is reflected to the screen display camera through the lens assembly.

[0036] Preferably, the remote end comprises:

[0037] A second screen display unit receives and plays the eyeball image and fundus image;

[0038] A second voice unit establishes voice interaction with the desktop terminal;

[0039] A second control unit establishes signal connections with the second display unit, the second voice unit, and the desktop end respectively;

[0040] When the patient looks directly at the under-screen camera, the doctor calibrates the geometric center point of the macula on the fundus image as the second tracking point through the second control unit;

[0041] When the patient moves his eyeball, the first control unit tracks the movement of the eyeball according to the first tracking point, and controls the automatic focusing of the under-screen camera with the second tracking point as the focus point;

[0042] When the doctor wants to take pictures of other retinal areas besides the macula, the third tracking point is calibrated on the fundus image through the second control unit, and the second control unit controls the automatic focus of the under-screen camera with the third tracking point as the focus point.

[0043] Preferably, when the patient moves the eye and the physician wants to image other areas of the retina besides the macula,

[0044] The eyeball model is reconstructed using the line between the eyeball center and the first tracking point after movement as the optical axis, the line between the third tracking point and the first tracking point as the visual axis, and the line between the eyeball center and the center of the under-screen camera as the reference axis;

[0045] When observing along the first direction, when the doctor calibrates the third tracking point on the fundus image, observing along the second direction, the second control unit locates the position of the third tracking point on the retina in the eyeball model based on the relative position of the third tracking point and the second tracking point, and the second control unit controls the automatic focus of the under-screen camera based on the position of the third tracking point on the retina.

[0046] The present invention has the following beneficial effects:

[0047] By collecting the patient's iris image, real-time iris recognition and identity authentication are performed to ensure that the patient is always the same person, improving the effectiveness of health screening. By collecting the patient's ocular surface image, fundus image and eye position image, the patient's health status is analyzed from multiple modalities to ensure the diversity and accuracy of health screening.

[0048] By using the pupil as the first tracking point and the macula as the second tracking point, it is ensured that the macular area can maintain clear imaging when collecting fundus images, thereby improving the quality and usability of the image, especially improving the judgment of diabetic symptoms. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 This is a schematic diagram of the parts of the human eye;

[0051] Figure 2 A schematic diagram of the projection of the patient's eyes and the fill light unit on the first reference plane for observation along the first direction;

[0052] Figure 3 A schematic diagram of the projection of the patient's eyeball model, the lens assembly, the fill light unit, the screen display unit, and the image acquisition unit on the second reference plane for observation along the second direction;

[0053] Figure 4 A schematic diagram of the projection of the patient's eye and the fill light unit on the first reference plane when the patient's eyeball moves and is observed along the first direction;

[0054] Figure 5 A schematic diagram of projections of the patient's eyeball model, the lens assembly, the fill light unit, the screen display unit, and the image acquisition unit on the second reference plane when the patient's eyeball moves and is observed along the second direction;

[0055] Figure 6 is a structural block diagram of the multimodal recognition device of the present invention;

[0056] Figure 7 It is a structural schematic diagram of the multimodal recognition device of the present invention;

[0057] Figure 8 It is a logic diagram for information transmission of eyeball image and fundus image in the system;

[0058] Fig. 9It is a structural block diagram of the first display screen.

[0059] Description of the accompanying drawings: 100, device; 200, desktop end; 300, remote end; 220, first voice unit; 230, first screen display unit; 240, image acquisition unit; 250, first control unit; 260, fill light unit; 270, lens assembly; 310, second screen display unit; 320, second voice unit; 330, second control unit; 261, infrared light; 262, visible light; 231, first display screen; 2311, normal display unit; 2312, under-screen display unit; 241, under-screen camera; 271, linear Polarizer; 272, first wave plate; 273, first lens; 274, second wave plate; 275, infrared anti-reflection reflective polarizer; 276, second lens; 410, pupil; 420, macula; 430, eyeball; 510, first tracking point; 520, second tracking point; 530, third tracking point; L1, optical axis; L2, visual axis; L3, reference axis; 531, midpoint of eyeball; 440, retina; 540, iris image; 550, ocular surface image; 560, fundus image; 570, eye position image; F1, first direction; F2, second direction. DETAILED DESCRIPTION

[0060] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0061] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0063] Please refer to Figure 1 - Fig. 9An embodiment of the present invention provides a multimodal recognition device 100 for health screening based on eye imaging and eye tracking, including a desktop terminal for patients and a remote terminal that interacts with the desktop terminal and is used by doctors.

[0064] Among them, the multimodal recognition device 100 for health screening based on eye imaging and eye tracking can play an important role in a variety of application scenarios. For example:

[0065] Remote ophthalmology diagnosis and treatment: In remote areas or places with scarce medical resources, patients can use desktop devices to perform eye imaging and then conduct diagnosis and treatment with ophthalmologists through remote connections. Doctors can observe patients' eye images in real time and interact through the remote terminal to provide accurate diagnosis and treatment recommendations. This can greatly improve the efficiency and quality of remote ophthalmology diagnosis and treatment, allowing more people to receive timely ophthalmic medical services.

[0066] Community health screening: In community health service centers or medical institutions, patients can easily perform eye imaging and transmit the images to a remote end for doctors to evaluate. Doctors can use the multimodal recognition device 100 to quickly screen eye health conditions, detect eye diseases or lesions early, and provide corresponding treatment recommendations or refer patients to specialized hospitals.

[0067] Home health monitoring: The multimodal recognition device 100 can be used as one of the tools for home health monitoring. Patients can regularly perform eye imaging at home and send the images to doctors. Doctors can analyze eye images through remote terminals, monitor changes in eye health, and provide necessary advice and intervention measures. This home health monitoring method is convenient and real-time, and can help patients pay attention to eye health problems in a timely manner and prevent the development of eye diseases.

[0068] This embodiment takes community health screening as an example to introduce the structure and application of the device 100.

[0069] Specifically, the desktop terminal includes: a first voice unit 220, a first screen display unit 230, an image acquisition unit 240, and a first control unit 250. The first voice unit 220 establishes voice interaction with the remote terminal. The first screen display unit 230 is electrically connected to the first voice unit 220, and the first screen display unit 230 faces the patient. The image acquisition unit 240 is arranged in the first screen display unit 230, and collects image information of the patient's eyes along the first direction F1, and the image information includes: an iris image 540, an ocular surface image 550, a fundus image 560, and an eye position image 570. The first control unit 250 recognizes the user's identity in real time according to the iris image 540, and determines the user's health status according to the ocular surface image 550, the fundus image 560, and the eye position image 570.

[0070] The desktop device is equipped with a first voice unit 220, which enables the patient to have a real-time voice call with the doctor by establishing voice interaction with the remote end. This interactive method facilitates communication between the patient and the doctor, and the patient can describe his symptoms, ask questions, and get guidance and explanations from the doctor.

[0071] The first display unit 230 is electrically connected to the first voice unit 220, facing the patient, and providing information display and interactive interface. Through the first display unit 230, the patient can see the relevant information displayed during the voice interaction with the doctor, such as the instruction steps, explanation of the condition, etc. This helps the patient to better understand the entire detection process and diagnosis results.

[0072] The image acquisition unit 240 is located in the first display unit 230 and acquires image information of the patient's eye along the first direction F1. The image information includes an iris image 540, an ocular surface image 550, a fundus image 560, and an eye position image 570. Through high-quality image acquisition, doctors can obtain detailed eye structure information to accurately assess the patient's eye health.

[0073] Among them, the first control unit 250 is the core control component of the desktop end. The first control unit 250 performs identity authentication based on the collected iris image 540 and recognizes the identity of the user in real time. In addition, by analyzing the ocular surface image 550, the fundus image 560 and the eye position image 570, the first control unit 250 can also determine the user's health status, such as ocular surface inflammation, fundus vascular abnormalities, etc. Doctors can receive these image information at the remote end and make accurate diagnosis and evaluation.

[0074] Specifically, the fundus image 560 includes: a feature of the macula 420. The eye position image 570 includes: a feature of the pupil 410. The first control unit 250 marks a first tracking point 510 on the pupil 410 of the eye position image 570, marks a second tracking point 520 on the macula 420 of the fundus image 560, tracks the rotation of the patient's eyeball 430 according to the first tracking point 510, and controls the image acquisition unit 240 to automatically focus with the second tracking point 520 as the focus point.

[0075] The fundus image 560 includes the features of the macula 420. The macula 420 is one of the important parts of the eye, including the central area of ​​the retina 440, and its status and features can provide important information about the patient's visual health and eye diseases. Through the fundus image 560 collected by the device 100, the doctor can observe and analyze the changes and abnormalities in the macula 420 area to evaluate the patient's visual health.

[0076] The eye position image 570 refers to an image obtained by photographing or scanning the eye to obtain the eye position and the characteristics of the pupil 410. The pupil 410 is a black circular structure in the eye, and its size and shape may be affected by the rotation of the eyeball 430 and the contraction of the pupil 410. By collecting the eye position image 570 and analyzing the characteristics of the pupil 410, the device 100 can detect and track the rotation of the patient's eyeball 430.

[0077] Among them, in the first control unit 250 on the desktop, by analyzing the pupil 410 features of the eye position image 570, the rotation of the patient's eyeball 430 can be marked and tracked. Specifically, the control unit will mark the first tracking point 510 on the pupil 410 of the eye position image 570, and mark the second tracking point 520 on the macula 420 of the fundus image 560. By tracking the position change of the first tracking point 510, the device 100 can understand the rotation of the patient's eyeball 430 in real time.

[0078] According to the marked second tracking point 520, the device 100 can use it as a focus point to control the image acquisition unit 240 to perform automatic focusing. This ensures that when the fundus image 560 is acquired, the macula 420 area can maintain clear imaging, thereby improving the quality and usability of the image.

[0079] Specifically, the first tracking point 510 is the geometric center point of the pupil 410, and the second tracking point 520 is the geometric center point of the macula 420. The first control unit 250 locates the position of the pupil 410 in the image of the eyeball 430 according to the first tracking point 510 to track the rotation angle of the eyeball 430, and uses the second tracking point 520 as the focus point to control the image acquisition unit 240 to automatically focus.

[0080] The first tracking point 510 is the geometric center point of the pupil 410, and the second tracking point 520 is the geometric center point of the macula 420. By analyzing the eye position image 570 and the fundus image 560, the first control unit 250 can accurately locate and mark the two tracking points. The purpose of doing so is to provide accurate reference points for further tracking of the eyeball 430 and focus control.

[0081] Among them, by means of the positioning of the first tracking point 510, the first control unit 250 can track the rotation angle of the eyeball 430. By analyzing the position change of the first tracking point 510 in the image of the eyeball 430, the device 100 can monitor the movement of the patient's eyeball 430 in real time and calculate the rotation angle of the eyeball 430. This helps doctors evaluate the health of the eyeball 430 in terms of the range of motion, eye muscle function, etc.

[0082] The second tracking point 520 is the geometric center point of the macula 420 and is used as the focus point of the image acquisition unit 240. By using the second tracking point 520 as a reference, the device 100 can achieve automatic focusing to ensure that the macula 420 area remains clear when acquiring the fundus image 560. This helps doctors to observe and analyze the details of the macula 420 more accurately, thereby providing a more accurate eye health assessment.

[0083] Specifically, the desktop end also includes: a fill light unit 260. The fill light unit 260 includes an infrared light lamp 261 and a visible light lamp 262. The first screen display unit 230 includes: a first display screen 231. The extended plane where the side of the first display screen 231 facing the user is located is recorded as a first reference plane. When observed along the first direction F1, the first tracking point 510 is the projection of the geometric center of the pupil 410 of the patient when facing the first display screen 231 on the first reference plane, the projection of the midpoint of the infrared light lamp 261 on the first reference plane is recorded as a first marking point, and the projection of the visible light lamp 262 on the first reference plane is recorded as a second marking point. The first marking point is located above the first tracking point 510, and the second marking point is located on the same horizontal line as the first tracking point 510.

[0084] The fill light unit 260 includes an infrared light 261 and a visible light 262, which are used to provide appropriate lighting conditions to ensure the clarity and quality of the eye image. The combination of the infrared light 261 and the visible light 262 can provide different spectral ranges to meet different application requirements. The infrared light 261 can be used for tracking the pupil 410 and acquiring the eye position image 570 in eye imaging, while the visible light 262 can provide sufficient lighting to acquire the ocular surface image 550 and the fundus image 560.

[0085] The first display unit 230 includes a first display screen 231, which faces the patient. Through the first display screen 231, the patient can interact with the device 100 and perform eye imaging and examination according to the instructions. The extension plane where the first display screen 231 is located is defined as a first reference plane, observed in the first direction F1. On this reference plane, some key points are marked.

[0086] The first tracking point 510 is the projection of the geometric center of the pupil 410 on the first reference plane when the patient is looking directly at the first display screen 231. The projection of the midpoint of the infrared light 261 on the first reference plane is defined as the first marking point, and the projection of the visible light 262 on the first reference plane is defined as the second marking point. The first marking point is located above the first tracking point 510, and the second marking point is located on the same horizontal line as the first tracking point 510.

[0087] Through the above marking and positioning, the device 100 can provide more accurate and stable tracking and positioning functions of the eyeball 430. This helps to ensure the accuracy and consistency of the eye image acquisition and analysis process. At the same time, in the community health detection scenario, these functions can provide doctors and patients with a convenient and reliable eye imaging experience and improve the accuracy and efficiency of diagnosis.

[0088] Specifically, the first display screen 231 includes: a normal display unit 2311 and an under-screen display unit 2312. The normal display unit 2311 is configured in a part of the display area of ​​the first display screen 231. The under-screen display unit 2312 is configured in another part of the display area of ​​the first display screen 231.

[0089] The normal display unit 2311 is a part of the display area of ​​the first display screen 231, which is used to display information related to eye imaging and health detection. Through the normal display unit 2311, the doctor can view the patient's eye image, health status and diagnosis results. At the same time, the patient can also observe his or her own eye image on the normal display unit 2311, which increases the sense of participation and understanding of the examination process.

[0090] The under-screen display unit 2312 is another part of the display area of ​​the first display screen 231, which is located below the screen. It can be used to display auxiliary information, such as operation guides, notes or other information related to eye imaging and health detection. The position of the under-screen display unit 2312 makes it possible for such information to not interfere with or block the field of view when observing eye images, while still being easily viewable by doctors and patients.

[0091] Through the combination of the normal display unit 2311 and the under-screen display unit 2312, the first display screen 231 provides a comprehensive interface that can simultaneously display eye images and related information to support doctors and patients in eye examinations and health assessments. This design helps to improve the convenience of operation and the visibility of information, providing a better user experience and diagnostic effect for community health testing.

[0092] Specifically, the image acquisition unit 240 includes: an under-screen camera 241. The under-screen camera 241 is arranged on the side of the under-screen display unit 2312 away from the patient, and acquires an eyeball 430 image and a fundus image 560 along a first direction F1. The direction perpendicular to the first direction F1 is recorded as the second direction F2, and the plane perpendicular to the first reference plane and perpendicular to the second direction F2 is recorded as the second reference plane. Observing along the second direction F2, the first tracking point 510 is the projection of the geometric center of the pupil 410 when the patient looks straight at the first display screen 231 on the second reference plane, and the second tracking point 520 is the projection of the geometric center of the macula 420 when the patient looks straight at the first display screen 231 on the second reference plane. The first control unit 250 establishes an eyeball 430 model according to the eyeball 430 image, obtains the position of the center of the eyeball 430 in the second reference plane by edge detection method, and the projection of the midpoint of the under-screen camera 241 on the second reference plane is recorded as the third marking point, and the center of the eyeball 430 and the third marking point are located on the same horizontal line.

[0093] The under-screen camera 241 is located on the side of the under-screen display unit 2312 away from the patient, and collects the eyeball 430 image and the fundus image 560 along the first direction F1. Its position selection makes the camera not block or interfere with the patient, and can accurately capture the eyeball 430 image and the fundus image 560.

[0094] The second direction F2 is perpendicular to the first direction F1, and the second reference plane is perpendicular to the first reference plane and perpendicular to the second direction F2. By observing along the second direction F2, the positions of some key points in the image of the eyeball 430, such as the first tracking point 510 and the second tracking point 520, can be determined.

[0095] The first tracking point 510 is the projection of the geometric center of the pupil 410 on the second reference plane when the patient is looking straight at the first display screen 231, and the second tracking point 520 is the projection of the geometric center of the macula 420 on the second reference plane when the patient is looking straight at the first display screen 231. The positioning of these tracking points helps to determine the position and focus of the eyeball 430, so as to perform accurate image acquisition and diagnosis.

[0096] Among them, the first control unit 250 uses the image of the eyeball 430 to establish the eyeball 430 model, and determines the position of the center of the eyeball 430 on the second reference plane by edge detection. The projection of the midpoint of the under-screen camera 241 on the second reference plane is called the third marker point, and the center of the eyeball 430 and the third marker point are on the same horizontal line. The determination of these features and marker points helps to identify the position and posture of the eyeball 430 for accurate eye imaging and fundus examination.

[0097] Through the above features and controls, the device 100 can achieve accurate eyeball 430 tracking and eye imaging, helping doctors to perform community health tests on patients. This design can provide high-quality eye images and accurately measure the angle and focal length of the eyeball 430 rotation, thereby improving the accuracy and effect of the test. At the same time, for patients, this device can provide convenient operation and a comfortable examination experience.

[0098] Specifically, the desktop end further includes: a lens assembly 270. The lens assembly 270 is located on the side of the first display screen 231 away from the under-screen camera 241, and along the direction away from the display screen, the lens assembly 270 includes a linear polarizer 271, a first wave plate 272, a first lens 273, a second wave plate 274, an infrared anti-transmittance reflective polarizer 275, and a second lens 276 arranged in sequence. The linear polarizer 271 is in contact with the side of the first display screen 231 away from the under-screen camera 241. The first lens 273 is arranged at intervals on the side of the linear polarizer 271 away from the first display screen 231. The second lens 276 is arranged at intervals on the side of the first lens 273 away from the first display screen 231. The infrared anti-transmittance reflective polarizer 275 is used to reflect and pass polarized light of visible light, and pass infrared light. And the infrared anti-transmittance reflective polarizer 275 is located on the side of the second lens 276 close to the first display screen 231. The infrared light emitted by the infrared lamp 261 irradiates the pupil 410 and is reflected to the screen camera through the lens assembly 270. The visible light emitted by the visible light lamp 262 irradiates the eyeball 430 and is reflected to the screen camera through the lens assembly 270. The first wave plate 272 is arranged on the side of the linear polarizing plate 271 away from the under-screen camera 241, and the second wave plate 274 is arranged on the side of the first lens 273 away from the under-screen camera 241. Both the first wave plate 272 and the second wave plate 274 are 45° 1 / 4 wave plates.

[0099] The first lens 273 has a first surface formed on a side close to the display screen, and has a second surface formed on a side away from the display screen. The lens assembly 270 further includes a second wave plate 274. The second wave plate 274 is attached to the second surface, and the second wave plate 274 is a 45° 1 / 4 wave plate; the first surface is a convex surface, and the second surface is a plane. The circularly polarized light R is incident on the first lens 273 through the first surface, and the circularly polarized light R generates S polarized light S or P polarized light P when passing through the second wave plate 274.

[0100] The linear polarizer 271 is attached to the side of the first display screen 231 away from the under-screen camera 241. Its function is to adjust and control the polarization direction of the light passing through the lens assembly 270 so as to perform optical imaging.

[0101] The first lens 273 is disposed at a distance from the linear polarizer 271 on the side away from the first display screen 231, and the second lens 276 is disposed at a distance from the first lens 273 on the side away from the first display screen 231. These lenses constitute a part of the lens assembly 270, which is used to focus and adjust light to obtain a clear image.

[0102] The infrared anti-reflection reflective polarizer 275 is located on the side of the second lens 276 close to the first display screen 231. Its function is to reflect and transmit polarized light of visible light and transmit infrared light. This helps to control the path of light and filter out unnecessary light components to improve image quality and accuracy.

[0103] The infrared light emitted by the infrared light lamp 261 irradiates the pupil 410 and is reflected to the screen camera through the lens assembly 270. The visible light emitted by the visible light lamp 262 irradiates the eyeball 430 and is reflected to the screen camera through the lens assembly 270. Through the reflection and focusing of the lens assembly 270, the irradiation of the infrared light and the visible light can provide a clear image of the eyeball 430 and the fundus image 560.

[0104] By adjusting the polarization direction of light, focusing light, and filtering out unnecessary light components, it helps improve the quality and clarity of the image. The irradiation of infrared light and visible light can provide rich eye information, helping doctors to accurately track the eyeball 430 and examine the fundus. At the same time, the design of the lens assembly 270 enables it to work in conjunction with the under-screen camera 241 to provide patients with a comfortable and convenient examination experience.

[0105] Specifically, the remote end includes: a second screen display unit 310, a second voice unit 320, and a second control unit 330. The second screen display unit 310 receives and plays the eyeball 430 image and the fundus image 560. The second voice unit 320 establishes voice interaction with the desktop end. The second control unit 330 establishes signal connections with the second screen display unit 310, the second voice unit 320, and the desktop end respectively.

[0106] The second display unit 310 at the remote end receives and plays the eyeball 430 image and the fundus image 560 transmitted from the desktop end. In this way, the doctor can observe the patient's eye condition in real time and perform remote diagnosis and analysis.

[0107] The second voice unit 320 of the remote end establishes voice interaction with the desktop end, so that the doctor can communicate with the patient in real time. The doctor can provide guidance, explain the examination process or answer the patient's questions, thereby realizing remote medical services.

[0108] The second control unit 330 of the remote end establishes signal connections with the second display unit 310, the second voice unit 320 and the desktop end respectively. In this way, the doctor can control the display image, adjust the settings or perform other operations through the remote end to better assist in diagnosis and provide medical advice.

[0109] Through the above features, the remote end provides doctors with the function of real-time observation of patients' eye images and voice communication. Doctors can remotely view patients' eyeball 430 images and fundus images 560 for examination, analysis and diagnosis. At the same time, doctors can interact with patients by voice to provide accurate guidance and medical advice. This remote connection and interaction can help doctors and patients achieve remote eye health screening and diagnosis, and improve the convenience and efficiency of medical services.

[0110] Specifically, when the patient looks directly at the under-screen camera 241, the doctor calibrates the geometric center point of the macula 420 on the fundus image 560 as the second tracking point 520 through the second control unit 330. When the patient moves the eyeball 430, the first control unit 250 tracks the rotation of the eyeball 430 according to the first tracking point 510, and controls the under-screen camera 241 to automatically focus with the second tracking point 520 as the focus point. When the doctor wants to shoot other areas of the retina 440 other than the macula 420, the third tracking point 530 is calibrated on the fundus image 560 through the second control unit 330, and the second control unit 330 controls the under-screen camera 241 to automatically focus with the third tracking point 530 as the focus point.

[0111] When the patient looks directly at the under-screen camera 241, the doctor can mark the geometric center point of the macula 420 as the second tracking point 520 on the fundus image 560 through the second control unit 330. In this way, the doctor can accurately track and observe changes and abnormalities in the macula 420 area.

[0112] When the patient moves the eyeball 430, the first control unit 250 tracks the rotation of the eyeball 430 according to the first tracking point 510, and uses the second tracking point 520 as the focus point to control the under-screen camera 241 to automatically focus. This allows the doctor to maintain the clarity and focus accuracy of the image when the patient's eyeball 430 moves, so as to better observe the details and lesions of the eye.

[0113] When the doctor needs to shoot other retinal 440 areas other than the macula 420, the third tracking point 530 is calibrated on the fundus image 560 through the second control unit 330. The second control unit 330 uses the third tracking point 530 as the focus point to control the under-screen camera 241 to automatically focus. In this way, when the doctor needs to pay attention to other retinal 440 areas, the focus and clarity of the image can be ensured for a comprehensive fundus examination and diagnosis.

[0114] With the above features, doctors can accurately track the macula 420 area and eyeball 430 rotation while maintaining the focus accuracy of the image. This allows doctors to perform comprehensive eye screening and diagnosis in community health testing application scenarios. Through automatic focus and precise tracking, doctors can obtain clear fundus images 560, observe and analyze lesions and pathological characteristics in different areas, and provide accurate diagnosis and treatment recommendations for patients.

[0115] Specifically, when the patient moves the eyeball 430, and the doctor wants to take pictures of other areas of the retina 440 other than the macula 420, the line connecting the center of the eyeball 430 and the first tracking point 510 after the movement is used as the optical axis L1, the line connecting the third tracking point 530 and the first tracking point 510 is used as the visual axis L2, and the line connecting the center of the eyeball 430 and the center of the under-screen camera 241 is used as the reference axis L3 to reconstruct the eyeball 430 model. When the doctor marks the third tracking point 530 on the fundus image 560 along the first direction F1, the second control unit 330 locates the position of the third tracking point 530 on the retina 440 according to the relative position of the third tracking point and the second tracking point 520, and the second control unit 330 controls the under-screen camera 241 to automatically focus according to the position of the third tracking point 530 on the retina 440.

[0116] When the patient moves the eyeball 430 and the doctor needs to shoot other retinal 440 areas other than the macula 420, the eyeball 430 model is reconstructed by using the line between the center of the eyeball 430 and the first tracking point 510 after the movement as the optical axis L1, the line between the third tracking point 530 and the first tracking point 510 as the visual axis L2, and the line between the center of the eyeball 430 and the center of the under-screen camera 241 as the reference axis L3. Such a model provides reference axes and angles to accurately locate the positions of other retinal 440 areas.

[0117] According to the relative positions of the third tracking point 530 and the second tracking point 520 marked by the doctor on the fundus image 560, the second control unit 330 can determine the position of the third tracking point 530 on the retina 440 in the eyeball 430 model by observing along the second direction F2. This allows the doctor to accurately locate and select the retina 440 area of ​​interest.

[0118] Among them, by controlling the automatic focus of the under-screen camera 241, the second control unit 330 ensures the clarity and focus accuracy of the image when shooting other areas of the retina 440 according to the position of the third tracking point 530 on the retina 440. This allows doctors to obtain high-quality images, which is convenient for observing and analyzing the selected area of ​​the retina 440.

[0119] In summary, these structures and effects enable doctors to more accurately locate and photograph the retina 440 area of ​​interest in the application scenario of community health detection. By reconstructing the eyeball 430 model, locating the third tracking point 530 on the retina 440, and the autofocus function, doctors can obtain clear and accurately focused images for more accurate eye screening and diagnosis, and provide better medical services for patients.

[0120] In this way, by collecting the patient's iris image 540, real-time iris recognition and identity authentication are performed to ensure that the patient is always the same person, thereby improving the effectiveness of health screening. By collecting the patient's ocular surface image 550, fundus image 560 and eye position image 570, the patient's health status is analyzed from multiple modalities to ensure the diversity and accuracy of health screening. By using the pupil 410 as the first tracking point 510 and the macula 420 as the second tracking point 520, it is ensured that when collecting the fundus image 560, the macula 420 area can maintain clear imaging, thereby improving the quality and usability of the image, and especially improving the judgment of diabetic symptoms.

[0121] The above-mentioned embodiments only express several embodiments of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A multimodal recognition device for health screening based on eye imaging and eye tracking, comprising a desktop terminal for patients and a remote terminal for doctors to interact with the desktop terminal, characterized in that: The desktop terminal includes: A first voice unit, establishing voice interaction with the remote end; A first screen display unit, electrically connected to the first voice unit and facing the patient; An image acquisition unit is provided in the first display unit and acquires image information and eye movement information of the patient's eyes along a first direction, wherein the image information includes: an iris image, an ocular surface image, an ocular fundus image, and an eye position image; a first control unit, for identifying a user in real time according to the iris image, and determining a health status of the user according to the ocular surface image, the fundus image, and the eye position image; The fundus image includes: macular features; The eye position image includes: pupil features; The first control unit marks a first tracking point on the pupil of the eye position image, marks a second tracking point on the macula of the fundus image, tracks the rotation of the patient's eyeball according to the first tracking point, and controls the image acquisition unit to automatically focus with the second tracking point as the focus point; The image acquisition unit comprises: An under-screen camera is disposed on a side of the under-screen display unit facing away from the patient and collects an eyeball image and a fundus image along a first direction; The remote end includes: A second screen display unit receives and plays the eyeball image and fundus image; A second voice unit establishes voice interaction with the desktop terminal; A second control unit establishes signal connections with the second display unit, the second voice unit, and the desktop end respectively; When the patient looks directly at the camera under the screen, the doctor calibrates the geometric center point of the macula on the fundus image as the second tracking point through the second control unit; When the patient moves his eyeball, the first control unit tracks the movement of the eyeball according to the first tracking point, and controls the automatic focusing of the under-screen camera with the second tracking point as the focus point; When the doctor wants to take pictures of other retinal areas besides the macula, the third tracking point is calibrated on the fundus image through the second control unit, and the second control unit controls the automatic focus of the under-screen camera with the third tracking point as the focus point.

2. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 1, characterized in that: The first tracking point is the geometric center point of the pupil, and the second tracking point is the geometric center point of the macula. The first control unit locates the position of the pupil in the eyeball image according to the first tracking point to track the angle of eyeball rotation, and uses the second tracking point as a focus point to control the image acquisition unit to automatically focus.

3. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 2, characterized in that: The desktop terminal also includes: A fill light unit, including an infrared light and a visible light; The first screen display unit includes: a first display screen, wherein an extended plane on which a side of the first display screen facing the user is located is recorded as a first reference plane, Observing along the first direction, the first tracking point is the projection of the geometric center of the patient's pupil on the first reference plane when the patient is looking directly at the first display screen, the projection of the midpoint of the infrared lamp on the first reference plane is marked as the first marking point, and the projection of the visible light lamp on the first reference plane is marked as the second marking point. The first marking point is located above the first tracking point, and the second marking point is located on the same horizontal line as the first tracking point.

4. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 3, characterized in that: The first display screen comprises: A normal display unit, configured in a portion of the display area of ​​the first display screen; The under-screen display unit is configured in another display area of ​​the first display screen.

5. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 4, characterized in that: The direction perpendicular to the first direction is recorded as the second direction, and the surface perpendicular to the first reference plane and perpendicular to the second direction is recorded as the second reference plane; Observing along the second direction, the first tracking point is the projection of the geometric center of the pupil on the second reference plane when the patient is looking straight at the first display screen, and the second tracking point is the projection of the geometric center of the macula on the second reference plane when the patient is looking straight at the first display screen; The first control unit establishes an eyeball model based on the eyeball image, obtains the position of the eyeball center in the second reference plane through edge detection method, and the projection of the midpoint of the under-screen camera on the second reference plane is marked as a third marking point, and the eyeball center and the third marking point are located on the same horizontal line.

6. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 5, characterized in that: The desktop terminal also includes: A lens assembly is located on a side of the first display screen away from the under-screen camera, and along a direction away from the display screen, the lens assembly includes: A linear polarizer, attached to a side of the first display screen facing away from the under-screen camera; A first lens is spaced apart and arranged on a side of the linear polarizer away from the first display screen; a second lens, spaced apart and arranged on a side of the first lens away from the first display screen; an infrared anti-reflection reflective polarizer, used for reflecting and passing polarized light of visible light and passing infrared light, and located on a side of the second lens close to the first display screen; The infrared light emitted by the infrared lamp irradiates the pupil and is reflected to the screen display camera through the lens assembly; the visible light emitted by the visible light lamp irradiates the eyeball and is reflected to the screen display camera through the lens assembly.

7. The multimodal recognition device for health screening based on eye imaging and eye tracking according to claim 6, characterized in that: When the patient moves his eye and the doctor wants to take pictures of other areas of the retina besides the macula, The eyeball model is reconstructed using the line between the eyeball center and the first tracking point after movement as the optical axis, the line between the third tracking point and the first tracking point as the visual axis, and the line between the eyeball center and the center of the under-screen camera as the reference axis; When observing along the first direction, when the doctor calibrates the third tracking point on the fundus image, observing along the second direction, the second control unit locates the position of the third tracking point on the retina in the eyeball model based on the relative position of the third tracking point and the second tracking point, and the second control unit controls the automatic focus of the under-screen camera based on the position of the third tracking point on the retina.

Images