Closed-eye pupil monitoring system and method

Abstract

The present invention discloses a closed-eye pupil monitoring system and method, aiming to address the problems of low accuracy and inconvenient operation in existing closed-eye pupil monitoring technologies. The system includes a near-infrared luminous contact lens and a wearable eye mask-like device. The contact lens is equipped with a ring-shaped near-infrared LED and an ASIC chip, which are wirelessly powered by radio frequency and emit near-infrared light that is transmitted through the pupil. The eye mask-like device captures the light transmitted by the pupil with a micro CMOS camera and combines it with an FPGA main control system for real-time image processing to accurately monitor pupil size. This invention has the advantages of integrated data acquisition and processing, simple algorithm processing, flexible application scenarios, and low processing latency.

Description

Technical Field

[0001] The present invention relates to the technical field of pupil monitoring, and in particular to a closed-eye pupil monitoring system and method. Background Art

[0002] The pupil, a key physiological characteristic of the human eye, has a diameter that is closely linked to the state of the nervous system. In the medical field, pupil size is a key indicator in the diagnosis and monitoring of comatose patients and those with neurological disorders. Accurately and promptly observing pupil changes can help doctors determine the severity of their condition.

[0003] In clinical practice, traditional pupil measurement methods mainly include direct observation and pupillometer measurement. Direct observation is greatly affected by subjective factors and it is difficult to accurately measure pupil size and its changes. Although pupillometer measurement is relatively accurate, the equipment is usually complex and requires the patient to maintain a specific posture to cooperate with the measurement. In some special scenarios (such as when the patient is in a coma and the eyelids are closed), it is impossible to accurately obtain data. In addition, it is difficult to monitor the pupil for a long time and continuously in a natural state, which limits the clinical application of the dynamic change characteristics of the pupil.

[0004] Patent document (publication number CN115359548A) proposes a handheld intelligent pupil detection device and detection method. The device uses a camera to capture eye images, and then accurately measures pupil size through preprocessing, contour detection, ellipse fitting, and other steps, and then displays the results intuitively on the screen. However, this method requires the user to have their eyes open to extract the original human eye image, and then perform subsequent processing such as pupil image extraction, feature extraction, and pupil size detection. This requirement limits the application of this method in closed-eye scenarios. In closed-eye scenarios, due to the occlusion of the eyelids, a clear image of the pupil cannot be directly obtained, and subsequent pupil size calculation is impossible.

[0005] The patent document (publication number WO2017216800A1) proposes and discloses a closed eyelid pupil measurement device and method, which allows a near-infrared light source to be incident from the temple or the back of the head, and uses a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) detector to collect images of the closed eyelid pupil, and then calculate the pupil size. However, due to the absorption and scattering of human biological tissue, the infrared light source is incident from the temple and then transmitted through the eyelid. After at least two transmissions, its intensity will be greatly reduced or suppressed. In addition, when the eyelids are closed, the light source emitted from the eye epidermis is a pupil scattered light with non-sharp edges, so good imaging quality cannot be guaranteed. A large amount of algorithm fitting is required subsequently, resulting in inaccurate measurement results. Summary of the Invention

[0006] Based on this, the present invention aims to address the issues of low accuracy, inconvenient operation, and difficulty in achieving long-term continuous monitoring in a natural state with existing pupil monitoring technologies. Specifically, to address the issue of low eyelid transmitted light intensity and the resulting non-sharp edge light spot during real-time dynamic pupil size monitoring with closed eyes, a closed-eye pupil monitoring system and method are provided. These systems enable accurate and convenient pupil size detection, meeting the requirements of natural-eye monitoring to a certain extent.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The closed-eye pupil monitoring system consists of two main parts: a near-infrared luminous contact lens and a wearable eye mask-like image acquisition and processing device.

[0009] The wearable eye mask-like image acquisition and processing device is internally composed of a radio frequency power transmission module, an image acquisition module, a control and processing module, and a power supply module.

[0010] The near-infrared luminous contact lens consists of six near-infrared LEDs, a loop antenna, and an application-specific integrated circuit (ASIC) chip with a power management unit.

[0011] The near-infrared LED is connected to the ASIC chip via an antenna and is used to emit near-infrared light. The luminous intensity of the LED can be controlled by the ASIC chip. The near-infrared LED has an opaque flexible backplane, and the near-infrared light cannot be directly captured by the micro CMOS camera.

[0012] The near-infrared LED is located 1 mm to 2.5 mm from the edge of the near-infrared light-emitting contact lens, corresponding to the sclera of the eyeball.

[0013] The ASIC chip is responsible for controlling the luminous intensity and time of the near-infrared LED, and the chip area is approximately 1.0 mm×1.0 mm.

[0014] The RF power transmission module is designed specifically for near-infrared contact lenses and is implemented on a flexible printed circuit board. The RF power transmission board consists of an RF power transmitter and an external loop antenna with a matching network, and is connected to the FPGA main control system.

[0015] The antenna of the radio frequency power transmission module (as a primary coil) and the loop antenna on the near-infrared luminous contact lens (as a secondary coil) constitute a resonant induction wireless energy transmission system.

[0016] The image acquisition module and the control and processing module are composed of a micro CMOS camera, a field programmable logic gate array (FPGA) main control system, and a wireless transmission module.

[0017] The wireless transmission module is used to receive a terminal start signal and transmit it to the FPGA main control system. The FPGA main control system starts sending commands to allow the radio frequency power transmission module to transmit energy at a specified power to light up the near-infrared LED in the near-infrared light-emitting contact lens.

[0018] The FPGA main control system hardware includes a high-speed image input interface MIPI to receive image data and store it in a data buffer; a control interface I 2 C transmits configuration parameters to the image acquisition module, and the system control interface UART and USB communicate with the outside world; the storage interface covers the external large-capacity storage device DDR3 and the internal Flash storage unit, which is used to store system programs, processing results and temporary data; the power management module can convert voltage, monitor protection and dynamically manage power supply.

[0019] The image acquisition module is controlled by the FPGA main control system to control the micro CMOS camera to capture near-infrared light emitted by the pupil. The near-infrared light has the characteristics of the pupil size. The near-infrared light comes from the near-infrared LED in the near-infrared luminous contact lens and is emitted from the pupil after being reflected by the eyeball.

[0020] After the control and processing module receives the characteristic grayscale video stream containing pupil size captured by the micro CMOS camera, the FPGA main control enters the image processing mode, starts the image processing algorithm, extracts the pupil size, and then the wireless transmission module transmits the real-time pupil size data back to the terminal.

[0021] The FPGA image processing algorithm comprises the following specific steps:

[0022] Step 1: Median filtering is performed on the video stream image to complete denoising. The median filter uses a 3×3 matrix slider. The denoised image is binarized using the local threshold method. The threshold segmentation is based on a 15×15 window matrix. The local mean is calculated within the window and the threshold is dynamically determined with a confidence level of 0.9 to convert the grayscale image into a binary image.

[0023] Step 2: Perform morphological processing on the binary image to further remove noise in the image. First, perform a closing operation and then an opening operation, both using a 3×3 window matrix.

[0024] Step 3: For the noisy image part that cannot be filtered out by the previous steps, an improved connected domain algorithm is used. According to the characteristics of this device, the largest connected domain must be the pupil area. After the largest connected domain is detected, the number of its pixels is counted.

[0025] The improved connected domain algorithm is an algorithm adapted to FPGA pipeline operation. Traditional connected domain detection requires two scans of an image to complete. Different from the ordinary connected domain detection algorithm, the improved connected domain algorithm combines two FIFOs to cache two lines of images, and achieves labeling in a single scan by sharing information between lines, thereby reducing labeling conflicts. Then, the inter-frame blanking time is used to merge adjacent connected domain labels to obtain the maximum connected domain.

[0026] Step 4: After obtaining the maximum connected domain and its number of pixels, approximate it as a circle. Calculate the pupil diameter in pixels.

[0027] Step 5: Refer to the pre-tested calibration dataset to map the pixel size of the pupil diameter to the actual physical size.

[0028] The wearable eye mask image acquisition and processing device also has a power module inside, which is used to provide power to the micro CMOS camera, radio frequency power transmission module, FPGA main control system, and wireless transmission module.

[0029] The present invention proposes a real-time closed-eye pupil monitoring system and method by using near-infrared luminous contact lenses in combination with a wearable eye mask-like image acquisition and processing device. The beneficial effects are as follows:

[0030] 1) Through the collaboration of a wearable eye mask image acquisition and processing device and a near-infrared luminous contact lens, data acquisition, processing, and data visualization are integrated;

[0031] 2) Using near-infrared luminescent contact lenses, we can effectively enhance closed-eye pupil features, simplifying the difficulty of post-processing algorithms from a hardware perspective;

[0032] 3) The system independently realizes in-situ processing of collected data without relying on the computing power of the terminal device, and the application scenarios are more flexible and independent.

[0033] 4) The data processing equipment uses FPGA as the main control, making full use of the FPGA's parallel computing capabilities to achieve low-latency real-time processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the near-infrared luminescent contact lens.

[0035] Figure 2 Schematic diagram of the wearable eye mask image acquisition and processing device.

[0036] Figure 3 Schematic diagram of wearing near-infrared luminescent contact lenses.

[0037] Figure 4This is a flow chart of closed-eye pupil monitoring described in Example 1.

[0038] Figure 5 This is the flow chart of the FPGA closed-eye pupil monitoring algorithm.

[0039] Figure 6 This is the result of FPGA closed pupil processing.

[0040] Figure 7 The pupil diameter calculation result graph is displayed on the terminal. DETAILED DESCRIPTION

[0041] The present invention is further described below with reference to the accompanying drawings and Example 1.

[0042] Example 1:

[0043] The overall composition of the device ( Figures 1 and 2 )

[0044] like Figure 1 As shown, the near-infrared luminescent contact lens has an ASIC chip 1 , six near-infrared LED lamps 2 , 3 , 4 , 5 , 6 , 7 and a secondary coil 8 .

[0045] like Figure 2 As shown, the wearable eye mask-like image acquisition and processing device includes two miniature CMOS cameras 9 and 15, two loop antennas 10 and 16, two power modules 11 and 17, two strap buckles 12 and 18, an FPGA main controller 13, a wireless transmission module 14, and two RF power transmitters 19 and 20. The miniature CMOS cameras 9 and 15 also have fill lights 21 and 22 on their upper sides, respectively.

[0046] How to wear near-infrared luminous contact lenses Figure 1 The near-infrared luminescent contact lens shown is Figure 2 After the wearable eye mask image acquisition and processing device is installed, turn on the power, initialize the startup device, and the terminal can connect to the wireless transmission module. Figure 2 The FPGA master control system of the wearable eye mask-like image acquisition and processing device first activates the micro CMOS cameras 9 and 15 and the fill lights 21 and 22. At this point, the terminal receives real-time eye images from the image acquisition module. Using this real-time image, the device performs alignment checks and adjusts the wearer's position to center the eye image.

[0047] Next, take the right eye as an example to explain in detail.

[0048] Furthermore, when a pupil monitoring command is issued at the terminal, the FPGA main control 13 switches its working mode to image processing mode. At the same time, the loop antenna 16 on the wearable eye mask-like image acquisition and processing device is connected to a power source, generating an alternating magnetic field around it. The secondary coil 8 on the near-infrared light-emitting contact lens is in the alternating magnetic field generated by the loop antenna 16 (generated by the first coil). According to the law of electromagnetic induction, the magnetic flux passing through the secondary coil 8 changes, thereby inducing an electromotive force in the secondary coil 8. In addition, the secondary coil 8 and the loop antenna 16 have the same resonant frequency, so the secondary coil 8 on the near-infrared light-emitting contact lens will resonate with the loop antenna 16 on the wearable eye mask-like image acquisition and processing device, forming a pair of resonant dipoles. The two generate resonant coupling, and the load impedance of the system reaches a maximum, thereby achieving efficient energy transfer.

[0049] Furthermore, the secondary coil 8 senses radio frequency energy, which cannot directly power circuit components. Therefore, the induced radio frequency energy first enters the ASIC chip 1 of the near-infrared light-emitting contact lens. The ASIC chip 1 converts the alternating current of the induction coil into direct current. The rectified current is used by the six near-infrared LEDs.

[0050] Furthermore, based on the intensity of the near-infrared light emitted from the pupil collected by the micro CMOS camera 15, the FPGA main control 13 will change the power output of the RF power transmission module to control the luminous power of the near-infrared light-emitting contact lens, so that the near-infrared light emitted from the closed pupil can be collected.

[0051] Further, the FPGA master 13 starts to process the closed pupil video stream data and runs the image processing algorithm to obtain the following Figure 6 The result is then used to calculate the closed pupil diameter, and the real-time video stream and pupil diameter are transmitted to the terminal through the wireless transmission module. Figure 7 After one acquisition is completed, the FPGA main control system will enter low-power mode and wait for the next pupil monitoring command to be triggered according to the acquisition frequency set by the terminal, thus achieving long-term monitoring.

[0052] Obviously, the above embodiments of the present invention are merely examples for more clearly illustrating the present invention and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A closed-eye pupil monitoring system, characterized in that: include: It consists of two parts: near-infrared luminous contact lenses and wearable eye mask-like image acquisition and processing device; The near-infrared light-emitting contact lens includes: six near-infrared LEDs, a loop antenna, and an application-specific integrated circuit (ASIC) chip; The wearable eye mask-like image acquisition and processing device includes: a radio frequency power transmission module, an image acquisition module, a control and processing module, a wireless transmission module, and a power supply module; The image acquisition module is integrated into a wearable eye mask-like device and includes a fill light and a micro CMOS camera for collecting near-infrared light signals transmitted by the eyelid near the eye and is connected to the data processing module via a data cable; The control and processing module adopts a field programmable gate array (FPGA) main control system to receive image data from the image acquisition module, analyze and process the pupil image through an image processing algorithm, and calculate the pupil size and pupil change rate; The wireless transmission module is used to transmit real-time pupil monitoring results to the terminal and receive signals sent by the terminal; The power supply module is used to provide power to the micro CMOS camera, radio frequency power transmission module, FPGA main control system, and wireless transmission module.

2. The closed-eye pupil monitoring system according to claim 1, characterized in that: The emission wavelength range of the near-infrared LED on the near-infrared luminous contact lens is 630nm to 1000nm, and the luminous intensity is continuously adjustable in the range of 1mW to 10mW. The six near-infrared LEDs are distributed at 1mm to 2.5mm from the edge of the near-infrared luminous contact lens, precisely corresponding to the sclera part of the eyeball to achieve effective irradiation of the eyeball. The angle distribution is 30°, 90°, 150°, 210°, 270°, and 330°. There is uniform eyelid transmitted light within a deviation of ±10°, which meets the collection requirements.

3. The closed-eye pupil monitoring system according to claim 1, wherein: The ASIC chip on the near-infrared light-emitting contact lens is integrated with a power management module and an LED driving module for controlling the operation of the near-infrared LED.

4. The closed-eye pupil monitoring system according to claim 1, wherein: The RF power transmission module is designed for near-infrared luminous contact lenses and consists of a RF power transmitter and an external loop antenna with a matching network. The antenna (primary coil) of the RF power transmission module and the loop antenna (secondary coil) on the near-infrared luminous contact lenses constitute a resonant induction wireless energy transmission system.

5. A closed-eye pupil monitoring method, characterized in that: The following steps are involved: System startup steps: The wireless transmission module receives the start signal from the terminal. The FPGA main control system analyzes and verifies the signal. After confirming its validity and integrity, it sends precise control commands to the RF power transmission module. The RF power transmission module transmits energy at a specified power according to the received control command, and the power is always within a safe range. It starts and lights up the near-infrared LED in the near-infrared light-emitting contact lens. During this process, the energy transmission is monitored and fed back in real time to ensure the stability and safety of energy transmission. The FPGA master control system adjusts the aperture, exposure time, gain parameters of the micro CMOS camera and the output of the RF power transmission module based on the real-time light conditions of the user's eyes to ensure the system achieves optimal image acquisition status; After the micro CMOS camera is activated, it continuously and stably captures the near-infrared light that is reflected by the eyeball, emitted from the pupil, and passes through the eyelid; In the data processing step, the FPGA main control system executes an image processing algorithm, which includes: Step 1: Median filtering is performed on the video stream image to complete denoising. The median filter uses a 3×3 matrix slider. The denoised image is binarized using the local threshold method. The threshold segmentation is based on a 15×15 window matrix. The local mean is calculated within the window and the threshold is dynamically determined with a confidence level of 0.9 to convert the grayscale image into a binary image. Step 2: Perform morphological processing on the binary image to further remove noise in the image. First, perform a closing operation and then an opening operation, both using a 3×3 matrix for sampling; Step 3: For the noisy image parts that cannot be filtered out by the previous steps, an improved connected domain algorithm is used. According to the characteristics of this device, the largest connected domain must be the pupil area. After the largest connected domain is detected, the number of pixels in it is counted. The improved connected domain algorithm is an algorithm adapted for FPGA pipeline operation. Different from the traditional connected domain detection algorithm, the improved connected domain algorithm combines two first-in-first-out memories (FIFOs) to cache two lines of images, shares information between lines, completes labeling in a single scan, reduces labeling conflicts, and uses the inter-frame blanking time to merge adjacent connected domain labels, eliminate temporary labeling conflicts, and ultimately determine the largest connected domain. Step 4: After obtaining the maximum connected domain and its number of pixels, approximate it as a circle. Calculate the pixel size of the pupil diameter; Step 5: Map the pixel size of the pupil diameter to the actual physical size; Transmission and display steps: After obtaining the actual physical size of the mapped pupil, the FPGA main control system outputs data to the wireless transmission module, and the terminal receiving module sends the data and displays it.

6. The closed-eye pupil monitoring method according to claim 5, characterized in that: After one acquisition is completed, the FPGA main control system will enter low-power mode and wait for the next pupil monitoring command to be triggered according to the acquisition frequency set by the terminal.

Images