Robot-assisted tracking alignment method and device for ophthalmic optical coherence tomography

Through the robot-assisted tracking and alignment method, combined with deep learning image recognition and hand-eye system calibration, pupil tracking and alignment and three-dimensional optical coherence tomography scan of individuals in any posture are realized, solving the problems of large equipment and complex operation in the prior art. , expands the application scenarios of optical coherence tomography and improves diagnostic capabilities.

CN118948203AActive Publication Date: 2024-11-15CHENGDU MUGUANG MEDICAL TECHNOLOGY CO LTD

Patent Information

Application Number
CN202410935848.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-15
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

The existing ophthalmic optical coherence tomography system is difficult to use in emergency and routine care environments due to its large equipment and complex operation, and requires professional operation, which limits its application scenarios and diagnostic capabilities.

Method used

The robot-assisted tracking and alignment method is adopted, and camera calibration and hand-eye system calibration is used, combined with deep learning image recognition algorithms, pupil tracking and alignment of individuals in any posture is realized, and a three-dimensional optical coherence tomography scan is performed to obtain high-quality ophthalmic images.

Benefits of technology

It realizes efficient pupil tracking and alignment for individuals in any posture within a large range, reduces the requirements of imaging workspace and operator proficiency and patient coordination, expands the applicable scenarios of optical coherence tomography, and improves ophthalmic diagnostic capabilities. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118948203A_ABST
    Figure CN118948203A_ABST
Patent Text Reader

Abstract

The invention provides a robot-assisted tracking alignment method and device for ophthalmic optical coherence tomography, and relates to the technical field of robot control and optical coherence tomography, and the method comprises the steps: calibrating and setting important parameters of a system; synchronously acquiring camera images and correcting the images; image recognition based on a deep learning algorithm; calculating a pupil three-dimensional position and a gaze direction based on an image recognition result; calculating a pupil alignment target pose and a movement track; robot servo motion control; the invention relates to optical coherence tomography scanning and ophthalmic three-dimensional volume image registration. According to the method provided by the invention, efficient pupil tracking alignment can be carried out on an individual freely standing in any posture in a relatively large range, and three-dimensional optical coherence tomography scanning is carried out after alignment is completed, so that a high-quality ophthalmic optical coherence tomography image is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot control technology and optical coherence tomography, and in particular to a robot-assisted tracking and alignment method and device for ophthalmic optical coherence tomography. Background Art

[0002] Optical coherence tomography (OCT) can achieve powerful non-invasive, high-resolution three-dimensional imaging of biological tissues and materials, and has been widely used in biomedicine and other fields. OCT plays an important role in determining the diagnostic criteria and promoting treatment decisions for various common eye diseases such as age-related macular degeneration, diabetic retinopathy, glaucoma and corneal dysfunction.

[0003] However, clinical optical coherence tomography systems designed for ophthalmic diagnostic purposes are usually large tabletop instruments that are usually fixed in an ophthalmology clinic or a dedicated imaging room in a large ophthalmology center. In addition, they require mechanical head stabilization devices such as chin rests or forehead supports for head positioning, eye alignment and motion suppression, and require skilled professionals to operate and align. They also require the cooperation of patients, who must be able to sit upright and use the chin rest as instructed to focus their eyes on a fixed target. Therefore, emergency and routine care environments cannot readily use optical coherence tomography systems for ophthalmic medical diagnosis.

[0004] If the requirements for imaging workspace, operator proficiency, and patient cooperation can be lowered, the applicable scenarios of optical coherence tomography will be greatly expanded, and patients can even complete eye examinations independently without the presence of professionals, which is of great significance for expanding the application of optical coherence tomography and improving ophthalmic diagnostic capabilities. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies in the above-mentioned prior art and provide a robot-assisted tracking and alignment method and device for ophthalmic optical coherence tomography, which can efficiently track and align the pupils of individuals standing freely in any posture within a large range, and perform three-dimensional optical coherence tomography scanning after the alignment is completed to obtain high-quality ophthalmic optical coherence tomography images.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a robot-assisted tracking and alignment method for ophthalmic optical coherence tomography, the method comprising:

[0008] Camera calibration and hand-eye system calibration, robot payload and tool center setting;

[0009] Synchronously collecting facial images with multiple near-infrared cameras, and performing image correction on the images based on camera calibration results;

[0010] For the corrected image, the face, pupil and corneal reflection point of the annular infrared point light source array are detected based on the deep learning image recognition algorithm, and the pupil center pixel coordinates and the pixel coordinates of the center of the fitted circle of all corneal reflection points are obtained;

[0011] The three-dimensional position of the pupil is calculated based on the binocular disparity obtained from the pupil center pixel coordinates, the external parameters obtained from the camera calibration, and the triangulation principle, and the gaze direction is calculated based on the pupil center pixel coordinates and the center pixel coordinates of all corneal reflection points and the pupil center-corneal reflection theory;

[0012] Based on the pupil three-dimensional position and gaze direction, robot coordinate transformation is performed in combination with the hand-eye calibration result to obtain the pupil alignment target posture, and a smooth motion trajectory is calculated in combination with the current state and safety range limit;

[0013] Perform robot servo motion control based on the motion trajectory, and repeat the steps of synchronously collecting the face image to the robot servo motion control until pupil alignment is completed;

[0014] When pupil alignment is completed, optical coherence tomography scanning is started to obtain ophthalmic three-dimensional volumetric images and perform image registration.

[0015] Optionally, in a feasible manner provided by the present invention, camera calibration and hand-eye system calibration, robot load and tool center setting include:

[0016] For the cameras on both sides, refer to Zhang's calibration method, take multiple groups of different photos of the checkerboard calibration plate placed in different positions and postures, and calculate the camera internal and external parameters according to the theoretical model. The internal parameters include focal length, principal point coordinates, distortion coefficient, etc., and the external parameters include the relative position and posture between the cameras.

[0017] For the intermediate camera embedded in the optical coherence tomography optical path, the chessboard calibration is also used to obtain its internal parameters and the conversion relationship between pixel coordinates and physical coordinates at a fixed working distance;

[0018] For the hand-eye system, the optimal imaging position of the optical coherence tomography scan is used as the tool center point. The position of the tool center relative to the center of the robot end effector is determined according to the four-point method. The position of the camera relative to the center of the robot end effector is solved according to the traditional mathematical model of hand-eye calibration. Based on the above relationship, the relative position of the camera and the tool center can be easily obtained. All the above relative position relationships include translation and rotation transformation relationships.

[0019] Optionally, in a feasible manner provided by the present invention, multi-eye near-infrared camera face images are synchronously collected, and the face images are image corrected based on camera calibration results, including:

[0020] Based on the internal and external parameters of the binocular camera calibration results, the images collected by the cameras on both sides are subjected to distortion correction and stereo correction;

[0021] The image captured by the intermediate camera is distorted based on the internal parameters of the monocular camera calibration result.

[0022] Optionally, in a feasible method provided by the present invention, for the corrected image, the face, pupil and corneal reflection point detection of the annular infrared point light source array are performed based on a deep learning image recognition algorithm, and the pupil center pixel coordinates and the pixel coordinates of the fitting center of all corneal reflection points are obtained, including:

[0023] Collect face images covering different individual samples in advance, and construct an image dataset with face, eye, pupil and corneal reflection point as detection targets;

[0024] Build a deep learning neural network for image segmentation and train the network model on the dataset;

[0025] Use the image detection algorithm trained by the model to detect and identify pupil and corneal reflection points in the corrected image;

[0026] In the pupil segmentation area obtained by image recognition, an adaptive gray value threshold segmentation method is used to further segment the dark pupil and iris, an ellipse is fitted to the edge of the dark pupil, and the centroid of the fitted ellipse is taken as the pixel coordinate of the pupil center in the image;

[0027] The centroid coordinates of each reflection point segmentation area are calculated in the corneal reflection point segmentation result obtained by image recognition, and these coordinates are fitted with an ellipse and the centroid of the fitted ellipse is taken as the pixel coordinate of the reflection center of the annular infrared point light source in the image.

[0028] Optionally, in a feasible method provided by the present invention, the three-dimensional position of the pupil is calculated based on the binocular disparity obtained from the pupil center pixel coordinates and the external parameters obtained by the camera calibration and the triangulation principle, and the gaze direction is calculated based on the pupil center pixel coordinates and the center pixel coordinates of the fitted circle of all corneal reflection points and the pupil center-corneal reflection theory, including:

[0029] Calculate the disparity based on the pupil center pixel coordinates in the images collected from the cameras on both sides;

[0030] Based on the relative positions and postures between cameras represented by parallax and camera extrinsics, the coordinates of the pupil center in three-dimensional space are calculated using the triangulation principle;

[0031] Based on the coordinates of the pupil center and the corneal reflection center of the annular light source in the image captured by the intermediate camera, the yaw angle and pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the principle of polar coordinate system transformation.

[0032] Optionally, in a feasible manner provided by the present invention, based on the pupil three-dimensional position and gaze direction, robot coordinate transformation is performed in combination with the hand-eye calibration result to obtain the pupil alignment target posture, and a smooth motion trajectory is calculated in combination with the current state and safety range limit, including:

[0033] Determine the position and orientation of the center of the target tool in the current camera coordinate system based on the three-dimensional position of the pupil and the gaze direction;

[0034] Based on the relative position relationship between the robot end effector, tool center, and camera in the hand-eye calibration results, the target pose of the end effector in the world coordinate system is calculated;

[0035] Read the current position and speed of the robot's end effector, and set the maximum boundary conditions of position, speed, acceleration, etc.;

[0036] Based on the target posture of the robot end effector in the world coordinate system, the current posture and speed of the end effector and the maximum boundary value condition, an online trajectory generation algorithm is used to calculate the motion trajectory required for the robot to complete pupil alignment.

[0037] Optionally, in a feasible manner provided by the present invention, robot servo motion control is performed based on the trajectory, and steps from image acquisition to motion control are repeated until pupil alignment is completed, including:

[0038] Based on the motion trajectory, the robot's online servo function is used to continuously send down the servo points in the trajectory with a fixed servo cycle;

[0039] During the movement, the collected images and pupil positioning results are continuously refreshed. If the target posture changes, a new trajectory is switched to achieve real-time tracking of the pupil until the pupil alignment is completed.

[0040] Optionally, in a feasible manner provided by the present invention, when pupil alignment is completed, optical coherence tomography scanning is started to obtain an ophthalmic three-dimensional volume image, including:

[0041] Compare the robot's current state with the target state and pupil positioning results to confirm whether pupil alignment is complete;

[0042] If pupil alignment is confirmed, the optical coherence tomography system is activated to perform cross-sectional scanning of the fundus or anterior segment of the eye, and the tool center position is further fine-tuned based on the imaging result feedback to obtain a better imaging position and angle;

[0043] After the adjustment is completed, three-dimensional scanning imaging is performed and image registration is performed to obtain high-quality ophthalmic optical coherence tomography three-dimensional images.

[0044] In a second aspect, the present invention provides an autonomous pupil tracking and alignment robot device for optical coherence tomography, comprising:

[0045] A robot 3D vision system with the ability to perceive 3D target positions;

[0046] A 3D optical coherence tomography system that can perform 3D volumetric imaging of the fundus and anterior segment of the eye;

[0047] A mounted scanning pod that integrates the optical coherence tomography sample arm optical path and the robotic 3D vision system;

[0048] A six-axis collaborative robot with six degrees of freedom and its electronic control box;

[0049] Computer equipment and storage media for executing control programs.

[0050] Specifically, a robot 3D vision system with 3D target position perception capability includes:

[0051] A near-infrared camera group and an annular infrared point light source array, wherein the binocular cameras are placed on both sides of the scanning probe objective lens, the middle camera is embedded in the optical path of the optical coherence tomography system, shares the same objective lens with the optical coherence tomography system and is placed behind a dichroic mirror for light splitting, and an annular infrared point light source array for assisting in determining the gaze direction is placed around the objective lens;

[0052] Specifically, a three-dimensional optical coherence tomography system capable of performing three-dimensional volume imaging of the fundus and anterior segment of the eye includes:

[0053] Optical coherence tomography light source, reference arm optical path and sample arm optical path, wherein the light source and reference arm optical path are fixed on the optical laboratory table, and the sample arm optical path is fixed in the mounted scanning pod. The main optical components in the pod are, in the order of incident light, the objective lens, dichroic mirror, lens group, dual-axis scanning galvanometer, and fiber collimator, and finally connected to the laboratory table optical path through optical fiber;

[0054] Specifically, the mounted scanning pod integrates the optical coherence tomography sample arm optical path and the robot 3D vision system, fixes the camera and light source in the robot 3D vision system, and fixes the optical path component of the sample arm of the optical coherence tomography system, and the whole is mounted on the end effector of the collaborative robot, which can be controlled by the robot to move and rotate;

[0055] Specifically, for a six-axis collaborative robot with six degrees of freedom and its electric control box, the robot base is fixed to the experimental table and connected to the electric control box, the robot end effector is mounted with a scanning pod, and the position of the end effector relative to the base is determined by the six rotation axis joint angles between the two;

[0056] Specifically, the computer device and storage medium used to execute the control program include:

[0057] A memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography as described in the first aspect is implemented;

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, executes the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A schematic diagram of the main appearance of a robot-assisted tracking and alignment device for ophthalmic optical coherence tomography provided by an embodiment of the present invention; in the figure: 1-six-axis collaborative robot; 11-first joint of the robot; 12-second joint of the robot; 13-third joint of the robot; 14-fourth joint of the robot; 15-fifth joint of the robot; 16-sixth joint of the robot; 17-end effector of the robot; 2-mounted scanning pod; 21-objective lens; 22-annular infrared point light source array; 23-shortwave dichroic mirror; 24-binocular near-infrared camera; 25-embedded near-infrared camera; 26-dual-axis scanning galvanometer;

[0061] Figure 2 A schematic flow chart of a robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the process of pupil and corneal reflection recognition and gaze direction estimation in the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention;

[0063] Figure 4A schematic diagram of the hardware composition of a robot-assisted tracking and alignment device for ophthalmic optical coherence tomography provided by an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of a software program module for a robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the drawings in the present invention are only for the purpose of illustration and description and are not used to limit the scope of protection of the present invention. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowcharts used in the present invention illustrate the operations implemented according to some embodiments of the present invention. In addition, under the guidance of the content of the present invention, those skilled in the art may add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0066] In addition, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0067] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0068] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.

[0069] Reference Figure 1, shows a schematic diagram of the main structure of the robot-assisted tracking and alignment device for ophthalmic optical coherence tomography provided by an embodiment of the present invention, which is composed of a six-axis collaborative robot 1 loaded with an integrated optical coherence tomography sample arm optical path and a mounted scanning pod 2 of a robot three-dimensional vision system. The six-axis collaborative robot 1 has six rotation joints, namely, a first robot joint 11, a second robot joint 12, a third robot joint 13, a fourth robot joint 14, a fifth robot joint 15, and a sixth robot joint 16, with six degrees of freedom, which jointly determine the position and posture of the robot end effector 17. The mounted scanning pod 2 includes: an objective lens 21 shared by the optical coherence tomography system and the embedded near-infrared camera; an annular infrared point light source array 22 distributed around the objective lens for illumination and calculation of gaze angle; a short-wave dichroic mirror 23 for separating the optical coherence tomography optical path and the embedded near-infrared camera optical path; a binocular near-infrared camera 24 in the robot vision system that provides a larger field of view and depth information; an embedded near-infrared camera 25 in the robot vision system that provides more accurate two-dimensional information on the working plane; and a dual-axis scanning galvanometer 26 in the optical coherence tomography system. The mounted scanning pod 2 can be loaded on the robot end effector 17, so that the six-axis collaborative robot 1 can control the mounted scanning pod 2 to move in six degrees of freedom (including three translational degrees of freedom and three rotational degrees of freedom) within a larger range to achieve pupil alignment and scanning of subjects in any posture.

[0070] Reference Figure 2 , shows a schematic flow chart of a robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention comprises:

[0071] Step S100, camera calibration and hand-eye system calibration, robot load and tool center setting;

[0072] Specifically, for the cameras on both sides, refer to Zhang's calibration method, use a black and white checkerboard calibration plate with a grid spacing of 15 mm, take multiple sets of different photos of the checkerboard calibration plate placed in different positions and postures, and use the MATLAB stereo camera calibration application to repeatedly calculate and screen photos to reduce the reprojection error to within 0.1 pixels, and obtain the stereo camera intrinsic parameters (including focal length, principal point coordinates, distortion coefficient, etc.) and extrinsic parameters (including the relative position and posture between cameras);

[0073] Specifically, for the intermediate camera embedded in the optical coherence tomography optical path, a black and white chessboard with a grid spacing of 3 mm was used. Multiple sets of different photos were taken, and its internal parameters were calculated using the MATLAB single camera calibration application. Then, multiple sets of chessboard photos were taken again at a fixed working distance and the obtained internal parameters were used for image correction. The measurements were repeated multiple times to obtain the conversion relationship between pixel coordinates and physical coordinates.

[0074] Specifically, for the hand-eye system, the best imaging position of the optical coherence tomography scan is used as the tool center point, and the position of the tool center relative to the center of the robot end effector is determined according to the four-point method, that is, the robot is allowed to make the tool center coincide with a fixed reference point in four different postures, and the relative position of the tool center is solved by the end effector positions of these four points; the position of the camera relative to the center of the robot end effector is solved according to the traditional mathematical model of hand-eye calibration. In the embodiment of the present invention, the AX=XB type model is selected, wherein A is a homogeneous transformation corresponding to two different end effector postures, which is obtained by the robot reading the joint position information and solving it through inverse kinematics, B is a homogeneous transformation of the camera posture corresponding to the two end effector postures, which is calculated by the positioning results of the camera for the fixed target twice, and X is a homogeneous transformation between the end effector and the camera to be solved. The relative position of the end effector and the camera can be obtained by solving the translation vector and the rotation matrix respectively through the separation method; based on the relative position relationship between the end effector and the tool center and the camera, the relative position of the camera and the tool center is easily obtained, and the hand-eye calibration is completed. All the above relative position relationships include translation and rotation transformation relationships.

[0075] Step S200, multiple near-infrared cameras synchronously collect facial images. In this embodiment, the image collection adopts a soft trigger mode, and a computer program sends a soft trigger command to the three cameras simultaneously;

[0076] Step S300, performing image correction on the captured image based on the camera calibration result;

[0077] Specifically, image correction includes performing distortion correction and stereo correction on the images captured by the cameras on both sides based on the internal and external parameters of the binocular camera calibration results, and performing distortion correction on the images captured by the middle camera based on the internal parameters of the monocular camera calibration results. In this embodiment, the image correction module of OpenCV is used to import the calibration parameters obtained in S100 and process the image data converted into a matrix format to achieve the above image correction.

[0078] Step S400, for the corrected image, based on the deep learning image recognition algorithm, detect the corneal reflection points of the face, pupil and annular infrared point light source array, and obtain the pupil center pixel coordinates and the pixel coordinates of the fitting center of all corneal reflection points;

[0079] Specifically, face images covering different individual samples, different distances and different angles are collected in advance, and LabelMe annotation is used to construct an image dataset with face, eye, pupil and corneal reflection point as detection targets;

[0080] Specifically, in this embodiment, a YOLO V5s deep learning neural network for image segmentation is built based on PyTorch, and the network model is trained in advance in the above-mentioned image dataset, the trained model is deployed in a computer program, and based on this algorithm, pupil and corneal reflection points are detected and identified in the corrected image, such as Figure 3 (a)

[0081] Further, refer to Figure 3 (b)-(c), in the pupil segmentation area obtained by image recognition, the dark pupil and iris are further segmented using an adaptive gray value threshold segmentation method, and the dark pupil edge is fitted with an ellipse using the OpenCV ellipse fitting method and the centroid of the fitted ellipse is taken as the pixel coordinate of the pupil center in the image;

[0082] Further, refer to Figure 3 (e)-(f), in the corneal reflection point segmentation result obtained by image recognition, the centroid coordinates of each reflection point segmentation area are calculated, an ellipse is fitted to these coordinates, and the centroid of the fitted ellipse is taken as the pixel coordinate of the reflection center of the annular infrared point light source in the image.

[0083] Step S500, calculating the three-dimensional position of the pupil based on the binocular parallax of the recognition result, camera calibration external parameters and triangulation principle, and calculating the gaze direction based on the pupil center and the reflection point fitting circle center coordinates;

[0084] Specifically, the disparity is calculated based on the pupil center pixel coordinates in the images collected by the cameras on both sides, and the coordinates of the pupil center in three-dimensional space are calculated using the triangulation principle based on the camera's extrinsic parameters (relative position and posture between cameras) and the disparity;

[0085] Specifically, refer to Figure 3 (d) Based on the coordinates of the pupil center and the corneal reflection center of the annular light source in the image captured by the intermediate camera, the yaw angle and pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the principle of polar coordinate system transformation.

[0086] Step S600, performing robot coordinate transformation based on the pupil position and gaze direction hand-eye calibration results to obtain pupil alignment target posture, and calculating a smooth motion trajectory in combination with the current state and safety range limit;

[0087] Specifically, based on the pupil position and gaze direction, as well as the system hand-eye calibration result, the position and posture of the target tool center in the base coordinate system are determined, and the conversion and calculation relationship can be given by the following formula: B ξ dT = B ξ T T ξ CC ξ P ,in B ξ dT Indicates the position of the target tool center relative to the base coordinate system, B ξ T Indicates the position of the current tool center relative to the base coordinate system. T ξ C Indicates the current camera position relative to the tool center. C ξ P Represents the position of the current pupil relative to the current camera, that is, the pupil position and gaze direction in the camera coordinate system;

[0088] Specifically, the current position and speed of the robot are read. In this embodiment, the end effector information is converted into the tool center information based on the tool center calibration result. Therefore, the actual maximum boundary value conditions are set as the position, speed, acceleration, etc. of the tool center. The position boundary is in the space that the robot can reach. The maximum translation speed is 100 mm per second, the maximum translation acceleration is 150 mm per square second, the maximum rotation speed is 2 radians per second, and the maximum rotation acceleration is 5 radians per square second.

[0089] Furthermore, based on the current state, target state and boundary conditions of the robot, an online trajectory generation algorithm is used to calculate the motion trajectory required for the robot to complete pupil alignment. In the online trajectory generation type adopted in this embodiment, the target state speed, acceleration and jerk are all zero, the boundary conditions have constraints on the maximum acceleration value and no constraints on continuity, the trajectory is calculated in Cartesian space, and the decision tree determines the velocity profile and motion time on each degree of freedom according to various conditions, and then starts from the initial state and calculates a series of positions and velocities of corresponding time with the servo time as a single-step time increment until the end state, and the servo time is 6 milliseconds.

[0090] Step S700, performing robot servo motion control based on the trajectory, repeating the steps of image acquisition to motion control until pupil alignment is completed;

[0091] Specifically, in this embodiment, the points in the trajectory are sent down in sequence with the servo time as a period, and images are repeatedly collected at a higher frame rate to calculate the target position during the movement. When the trajectory is updated, it is determined whether the end point of the trajectory coincides with the end point of the world coordinate system with higher precision. If so, the current trajectory is continued to reduce the resource consumption and efficiency impact of trajectory calculation and switching. If not, the trajectory is recalculated and switched to achieve real-time tracking of the moving pupil.

[0092] Step S800, when pupil alignment is completed, optical coherence tomography scanning is started to obtain an ophthalmic three-dimensional volume image;

[0093] Specifically, if it is confirmed that the pupil alignment is completed, the optical coherence tomography system is started, and a cross-sectional scan of the fundus or the anterior segment of the eye is performed. A certain feature of the imaging object is identified in the cross-sectional image, and its distance from the optimal imaging position is calculated. The tool center position is then further fine-tuned based on the distance feedback. After the adjustment is completed, three-dimensional scanning imaging is performed, and image registration is performed based on the similarity metric to eliminate motion artifacts and obtain high-quality ophthalmic optical coherence tomography three-dimensional images.

[0094] Reference Figure 4 , shows a schematic diagram of the hardware composition of a robot-assisted tracking and alignment device for ophthalmic optical coherence tomography provided by an embodiment of the present invention. The hardware composition of the eyeball pupil three-dimensional tracking and alignment device for ophthalmic robot optical coherence tomography provided by an embodiment of the present invention includes:

[0095] A robot 3D vision system with 3D target position perception capability, consisting of a near-infrared camera group and an annular infrared point light source array, wherein the binocular cameras are placed on both sides of the scanning probe objective lens, the middle camera is embedded in the optical path of the optical coherence tomography system, shares the same objective lens with the optical coherence tomography system and is placed behind a dichroic mirror for light splitting, and an annular infrared point light source array for assisting in determining the gaze direction is placed around the objective lens;

[0096] A three-dimensional optical coherence tomography system capable of three-dimensional volume imaging of the fundus and anterior segment of the eye, wherein the light source and reference arm optical path are fixed on an optical laboratory table, and the sample arm optical path is fixed in a mounted scanning pod. The main optical components in the pod are, in the order in which the incident light passes, an objective lens, a dichroic mirror, a lens group, a dual-axis scanning galvanometer, and a fiber collimator, and are finally connected to the optical path of the laboratory table through an optical fiber.

[0097] The mounted scanning pod integrates the optical coherence tomography sample arm optical path and the robot 3D vision system. The pod fixes the camera and light source in the robot 3D vision system, and fixes the optical path components of the sample arm of the optical coherence tomography system. The pod is mounted on the end effector of the collaborative robot as a whole, and can be controlled by the robot to move and rotate.

[0098] A six-axis collaborative robot with six degrees of freedom and its electric control box. The robot base is fixed on the experimental table and connected to the electric control box. The robot end effector is mounted with a scanning pod. The position of the end effector relative to the base is determined by the six rotation axis joint angles between the two.

[0099] Computer equipment and storage media used to execute control programs. The computer is connected to the robot's three-dimensional vision system, optical coherence tomography system and collaborative robot through network cables to achieve overall control of the system;

[0100] Specifically, in an embodiment of the present invention, in order to prevent mutual interference between the optical paths, the central wavelength of the camera working band is 850 nanometers, and the central wavelength of the optical coherence tomography band is 1060 nanometers. To ensure the coaxiality of the central camera and the optical coherence tomography, they share the same objective lens in the optical path. A short-wave pass dichroic mirror with a cut-off wavelength of 950 nanometers is installed behind the objective lens, allowing light emitted from the annular infrared point light source to enter the camera, but reflecting the optical coherence tomography beam from the scanning source laser. The optical coherence tomography beam is emitted from the optical fiber, collimated by the fiber collimator, deflected by the dual-axis galvanometer, and shaped by the 4F system. The reshaped beam is then reflected by the two-resolution mirror and shot into the pupil through the objective lens. The beam reflected from the pupil returns along the same path and re-enters the optical fiber.

[0101] Reference Figure 5 , shows a schematic diagram of the main modules of the software program of the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography provided by an embodiment of the present invention, and the software program modules of the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography include:

[0102] An image recognition module for detecting faces, eyes, pupils, etc. based on a deep learning method, for executing step S400;

[0103] A pupil 3D position and gaze direction calculation module based on robot 3D vision system calibration and image detection results, used to execute steps S300 and S500;

[0104] A robot visual servo control module that controls the six-axis robotic arm based on the three-dimensional positioning result so that the optical axis of the objective lens in the pod is aligned with the pupil at a specified distance and angle and tracks in real time, and is used to execute steps S600 and S700;

[0105] The optical coherence tomography system image acquisition module is used to execute step S800.

[0106] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in the present invention. In the several embodiments provided by the present invention, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0107] In addition, each functional unit in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0108] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A robot-assisted tracking and alignment method for ophthalmic optical coherence tomography, characterized in that: The method comprises: Camera calibration and hand-eye system calibration, robot payload and tool center setting; Synchronously collecting facial images with multiple near-infrared cameras, and performing image correction on the facial images based on camera calibration results; For the corrected image, the face, pupil and corneal reflection point of the annular infrared point light source array are detected based on the deep learning image recognition algorithm, and the pupil center pixel coordinates and the pixel coordinates of the center of the fitted circle of all corneal reflection points are obtained; The three-dimensional position of the pupil is calculated based on the binocular disparity obtained from the pupil center pixel coordinates, the external parameters obtained from the camera calibration, and the triangulation principle, and the gaze direction is calculated based on the pupil center pixel coordinates and the center pixel coordinates of all corneal reflection points and the pupil center-corneal reflection theory; Based on the pupil three-dimensional position and gaze direction, robot coordinate transformation is performed in combination with the hand-eye calibration result to obtain the pupil alignment target posture, and a smooth motion trajectory is calculated in combination with the current state and safety range limit; Perform robot servo motion control based on the motion trajectory, and repeat the steps of synchronously collecting the face image to the robot servo motion control until pupil alignment is completed; When pupil alignment is completed, optical coherence tomography scanning is started to obtain ophthalmic three-dimensional volumetric images and perform image registration.

2. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: The camera calibration and hand-eye system calibration, robot payload and tool center settings include: For the cameras on both sides, refer to Zhang's calibration method, take multiple groups of different photos of the checkerboard calibration plate placed in different positions and postures, and calculate the camera internal and external parameters according to the theoretical model. The internal parameters include focal length, principal point coordinates, distortion coefficient, etc., and the external parameters include the relative position and posture between the cameras. For the intermediate camera embedded in the optical coherence tomography optical path, the chessboard calibration is also used to obtain its internal parameters and the conversion relationship between pixel coordinates and physical coordinates at a fixed working distance; For the hand-eye system, the optimal imaging position of the optical coherence tomography scan is used as the tool center point. The position of the tool center relative to the center of the robot end effector is determined according to the four-point method. The position of the camera relative to the center of the robot end effector is solved according to the traditional mathematical model of hand-eye calibration. Based on the above relationship, the relative position of the camera and the tool center can be easily obtained. All the above relative position relationships include translation and rotation transformation relationships.

3. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: The multi-camera near-infrared camera synchronously collects facial images, and performs image correction on the facial images based on camera calibration results, including: Based on the internal and external parameters of the binocular camera calibration results, the images collected by the cameras on both sides are subjected to distortion correction and stereo correction; The image captured by the intermediate camera is distorted based on the internal parameters of the monocular camera calibration result.

4. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: For the corrected image, the face, pupil and corneal reflection point detection of the annular infrared point light source array are performed based on the deep learning image recognition algorithm, and the pupil center pixel coordinates and the pixel coordinates of the fitting center of all corneal reflection points are obtained, including: Collect face images covering different individual samples in advance, and construct an image dataset with face, eye, pupil and corneal reflection point as detection targets; Building a deep learning neural network for image segmentation and training the network model on the image dataset; Use the image detection algorithm trained by the model to detect and identify pupil and corneal reflection points in the corrected image; In the pupil segmentation area obtained by image recognition, an adaptive gray value threshold segmentation method is used to further segment the dark pupil and iris, an ellipse is fitted to the edge of the dark pupil, and the centroid of the fitted ellipse is taken as the pixel coordinate of the pupil center in the image; The centroid coordinates of each reflection point segmentation area are calculated in the corneal reflection point segmentation result obtained by image recognition, and these coordinates are fitted with an ellipse and the centroid of the fitted ellipse is taken as the pixel coordinate of the reflection center of the annular infrared point light source in the image.

5. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: The three-dimensional position of the pupil is calculated based on the binocular disparity obtained based on the pupil center pixel coordinates, the external parameters obtained by the camera calibration and the triangulation principle, and the gaze direction is calculated based on the pupil center pixel coordinates and the center pixel coordinates of all corneal reflection points and the pupil center-corneal reflection theory, including: Calculate the disparity based on the pupil center pixel coordinates in the images collected from the cameras on both sides; Based on the relative positions and postures between cameras represented by parallax and camera extrinsics, the coordinates of the pupil center in three-dimensional space are calculated using the triangulation principle; Based on the coordinates of the pupil center and the corneal reflection center of the annular light source in the image captured by the intermediate camera, the yaw angle and pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the principle of polar coordinate system transformation.

6. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: The method comprises: performing robot coordinate transformation based on the pupil three-dimensional position and gaze direction in combination with the hand-eye calibration result to obtain the pupil alignment target posture, and calculating the smooth motion trajectory in combination with the current state and the safety range limit. Determine the position and orientation of the center of the target tool in the current camera coordinate system based on the three-dimensional position of the pupil and the gaze direction; Based on the relative position relationship between the robot end effector, tool center, and camera in the hand-eye calibration results, the target pose of the end effector in the world coordinate system is calculated; Read the current position and speed of the robot's end effector, and set the maximum boundary conditions of position, speed, acceleration, etc.; Based on the target posture of the robot end effector in the world coordinate system, the current posture and speed of the end effector and the maximum boundary value condition, an online trajectory generation algorithm is used to calculate the motion trajectory required for the robot to complete pupil alignment.

7. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: The robot servo motion control is performed based on the motion trajectory, and the steps of synchronously collecting the face image to the robot servo motion control are repeated until the pupil alignment is completed, including: Based on the motion trajectory, the robot's online servo function is used to continuously send down the servo points in the trajectory with a fixed servo cycle; During the movement, the collected images and pupil positioning results are continuously refreshed. If the target posture changes, a new trajectory is switched to achieve real-time tracking of the pupil until the pupil alignment is completed.

8. The robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to claim 1, characterized in that: When the pupil alignment is completed, optical coherence tomography scanning is started to obtain an ophthalmic three-dimensional volume image and perform image registration, including: Compare the robot's current state with the target state and pupil positioning results to confirm whether pupil alignment is complete; If pupil alignment is confirmed, the optical coherence tomography system is activated to perform cross-sectional scanning of the fundus or anterior segment of the eye, and the tool center position is further fine-tuned based on the imaging result feedback to obtain a better imaging position and angle; After the adjustment is completed, three-dimensional scanning imaging is performed and image registration is performed to obtain high-quality ophthalmic optical coherence tomography three-dimensional images.

9. A robot-assisted tracking and alignment device for ophthalmic optical coherence tomography, characterized in that: include: The device adopts the robot-assisted tracking and alignment method for ophthalmic optical coherence tomography according to any one of claims 1 to 8 for robot-assisted tracking and alignment.

Citation Information

Patent Citations

  • Spatial self-positioning ophthalmic optical coherence tomography system

    CN112842252A

  • Pupil tracking device

    KR1020180078611A

  • Systems and methods for performing eye gaze tracking

    US20190156100A1

  • Systems and methods for imaging a target feature of a subject based on tracked positions of the subject and the target feature

    US20220039648A1

Cited By

  • Eye ground OCT three-dimensional image generation method and device

    CN121154078A

  • Systems, apparatuses, methods, and non-transitory computer-readable storage devices for corneal-reflection-based eye-tracking using multiple cameras

    WO2026138412A1