Robotically assisted tracking and alignment method and apparatus for optical coherence tomography of the eye

Abstract

The application provides a robot-assisted tracking and alignment method and device for ophthalmic optical coherence tomography, and relates to the fields of robot control technology and optical coherence tomography technology.The method comprises the following steps: system important parameter calibration and setting; camera image synchronous acquisition and image correction; image recognition based on a deep learning algorithm; calculation of a three-dimensional position and a gaze direction of a pupil based on an image recognition result; calculation of a target pose and a motion trajectory of the pupil alignment; robot servo motion control; optical coherence tomography scanning; and ophthalmic three-dimensional volume image registration.The proposed method can efficiently track and align the pupil of an individual standing freely in any posture within a large range, and can perform three-dimensional optical coherence tomography scanning after the alignment is completed, so that high-quality ophthalmic optical coherence tomography images are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot control technology and optical coherence tomography, in particular to a robot-assisted tracking and alignment method and device for ophthalmic optical coherence tomography. BACKGROUND

[0002] Optical coherence tomography (OCT) can achieve powerful non-invasive and high-resolution three-dimensional imaging of biological tissues and materials, and has been widely used in the field of biomedicine. Optical coherence tomography plays an important role in determining the diagnostic criteria and promoting treatment decisions for various common eye diseases such as age-related macular degeneration, diabetic retinopathy, glaucoma and corneal dysfunction.

[0003] However, the clinical optical coherence tomography systems designed for ophthalmic diagnostic purposes are usually large desktop instruments, which are usually fixed in a dedicated imaging room in an ophthalmic clinic or a large ophthalmic center. In addition, they also need mechanical head stabilization devices such as chin rests or forehead supports to position the head, align the eyeball and suppress the movement, and need skilled professionals to operate and align, and also need the cooperation of patients, who must be able to sit straight and use the chin rest as instructed, and direct their gaze to a fixed target. Therefore, the optical coherence tomography system cannot be used for ophthalmic medical diagnosis at any time in emergency and routine care environments.

[0004] If the imaging workspace and the requirements for operator proficiency and patient cooperation can be reduced, the application scenarios of optical coherence tomography will be greatly expanded, and patients can even complete ophthalmic examinations independently without the presence of professionals, which is of great significance for expanding the application of optical coherence tomography and improving the ability of ophthalmic diagnosis. SUMMARY

[0005] The purpose of the present application is to provide a robot-assisted tracking and alignment method and device for ophthalmic optical coherence tomography, which can efficiently track and align the pupils of individuals standing freely in any posture within a larger range, and perform three-dimensional optical coherence tomography scanning after alignment to obtain high-quality ophthalmic optical coherence tomography images, in view of the deficiencies in the prior art.

[0006] To achieve the above purpose, the technical solutions adopted by the present application are as follows:

[0007] In a first aspect, the present application provides a robot-assisted tracking and alignment method for ophthalmic optical coherence tomography, which comprises:

[0008] Camera calibration and hand-eye system calibration, robot payload and tool center setting;

[0009] Multi-view near-infrared camera synchronously captures face images, and performs image correction on the images based on camera calibration results;

[0010] For the corrected images, deep learning image recognition algorithm is used to detect the face, pupil and corneal reflection points of the annular infrared point light source array, and to obtain the pixel coordinates of the pupil center and the pixel coordinates of the center of the fitted circle of all corneal reflection points;

[0011] Based on the binocular disparity obtained from the pixel coordinates of the pupil center, the extrinsic parameters obtained from the camera calibration and the triangulation principle, the three-dimensional position of the pupil is calculated, and based on the pixel coordinates of the pupil center and the pixel coordinates of the center of the fitted circle of all corneal reflection points and the pupil center-corneal reflection theory, the gaze direction is calculated.

[0012] Based on the three-dimensional position of the pupil and the gaze direction, combined with the hand-eye calibration results, the robot coordinate conversion is performed to obtain the target pose of the pupil alignment, and the smooth motion trajectory is calculated based on the current state and the safety range limit.

[0013] Based on the motion trajectory, the robot servo motion control is performed, and the steps of synchronously capturing face images and robot servo motion control are repeated until the pupil alignment is completed.

[0014] When the pupil alignment is completed, optical coherence tomography scanning is started to obtain ophthalmic three-dimensional volume images, and image registration is performed.

[0015] Optionally, in one feasible manner provided by the application, the camera calibration and hand-eye system calibration, the robot load and tool center setting include:

[0016] For the two cameras, referring to Zhang's calibration method, multiple groups of different photos are taken for the checkerboard calibration board placed in different positions and attitudes, and the camera intrinsic parameters and extrinsic parameters are calculated according to the theoretical model, the intrinsic parameters include focal length, principal point coordinates, distortion coefficient, etc., and the extrinsic parameters include the relative position and attitude between the cameras.

[0017] For the intermediate camera embedded in the optical coherence tomography light path, the checkerboard calibration is also used to obtain its intrinsic parameters and the conversion relationship between the pixel coordinates and the physical coordinates at a fixed working distance.

[0018] For the hand-eye system, the optical coherence tomography scanning optimal imaging position is taken as the tool center point, the four-point method is used to determine the position of the tool center relative to the robot end effector center, the camera relative to the robot end effector center is solved according to the traditional mathematical model of hand-eye calibration, and based on the above relationship, the relative position of the camera and the tool center is easily obtained. All the above relative position relationships include translation and rotation conversion relationships.

[0019] Optionally, in a feasible manner provided by the present application, the multi-view near-infrared camera synchronously collects face images, and performs image correction on the face images based on a camera calibration result, including:

[0020] Based on the internal and external parameters of the binocular camera calibration result, the images collected by the two side cameras are corrected for distortion and stereoscopic correction;

[0021] Based on the internal parameter of the monocular camera calibration result, the image collected by the middle camera is corrected for distortion.

[0022] Optionally, in a feasible manner provided by the present application, for the corrected image, a deep learning image recognition algorithm is used to detect the face, pupil, and corneal reflection point of the annular infrared point light source array, and to obtain the pixel coordinates of the pupil center and the pixel coordinates of the center of the fitted circle of all corneal reflection points, including:

[0023] Face images covering different individual samples are collected in advance, and an image dataset with face, human eye, pupil, and corneal reflection point as detection targets is constructed;

[0024] A deep learning neural network for image segmentation is built, and a network model is trained on the dataset;

[0025] The image detection algorithm trained using the completed model is used to detect and identify the pupil and corneal reflection point in the corrected image;

[0026] An adaptive gray value threshold segmentation method is used in the pupil segmentation area obtained by image recognition to further segment the dark pupil and iris, an ellipse is fitted to the edge of the dark pupil, and the center of the fitted ellipse is taken as the pixel coordinates of the pupil center in the image;

[0027] In the corneal reflection point segmentation result obtained by image recognition, the center coordinates of each reflection point segmentation area are calculated, these coordinates are fitted with an ellipse, and the center of the fitted ellipse is taken as the pixel coordinates of the reflection center of the annular infrared point light source in the image.

[0028] Optionally, in a feasible manner provided by the present application, based on the binocular disparity obtained from the pixel coordinates of the pupil center and the external parameters obtained from the camera calibration and the principle of triangulation, the three-dimensional position of the pupil is calculated, and based on the pixel coordinates of the pupil center and the pixel coordinates of the center of the fitted circle of all corneal reflection points and the pupil center-corneal reflection theory, the gaze direction is calculated, including:

[0029] Based on the pixel coordinates of the pupil center in the images collected by the two side cameras, the disparity is calculated;

[0030] Based on the relative position and attitude between the cameras represented by the disparity and the camera external parameters, the coordinates of the pupil center in the three-dimensional space are calculated using the principle of triangulation.

[0031] Based on the pupil center and the annular light source corneal reflection center coordinates in the image collected by the intermediate camera, the yaw angle and the pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the polar coordinate system transformation principle.

[0032] Optionally, in a feasible manner provided by the application, based on the three-dimensional position of the pupil and the gaze direction, the robot coordinate conversion is performed in combination with the hand-eye calibration result to obtain the target pose of the pupil alignment target, and the smooth motion trajectory is calculated in combination with the current state and the safety range limit, including:

[0033] Based on the three-dimensional position of the pupil and the gaze direction, the pose of the target tool center in the current camera coordinate system is determined;

[0034] Based on the relative position relationship among the robot end effector, the tool center and the camera in the hand-eye calibration result, the target pose of the end effector in the world coordinate system is calculated;

[0035] The current pose and speed of the robot end effector are read, and the maximum boundary value conditions of position, speed, acceleration and the like are set;

[0036] Based on the target pose of the robot end effector in the world coordinate system, the current pose and speed of the end effector and the maximum boundary value conditions, the motion trajectory required for the robot to complete the pupil alignment is calculated by using an online trajectory generation algorithm.

[0037] Optionally, in a feasible manner provided by the application, based on the trajectory, the robot servo motion control is performed, and the image acquisition and the motion control steps are repeated until the pupil alignment is completed, including:

[0038] Based on the motion trajectory, the servo points in the trajectory are continuously issued by using the online servo function of the robot at a fixed servo period;

[0039] During the motion process, the image and the pupil positioning result are continuously refreshed, if the target pose is changed, a new trajectory is switched to realize real-time tracking of the pupil, until the pupil alignment is completed.

[0040] Optionally, in a feasible manner provided by the application, when the pupil alignment is completed, optical coherence tomography is started to obtain ophthalmic three-dimensional volume images, including:

[0041] The current state and the target state of the robot and the pupil positioning result are compared to confirm whether the pupil alignment is completed;

[0042] If it is confirmed that the pupil alignment is completed, the optical coherence tomography system is started to perform cross-section scanning on the fundus or the anterior segment of the eye, and the tool center pose is further fine-tuned according to the imaging result feedback to obtain a better imaging position and angle;

[0043] After adjustment, three-dimensional scanning imaging is performed, and image registration is performed to obtain high-quality optical coherence tomography three-dimensional images of the eye.

[0044] In a second aspect, the present application provides an autonomous pupil tracking and alignment robot device for optical coherence tomography imaging, comprising:

[0045] A robot three-dimensional vision system with three-dimensional target position sensing capability;

[0046] A three-dimensional optical coherence tomography imaging system capable of three-dimensional volume imaging of the fundus and anterior segment of the eye;

[0047] A mounted scanning pod integrating the optical coherence tomography sample arm optical path and the robot three-dimensional vision system;

[0048] A six-axis collaborative robot with six degrees of freedom and its electrical control box;

[0049] A computer device and storage medium for executing a control program.

[0050] Specifically, the robot three-dimensional vision system with three-dimensional target position sensing capability comprises:

[0051] A near-infrared camera group and a ring-shaped infrared point light source array, wherein the binocular camera is placed on both sides of the scanning probe objective lens, the middle camera is embedded in the optical coherence tomography system optical path, shares the same objective lens with the optical coherence tomography system, and is placed behind the dichroic mirror for light splitting, and the ring-shaped infrared point light source array for assisting in determining the gaze direction is placed around the objective lens;

[0052] Specifically, the three-dimensional optical coherence tomography imaging system capable of three-dimensional volume imaging of the fundus and anterior segment of the eye comprises:

[0053] An optical coherence tomography light source, a reference arm optical path and a sample arm optical path, wherein the light source and the reference arm optical path are fixed to an optical experiment table, the sample arm optical path is fixed in the mounted scanning pod, and the main optical components in the pod are arranged in order of incident light passing through them, i.e., objective lens, dichroic mirror, lens group, two-axis scanning galvanometer, and optical fiber collimator, and finally connected to the experiment table optical path through an optical fiber;

[0054] Specifically, the mounted scanning pod integrating the optical coherence tomography sample arm optical path and the robot three-dimensional vision system fixes the camera and light source in the robot three-dimensional vision system, fixes the sample arm optical path components of the optical coherence tomography system, and is mounted on the end effector of the collaborative robot as a whole, and can be moved and rotated by the robot control;

[0055] Specifically, for the six-axis collaborative robot with six degrees of freedom and its electric control box, the robot base is fixed to the experiment table and connected with the electric control box, the robot end effector is hung with a scanning pod, and the pose of the end effector relative to the base is determined by the joint angles of the six rotation axes between them.

[0056] Specifically, the computer device and storage medium for executing the control program comprise:

[0057] The memory stores a computer program executable by the processor, and the processor executes the computer program to realize the robot-assisted tracking alignment method for ophthalmic optical coherence tomography as described in the first aspect.

[0058] The computer-readable storage medium stores a computer program, which is run by the processor to execute the robot-assisted tracking alignment method for ophthalmic optical coherence tomography as described in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0060] Figure 1 The main appearance schematic diagram of the robot-assisted tracking alignment device for ophthalmic optical coherence tomography provided by the embodiment of the present application; in the figure: 1-six-axis collaborative robot; 11-robot first joint; 12-robot second joint; 13-robot third joint; 14-robot fourth joint; 15-robot fifth joint; 16-robot sixth joint; 17-robot end effector; 2-hung scanning pod; 21-objective lens; 22-ring infrared point light source array; 23-short-wave pass dichroic mirror; 24-binocular near-infrared camera; 25-embedded near-infrared camera; 26-biaxial scanning galvanometer;

[0061] Figure 2 The flowchart of the robot-assisted tracking alignment method for ophthalmic optical coherence tomography provided by the embodiment of the present application;

[0062] Figure 3 The process schematic diagram of pupil and corneal reflection identification and gaze direction estimation in the robot-assisted tracking alignment method for ophthalmic optical coherence tomography provided by the embodiment of the present application;

[0063] Figure 4The schematic diagram of the hardware composition of the robot-assisted tracking and positioning device for ophthalmic optical coherence tomography provided by the embodiment of the present application is shown in FIG. 1.

[0064] Figure 5 The schematic diagram of the software program module of the robot-assisted tracking and positioning method for ophthalmic optical coherence tomography provided by the embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0065] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowchart shows the operations implemented according to some embodiments of the present application. In addition, one or more other operations can be added to the flowchart or one or more operations can be removed from the flowchart by those skilled in the art under the guidance of the content of the present application.

[0066] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative labor are within the scope of protection of the present application.

[0067] Hereinafter, the terms "include", "have", and their conjugates used in various embodiments of the present application only mean to indicate that specific features, numbers, steps, operations, elements, components, or combinations thereof are present, and should not be understood as excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof in advance.

[0068] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized or overly formal meaning unless clearly defined in various embodiments of the present application.

[0069] Reference Figure 1, shows the main structure of the robot-assisted tracking and positioning device for ophthalmic optical coherence tomography provided by the embodiment of the application, which is composed of a six-axis collaborative robot 1 and a mounted scanning pod 2 integrated with an optical coherence tomography sample arm optical path and a robot three-dimensional vision system. The six-axis collaborative robot 1 has six rotary joints, i.e., a robot first joint 11, a robot second joint 12, a robot third joint 13, a robot fourth joint 14, a robot fifth joint 15 and a robot sixth joint 16, and has six degrees of freedom, which jointly determine the position and posture of a robot end effector 17. The mounted scanning pod 2 includes: an objective lens 21 shared by the optical coherence tomography system and the embedded near-infrared camera; an annular infrared point light source array 22 distributed around the objective lens for illumination and calculation of the gaze angle; a short-wave pass dichroic mirror 23 for separating the optical coherence tomography optical path and the embedded near-infrared camera optical path; a binocular near-infrared camera 24 in the robot vision system for providing a larger field of view and depth information; an embedded near-infrared camera 25 in the robot vision system for providing more accurate two-dimensional information on the working plane; and a two-axis scanning galvanometer 26 in the optical coherence tomography system. The mounted scanning pod 2 can be loaded on the robot end effector 17, so that the six-axis collaborative robot 1 can control the mounted scanning pod 2 to move in a larger range in six degrees of freedom (including three translational degrees of freedom and three rotational degrees of freedom) to realize pupil alignment and scanning on a subject in any posture.

[0070] Referring to Figure 2 , shows the flowchart of the robot-assisted tracking and positioning method for ophthalmic optical coherence tomography provided by the embodiment of the application. The robot-assisted tracking and positioning method for ophthalmic optical coherence tomography provided by the embodiment of the application includes the following steps.

[0071] In step S100, camera calibration and hand-eye system calibration, robot loading and tool center setting are performed.

[0072] Specifically, for the two side cameras, Zhang's calibration method is used, a black and white checkerboard calibration board with a grid point spacing of 15 mm is used, a plurality of different photos are taken for the checkerboard calibration board placed in different positions and postures, the stereo camera calibration application of MATLAB is used, the re-projection error is reduced to within 0.1 pixels through repeated calculation and selection of photos, and the intrinsic parameters (including focal length, principal point coordinates, distortion coefficient, etc.) and extrinsic parameters (including the relative position and posture between the cameras) of the stereo camera are obtained.

[0073] Specifically, for the intermediate camera embedded in the optical coherence tomography optical path, a black and white checkerboard with a grid point spacing of 3 mm is used, a plurality of different photos are taken, the intrinsic parameters of the intermediate camera are calculated by using the single camera calibration application of MATLAB, then a plurality of checkerboard photos are taken again at a fixed working distance and the obtained intrinsic parameters are used for image correction, and the conversion relationship between the pixel coordinates and the physical coordinates is obtained through repeated measurement.

[0074] Specifically, for the hand-eye system, the optimal imaging position of the optical coherence tomography is taken as the tool center point, the position of the tool center relative to the robot end effector center is determined according to the four-point method, that is, the robot is caused to coincide the tool center with a certain fixed reference point in four different postures, and the relative position of the tool center is solved through the end effector positions of the four points; the position of the camera relative to the robot end effector center is solved according to the traditional mathematical model of hand-eye calibration, and in the embodiment of the application, an AX=XB type model is selected, wherein A is the homogeneous transformation corresponding to two different end effector postures respectively, which is obtained by reading the joint position information of the robot and through inverse kinematics solution, B is the homogeneous transformation of the camera pose corresponding to two end effector postures respectively, which is calculated from the positioning results of the camera on the fixed target twice, X is the homogeneous transformation to be solved between the end effector and the camera, and the relative position of the end effector and the camera can be obtained by solving the translation vector and the rotation matrix respectively through the separation method; based on the relative position relationship between the end effector and the tool center and the camera, the relative position of the camera and the tool center is easily obtained, the hand-eye calibration is completed, and all the above relative position relationships include translation and rotation conversion relationships.

[0075] In step S200, the multi-view near-infrared camera synchronously collects face images, in the embodiment, the image collection adopts a soft trigger mode, and a computer program simultaneously sends a soft trigger command to the three cameras;

[0076] In step S300, the collected images are corrected based on the camera calibration results;

[0077] Specifically, the image correction includes distortion correction and stereoscopic correction of the images collected by the two side cameras based on the internal and external parameters of the binocular camera calibration results, and distortion correction of the image collected by the middle camera based on the internal parameter of the monocular camera calibration result, in the embodiment, the image correction module of OpenCV is used to import the calibration parameters obtained in S100 and process the image data converted into a matrix format to realize the above-mentioned image correction.

[0078] In step S400, for the corrected images, face, pupil and corneal reflection point detection of the annular infrared point light source array are performed based on a deep learning image recognition algorithm, and the pixel coordinates of the pupil center and the pixel coordinates of the center of the fitted circle of all corneal reflection points are obtained.

[0079] Specifically, face images covering different individual samples, different distances and different angles are collected in advance, and a LabelMe label is used to construct an image data set with face, eye, pupil and corneal reflection point as detection targets;

[0080] Specifically, in this embodiment, a YOLO V5s deep learning neural network for image segmentation is built based on PyTorch, and the network model is trained in advance in the above image data set. The trained model is deployed in a computer program, and the pupil and corneal reflection points in the corrected image are detected and identified based on this algorithm, as shown in Figure 3 (a);

[0081] Further, referring to Figure 3 (b)-(c), the adaptive gray value threshold segmentation method is used to further segment the dark pupil and iris in the pupil segmentation area obtained by image recognition, and the centroid of the fitted ellipse is taken as the pixel coordinates of the pupil center in the image by using the OpenCV ellipse fitting method for the dark pupil edge;

[0082] Further, referring to Figure 3 (e)-(f), the centroid coordinates of each reflection point segmentation area are calculated in the corneal reflection point segmentation result obtained by image recognition, and the centroid of the fitted ellipse is taken as the pixel coordinates of the center of the annular infrared point light source reflection in the image by performing ellipse fitting on these coordinates.

[0083] Step S500, based on the binocular disparity and camera calibration external parameters of the recognition result and the triangulation principle, the three-dimensional position of the pupil is calculated, and the gaze direction is calculated based on the pupil center and the reflection point fitting circle center coordinates;

[0084] Specifically, the disparity is calculated based on the pupil center pixel coordinates in the images collected by the two cameras respectively, and the coordinates of the pupil center in the three-dimensional space are calculated based on the external parameters of the camera (the relative position and attitude between the cameras) and the disparity by using the triangulation principle.

[0085] Specifically, referring to Figure 3 (d), based on the pupil center and the annular light source corneal reflection center coordinates in the image collected by the intermediate camera, the yaw angle and pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the polar coordinate system transformation principle.

[0086] Step S600, based on the pupil position and gaze direction hand-eye calibration result, the robot coordinate conversion is performed to obtain the pupil alignment target pose, and the smooth motion trajectory is calculated combined with the current state and safety range limit;

[0087] Specifically, based on the pupil position and gaze direction, and the system hand-eye calibration result, the pose of the target tool center in the base coordinate system is determined, and the conversion and calculation relationship can be given by the following formula: B ξ dT = B ξ T T ξ CC ξ P wherein B ξ dT represents the pose of the target tool center relative to the base coordinate system, B ξ T represents the pose of the current tool center relative to the base coordinate system, T ξ C represents the pose of the current camera relative to the tool center, C ξ P represents the pose of the current pupil relative to the current camera, i.e. the pupil position and gaze direction in the camera coordinate system;

[0088] Specifically, the current pose and speed of the robot are read, and in this embodiment, the end effector information is converted into tool center information based on the tool center calibration result, so the actual maximum boundary value conditions are set as the position, speed, acceleration, etc. of the tool center, the position boundary is in the space reachable by the robot, the maximum translational speed is 100 mm / s, the maximum translational acceleration is 150 mm / s2, the maximum rotational speed is 2 rad / s, and the maximum rotational acceleration is 5 rad / s2;

[0089] Further, based on the current state, target state and boundary conditions of the robot, an online trajectory generation algorithm is used to calculate the motion trajectory required for the robot to complete the pupil alignment. In the type of online trajectory generation used in this embodiment, the target state speed, acceleration and jerk are all zero, the boundary condition has a constraint on the maximum acceleration and no constraint on continuity, and the trajectory is calculated in Cartesian space. According to the conditions, the decision tree determines the speed profile and motion time in each degree of freedom, and then calculates a series of positions and speeds at corresponding times from the initial state with a servo time as a single time increment until the end state. The servo time is 6 ms.

[0090] Step S700, based on the trajectory, the robot servo motion control is performed, and the image acquisition and motion control steps are repeated until the pupil alignment is completed;

[0091] Specifically, in this embodiment, the points in the trajectory are issued in turn with a servo time as a period, and in the motion process, images are repeatedly acquired at a high frame rate to calculate the target position, and when updating the trajectory, it is judged whether the trajectory endpoint in the world coordinate system coincides in high precision. If it coincides, the current trajectory is continued to reduce the resource consumption and efficiency impact of trajectory calculation and switching, and if it does not coincide, the trajectory is recalculated and switched to realize real-time tracking of the moving pupil.

[0092] Step S800, when the pupil alignment is completed, optical coherence tomography scanning is started to obtain an ophthalmic three-dimensional volume image;

[0093] Specifically, if the pupil alignment is confirmed to be completed, the optical coherence tomography system is started, a cross-section scan is performed on the fundus or the anterior segment of the eye, a certain feature of the imaging object is identified in the cross-section image, and the distance of the feature from the optimal imaging position is calculated, and then the distance is fed back to further fine-tune the pose of the tool center, and after adjustment, a three-dimensional scan is performed, and image registration is performed according to the similarity measure to eliminate motion artifacts, thereby obtaining a high-quality ophthalmic optical coherence tomography three-dimensional image.

[0094] Referring to Figure 4 , a schematic diagram of the hardware composition of the robot-assisted tracking and alignment device for ophthalmic optical coherence tomography imaging provided by the embodiment of the application is shown, and the hardware composition of the eyeball pupil three-dimensional tracking and alignment device for ophthalmic robot optical coherence tomography imaging provided by the embodiment of the application comprises:

[0095] The robot three-dimensional vision system has a three-dimensional target position sensing capability, and is composed of a near-infrared camera group and a ring-shaped infrared point light source array, wherein the binocular camera is arranged on both sides of the scan probe objective lens, the middle camera is embedded in the optical path of the optical coherence tomography system, shares the same objective lens with the optical coherence tomography system, and is arranged behind a dichroic mirror for light splitting, and the ring-shaped infrared point light source array for assisting in determining the gaze direction is arranged around the objective lens.

[0096] The three-dimensional optical coherence tomography system can perform three-dimensional volume imaging on the fundus and the anterior segment of the eye, the light source and the reference arm optical path are fixed on the optical experiment table, the sample arm optical path is fixed in the mounted scan pod, and the main optical components in the pod are sequentially arranged in the order of the objective lens, the dichroic mirror, the lens group, the two-axis scanning galvanometer, and the optical fiber collimator, and finally connected with the experiment table optical path through the optical fiber.

[0097] The mounted scan pod integrates the optical coherence tomography sample arm optical path and the robot three-dimensional vision system, the pod fixes the camera and the light source in the robot three-dimensional vision system, simultaneously fixes the sample arm optical path assembly of the optical coherence tomography system, and is integrally mounted on the end effector of the collaborative robot, and can be moved and rotated by the robot.

[0098] The six-axis collaborative robot with six degrees of freedom and the electric control box thereof, the robot base is fixed on the experiment table and connected with the electric control box, the scan pod is mounted on the end effector of the robot, and the pose of the end effector relative to the base is determined by the joint angles of the six rotation axes therebetween.

[0099] The computer device and the storage medium for executing the control program, the computer is connected with the robot three-dimensional vision system, the optical coherence tomography system and the collaborative robot through network cables respectively, and the overall control of the system is achieved.

[0100] Specifically, in the embodiment of the present application, to prevent mutual interference between the light paths, the center wavelength of the camera working band is 850 nm, and the center wavelength of the optical coherence tomography band is 1060 nm. To ensure the coaxiality of the center camera and the optical coherence tomography, they share the same objective lens in the light path. A short-wave pass dichroic mirror with a cutoff wavelength of 950 nm is installed behind the objective lens, which allows the light emitted by the annular infrared point light source to enter the camera, but reflects the optical coherence tomography beam from the scanning source laser. The optical coherence tomography beam is emitted from the optical fiber, collimated by the optical fiber collimator, then deflected by the two-axis galvanometer, and shaped by the 4F system. The reshaped beam is then reflected by the dichroic mirror and enters the pupil through the objective lens. The beam reflected from the pupil returns along the same path and reenters the optical fiber.

[0101] Referring to Figure 5 , a schematic diagram of the main modules of the software program of the robot-assisted tracking alignment method for ophthalmic optical coherence tomography provided by the embodiment of the present application is shown, and the software program modules of the robot-assisted tracking alignment method for ophthalmic optical coherence tomography include:

[0102] An image recognition module based on deep learning method for detecting human face, human eye and pupil, etc., used for executing step S400;

[0103] A pupil three-dimensional position and gaze direction calculation module based on robot three-dimensional vision system calibration and image detection results, used for executing steps S300 and S500;

[0104] A robot vision servo control module based on three-dimensional positioning results for controlling the six-axis mechanical arm to align and track the objective lens optical axis in the pod carried by the six-axis mechanical arm with the pupil at a specified distance and angle in real time, used for executing steps S600 and S700;

[0105] An optical coherence tomography system image acquisition module, used for executing step S800.

[0106] Those skilled in the art can clearly understand the specific working process of the system and the device described above for the convenience and brevity of description, and the corresponding process in the method embodiment can be referred to, and the present application will not be repeated. In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and the actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed elements can be indirect coupling or communication connection through some communication interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0107] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. When the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A robot-assisted tracking and alignment method for optical coherence tomography of the eye, characterized in that The method comprises: Camera calibration and hand-eye system calibration, robot load and tool center setting; Multi-view near-infrared camera face image synchronous acquisition, and image correction of the face image based on the camera calibration result; For the corrected image, face, pupil and corneal reflection point detection of the annular infrared point light source array are performed based on a deep learning image recognition algorithm, and the pupil center pixel coordinates and all corneal reflection point fitting circle center pixel coordinates are obtained; Based on the binocular disparity obtained from the pupil center pixel coordinates, the external parameters obtained from the camera calibration and the triangulation principle, the three-dimensional position of the pupil is calculated, and based on the pupil center pixel coordinates and all corneal reflection point fitting circle center pixel coordinates and the pupil center-corneal reflection theory, the gaze direction is calculated; Based on the three-dimensional position of the pupil and the gaze direction, combined with the hand-eye calibration result, the robot coordinate conversion is performed to obtain the pupil alignment target pose, and the smooth motion trajectory is calculated in combination with the current state and the safety range limit; Based on the motion trajectory, robot servo motion control is performed, and the steps of face image synchronous acquisition and robot servo motion control are repeated until the pupil alignment is completed; When the pupil alignment is completed, optical coherence tomography scanning is started, ophthalmic three-dimensional volume images are obtained, and image registration is performed; The camera calibration and hand-eye system calibration, robot load and tool center setting comprise: For the two side cameras, referring to Zhang's calibration method, a plurality of different photos of the chessboard calibration plate placed in different positions and attitudes are taken, the camera internal and external parameters are calculated according to the theoretical model, the internal parameters include focal length, principal point coordinates and distortion coefficient, and the external parameters include the relative positions and attitudes of the cameras; For the middle camera embedded in the optical coherence tomography light path, the internal parameters and the conversion relationship between the pixel coordinates and the physical coordinates at the fixed working distance are obtained by using the chessboard calibration; For the hand-eye system, the optical coherence tomography scanning optimal imaging position is taken as the tool center point, the position of the tool center relative to the robot end effector center is determined according to the four-point method, the position of the camera relative to the robot end effector center is solved according to the traditional mathematical model of hand-eye calibration, and based on the above relationship, the relative position of the camera and the tool center is easily obtained. All the above relative position relationships include translation and rotation conversion relationships.

2. The robot-assisted tracking alignment method for optical coherence tomography of the eye according to claim 1, wherein, The multi-view near-infrared camera face image synchronous acquisition and the image correction of the face image based on the camera calibration result comprise: Based on the internal and external parameters of the binocular camera calibration result, the distortion correction and stereo correction of the images collected by the two side cameras are performed; Based on the internal parameters of the monocular camera calibration result, the distortion correction of the image collected by the middle camera is performed.

3. The robot-assisted tracking alignment method for optical coherence tomography of the eye according to claim 1, wherein, The face, pupil and corneal reflection point detection of the annular infrared point light source array based on the deep learning image recognition algorithm, and the pupil center pixel coordinates and all corneal reflection point fitting circle center pixel coordinates obtained from the corrected image comprise: Face images covering different individual samples are collected in advance, and an image dataset with face, human eye, pupil and corneal reflection point as detection targets is constructed; A deep learning neural network for image segmentation is built, and the network model is trained on the image dataset; The image detection algorithm trained using the model is used to detect and identify the pupil and corneal reflection points in the corrected image; An adaptive gray value threshold segmentation method is used in the pupil segmentation area obtained by image recognition to further segment the dark pupil and iris, and an ellipse fitting is performed on the dark pupil edge to obtain the pixel coordinates of the pupil center in the image; The center coordinates of each reflection point segmentation area are calculated in the corneal reflection point segmentation result obtained by image recognition, and an ellipse fitting is performed on these coordinates to obtain the pixel coordinates of the fitted ellipse as the reflection center of the annular infrared light source in the image.

4. The robotically-assisted tracking and alignment method for optical coherence tomography of the eye according to claim 1, wherein, The binocular disparity based on the pupil center pixel coordinates, the external parameters obtained by camera calibration, and the triangulation principle are used to calculate the three-dimensional position of the pupil, and the gaze direction is calculated based on the pupil center pixel coordinates, the fitted circle center pixel coordinates of all corneal reflection points, and the pupil center-corneal reflection theory, including: Calculating the disparity based on the pupil center pixel coordinates in the images collected by the two cameras respectively; Based on the relative position and attitude between the cameras represented by the disparity and the camera external parameters, the coordinates of the pupil center in the three-dimensional space are calculated using the triangulation principle; Based on the pupil center and the annular light source corneal reflection center coordinates in the image collected by the intermediate camera, the yaw angle and pitch angle of the eyeball optical axis relative to the camera optical axis are calculated according to the eyeball corneal reflection model and the polar coordinate system transformation principle.

5. The robotically-assisted tracking and alignment method for optical coherence tomography of the eye according to claim 1, wherein, The pupil alignment target pose is obtained by combining the pupil three-dimensional position and gaze direction with the hand-eye calibration result, and the smooth motion trajectory is calculated based on the current state and safety range limit, including: Determining the pose of the target tool center in the current camera coordinate system based on the pupil three-dimensional position and gaze direction; Based on the relative position relationship between the robot end effector, tool center, and camera in the hand-eye calibration result, the target pose of the end effector in the world coordinate system is calculated; Reading the current pose and speed of the robot end effector, and setting the maximum boundary value conditions of position, speed, and acceleration; Based on the target pose of the robot end effector in the world coordinate system, the current pose and speed of the end effector, and the maximum boundary value conditions, an online trajectory generation algorithm is used to calculate the motion trajectory required for the robot to complete the pupil alignment.

6. The robotically-assisted tracking and alignment method for optical coherence tomography of the eye according to claim 1, wherein, Based on the motion trajectory, the online servo function of the robot is used to continuously issue servo points in the trajectory at a fixed servo period; During the motion process, the image and pupil positioning result are continuously refreshed, and if the target pose is changed, the new trajectory is switched to realize real-time tracking of the pupil until the pupil alignment is completed. When the pupil alignment is completed, optical coherence tomography is started to obtain ophthalmic three-dimensional volume images, and image registration is performed, including:

7. The robotically-assisted tracking and alignment method for optical coherence tomography of the eye according to claim 1, wherein, Comparing the current state of the robot with the target state and the pupil positioning result to confirm whether the pupil alignment is completed; ​ If the pupil alignment is confirmed, the optical coherence tomography system is started, cross-sectional scanning of the fundus or anterior segment is performed, and the tool center position is further fine-tuned according to the imaging results to obtain a better imaging position and angle; After adjustment, three-dimensional scanning imaging is performed, and image registration is performed to obtain high-quality ophthalmic optical coherence tomographic three-dimensional images.

8. A robot-assisted tracking and alignment device for optical coherence tomography of the eye, characterized in that The device comprises: The device adopts the robot-assisted tracking alignment method of ophthalmic optical coherence tomography in any one of claims 1-7 for robot-assisted tracking alignment.

Images