Eyeball tracking method and tracking system

Abstract

The present application relates to an eyeball tracking method and a tracking system. The method comprises: on the basis of a deep learning segmentation model, determining a first segmentation result of a reference image and a second segmentation result of a target image, and on the basis of the first segmentation result and the second segmentation result, determining displacement information of an eyeball to be tracked, so as to track said eyeball on the basis of the displacement information of said eyeball, wherein the reference image and the target image are images of said eyeball acquired by using a visual sensor, and the first segmentation result and the second segmentation result each comprise an iris region and a pupil region. By using the method, the accuracy of tracking an eyeball to be tracked can be improved.

Description

Eye tracking method and tracking system Related applications

[0001] This application claims priority to Chinese patent application number 2023117138747, filed on December 13, 2023, entitled “Eye Tracking Method and Tracking System,” which is hereby incorporated by reference in its entirety. Technical Field

[0002] The present application relates to the field of image technology, and in particular to an eye tracking method and tracking system. Background Art

[0003] For example, using an optical coherence tomography (OCT) scanner to track an eye is crucial. When using OCT to collect eye data, the eye is tracked to determine the target position after movement. The OCT scan position is then updated based on the target position, completing the acquisition of the patient's eye data.

[0004] In the related art, the eye to be tracked is tracked based on the reflective points on the cornea of ​​the eye to be tracked. However, since the relative positions of the reflective points on the cornea may change, the accuracy of tracking the eye to be tracked is low. Summary of the Invention

[0005] Based on this, it is necessary to provide an eye tracking method and tracking system that can improve the accuracy of tracking the eye to be tracked in order to address the above technical problems.

[0006] In a first aspect, the present application provides an eye tracking method, comprising:

[0007] Determining a first segmentation result of a reference image and a second segmentation result of a target image based on a deep learning segmentation model; the reference image and the target image are images of the eye to be tracked acquired using a visual sensor;

[0008] determining displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result;

[0009] The eye to be tracked is tracked according to the displacement information of the eye to be tracked.

[0010] In one embodiment, determining the displacement information of the eye to be tracked based on the first segmentation result and the second segmentation result includes:

[0011] The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result; the boundary feature information is determined according to the boundary between the pupil area and the iris area in the first segmentation result.

[0012] In one embodiment, both the first segmentation result and the second segmentation result further include a white eye region; and determining displacement information of the eye to be tracked based on the first segmentation result and the second segmentation result includes:

[0013] The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result, wherein the boundary feature information is determined according to the boundary between the iris area and the eye white area in the first segmentation result.

[0014] In one embodiment, determining displacement information based on boundary feature information in the first segmentation result and the second segmentation result includes:

[0015] determining a reference template from the first segmentation result according to boundary feature information in the first segmentation result;

[0016] Determine the area to be matched according to the reference template;

[0017] According to the to-be-matched area and the second segmentation result, the to-be-matched area with the largest matching coefficient with the reference template in the second segmentation result is used as the target matching area;

[0018] The displacement information of the eyeball to be tracked is determined according to a first preset position in the reference template and a second preset position in the target matching area corresponding to the first preset position.

[0019] In one embodiment, determining the displacement information of the eye to be tracked based on the first segmentation result and the second segmentation result includes:

[0020] Determining a first centroid position of the pupil region in the first segmentation result;

[0021] determining a second centroid position of the pupil region in the second segmentation result;

[0022] The displacement information of the eyeball to be tracked is determined according to the first center of mass position and the second center of mass position.

[0023] In one embodiment, tracking the eye to be tracked according to the displacement information of the eye to be tracked includes:

[0024] Performing polar coordinate conversion processing on the first iris region in the first segmentation result and the second iris region in the second segmentation result, respectively, to obtain a first iris polar coordinate map corresponding to the first iris region and a second iris polar coordinate map corresponding to the second iris region;

[0025] Performing template matching on the first iris polar coordinate image and the second iris polar coordinate image to obtain rotation information of the eyeball to be tracked;

[0026] The eyeball to be tracked is tracked based on the displacement information and rotation information.

[0027] In one embodiment, the image to be detected includes a segmentation result of the first eye image or a segmentation result of the second eye image; and the method further includes:

[0028] If the image to be detected meets all the preset conditions, the quality test result of the image to be detected is determined to be passed; wherein the preset conditions include at least one of the following:

[0029] The ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is greater than a preset ratio;

[0030] The near-circularity index of the pupil area in the image to be detected is greater than a preset index;

[0031] The ratio of the minimum distance from the pupil center to the iris region boundary of the image to be detected to the maximum distance from the pupil center to the iris boundary is greater than a preset ratio.

[0032] In one embodiment, the method further comprises:

[0033] If the area change between the pupil area in the reference image and the pupil area in the target image is greater than a first preset area difference, and / or the motion information corresponding to the target image is greater than a first preset difference, the scanning data obtained by the scanning device at the first scanning position is discarded, and the scanning device is controlled to re-acquire the scanning data of the eye to be tracked at the first scanning position; the motion information includes displacement information and / or rotation information; the first scanning position includes a scanning position that overlaps with the acquisition time of the target image.

[0034] In one embodiment, the method further comprises:

[0035] If a change in area between the pupil region in the first image and the pupil region in the second image is greater than a second preset area difference, and / or a difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference, then discarding scan data acquired by the scanning device at the first scanning position, and controlling the scanning device to reacquire scan data of the eye to be tracked at the second scanning position; the second scanning position includes a scanning position that overlaps with the acquisition time of the second image;

[0036] The first image and the second image are target images at two different time points, and the time point of the first image is earlier than the time point of the second image.

[0037] In a second aspect, the present application further provides an eye tracking device, comprising:

[0038] a first determining module, configured to determine, based on a deep learning segmentation model, a first segmentation result of a reference image and a second segmentation result of a target image; the reference image and the target image are images of the eye to be tracked acquired using a visual sensor; the first segmentation result and the second segmentation result both include an iris region and a pupil region;

[0039] A second determining module is used to determine the displacement information of the eye to be tracked according to the first segmentation result and the second segmentation result;

[0040] The tracking module is used to track the eyeball to be tracked according to the displacement information of the eyeball to be tracked.

[0041] In a third aspect, the present application further provides a tracking system, the tracking system comprising a visual sensor, a scanning device, and a tracking apparatus; the visual sensor comprises a pupil camera; the scanning device comprises an OCT;

[0042] A tracking device for executing any of the above-mentioned eye tracking methods.

[0043] In a fourth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0044] In a fifth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0045] In a sixth aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0046] In the above-mentioned eye tracking method and tracking system, the reference image and the target image are images of the eye to be tracked, acquired using a visual sensor. A first segmentation result of the reference image and a second segmentation result of the target image are then determined based on a deep learning segmentation model. Since both the first segmentation result and the second segmentation result include the iris region and the pupil region, the deep learning segmentation model is used to output a pixel-level segmentation model that at least includes the pupil and the iris. Furthermore, using the deep learning segmentation model, there is no need to track the eye based on reflective points on the cornea of ​​the eye to be tracked, thus avoiding the relative change in the position of the reflective points on the cornea. Based on the first segmentation result and the second segmentation result, the displacement information of the eye to be tracked can be more accurately determined, thereby improving the accuracy of tracking the eye to be tracked.

[0047] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without creative work.

[0049] FIG1 is a diagram illustrating an application environment of an eye tracking method according to an embodiment of the present application;

[0050] FIG2 is a schematic diagram of a flow chart of an eye tracking method in an embodiment of the present application;

[0051] FIG3 is a schematic diagram of an eye image provided in an embodiment of the present application;

[0052] FIG4 is a schematic diagram of a classification image in an embodiment of the present application;

[0053] FIG5 is a schematic diagram of boundary feature information in an embodiment of the present application;

[0054] FIG6 is a schematic diagram of a process for determining displacement information in an embodiment of the present application;

[0055] FIG7 is a schematic diagram of the size of a candidate matching area in an embodiment of the present application;

[0056] FIG8 is a schematic diagram of another process for determining displacement information in an embodiment of the present application;

[0057] FIG9 is a schematic diagram of a process for tracking an eye to be tracked according to an embodiment of the present application;

[0058] FIG10 is a schematic diagram of a process for determining rotation information according to an embodiment of the present application;

[0059] FIG11 is a schematic diagram of coordinate conversion in an embodiment of the present application;

[0060] FIG12 is a schematic diagram of a process for determining a reference image in an embodiment of the present application;

[0061] FIG13 is a schematic diagram of a rescanning process of a scanning device according to an embodiment of the present application;

[0062] FIG14 is a schematic diagram of pupil ratio in an embodiment of the present application;

[0063] FIG15 is a schematic diagram of the process of an eye tracking method according to an embodiment of the present application;

[0064] FIG16 is a structural block diagram of an eye tracking adjustment device according to an embodiment of the present application;

[0065] FIG17 is a schematic structural diagram of a tracking system in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0067] Currently, when tracking an eye, in some cases, tracking is performed based on the reflective points on the cornea of ​​the eye. In other cases, the pupil of the eye is segmented using threshold segmentation, and the eye is tracked based on the segmented pupil.

[0068] However, the relative positions of reflective points on the cornea vary. Furthermore, pupils vary in size, and the size of the pupils of diseased eyes may not be absolutely symmetrical. Consequently, current tracking accuracy of the eye is low. To address these technical issues, it is necessary to propose an eye tracking method, which will be described below.

[0069] FIG1 is a diagram of the application environment of the eye tracking method in the embodiment of the present application. The eye tracking method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . Among them, the computer device 102 can communicate with the visual sensor 101 and the scanning device 103 respectively. The visual sensor 101 is used to collect the eye image of the eye to be tracked. Furthermore, the computer device 102 can determine the reference image and the target image by using the image of the eye to be tracked acquired by the visual sensor 101, and determine the first segmentation result of the reference image and the second segmentation result of the target image based on the deep learning segmentation model, so as to determine the displacement information of the eye to be tracked according to the first segmentation result and the second segmentation result. Exemplarily, the computer device 102 can control the scanning device 103 to track the eye to be tracked according to the displacement information of the eye to be tracked.

[0070] The visual sensor 101 includes, but is not limited to, a pupil camera. The computer device 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The portable wearable devices may include smart watches, smart bracelets, head-mounted devices, etc. Of course, the computer device 102 may also be implemented as a standalone server or a server cluster consisting of multiple servers. The scanning device 103 includes, but is not limited to, an OCT device.

[0071] In some embodiments, the computer device 102 may also be disposed inside the scanning device 103. The computer device 102 includes but is not limited to at least one of a central processing unit (CPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices.

[0072] It should be noted that FIG1 only illustrates an application scenario for tracking an eye to be tracked. The eye tracking method provided in this embodiment can also be applied to other application scenarios, such as femtosecond surgery.

[0073] FIG2 is a flow chart of the eye tracking method in an embodiment of the present application. In an exemplary embodiment, as shown in FIG2 , an eye tracking method is provided, which is described by taking the method applied to the computer device in FIG1 as an example, and includes the following S201 to S203.

[0074] S201, based on the deep learning segmentation model, determine a first segmentation result of a reference image and a second segmentation result of a target image; the reference image and the target image are images of the eye to be tracked obtained using a visual sensor; the first segmentation result and the second segmentation result both include an iris area and a pupil area.

[0075] In this embodiment, the visual sensor can capture an eye image of the eye to be tracked. Figure 3 is a schematic diagram of an eye image provided in an embodiment of the present application. In some embodiments, the visual sensor can periodically capture an eye image of the eye to be tracked at a certain frequency and send the eye image to a computer device. Furthermore, the computer device can use the visual sensor to obtain a reference image and a target image of the eye to be tracked.

[0076] In some embodiments, the computer device may determine a reference image from multiple eye images according to preset rules. For example, the computer device may select an eye image with the center of the pupil as the reference image. For another example, the computer device may use an unobstructed eye image of the first eye as the reference image. In some embodiments, the computer device may also determine a reference image from multiple eye images based on a user selection in response to a user's selection.

[0077] Furthermore, after determining the reference image, the computer device may continue to use the visual sensor to acquire a target image of the eye to be tracked. For example, if the computer device determines to provide a reference image at time 1, the computer device may use the eye image of the eye to be tracked, captured by the image sensor at time 2, as the target image.

[0078] Furthermore, the computer device can determine a first segmentation result of the reference image and a second segmentation result of the target image based on the deep learning segmentation model. In some embodiments, the computer device can directly input the reference image into the deep learning segmentation model to obtain the first segmentation result, or it can pre-process the reference image and then input it into the deep learning segmentation model to obtain the first segmentation result. Similarly, the computer device can directly input the target image into the deep learning segmentation model to obtain the second segmentation result, or it can pre-process the reference image and then input it into the deep learning segmentation model to obtain the second segmentation result. Pre-processing includes but is not limited to grayscale processing, filtering processing, etc.

[0079] The deep learning segmentation model can be a model trained based on eye image samples and labels for each eye structure in the eye image samples. The deep learning segmentation model is used to segment the eye structures in the reference image or the target image, where the eye structures include at least the iris and the pupil. Furthermore, both the first segmentation result and the second segmentation result include the iris region and the pupil region.

[0080] In some embodiments, the eye structure may also include physiological structures such as eyelids and whites of the eyes, but this embodiment is not limited thereto.

[0081] The deep learning segmentation model can be a supervised learning model, a semi-supervised learning model, an unsupervised learning model, etc. The deep learning segmentation model can include, but is not limited to, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a fully convolutional neural network (FCN) model, a generative adversarial network (GAN) model, a radial basis function (RBF) model, a deep belief network (DBN) model, an Elman model, or at least one of a combination thereof. Exemplarily, the deep learning segmentation model can be a UNet model.

[0082] In an exemplary embodiment, the first segmentation result may be in the form of at least one of a probability function and a classification image; and the second segmentation result may be in the form of at least one of a probability function and a classification image.

[0083] Taking the first segmentation result as an example, the probability function in the first segmentation result is used to indicate the probability that each pixel in the reference image belongs to each eye structure. In other words, when the reference image is input into the deep learning segmentation model, the first segmentation result output by the deep learning segmentation model can include two channels of output image A and output image B.

[0084] The pixel value of each pixel in output image A represents the probability that the corresponding pixel belongs to the iris, and the pixel value of each pixel in output image B represents the probability that the corresponding pixel belongs to the pupil. For pixels at the same location in each output image, the sum of the probabilities is equal to 1. The second segmentation result is similar and will not be further described here.

[0085] The above example uses a two-channel output image as an example. In some embodiments, the first segmentation result output by the deep learning segmentation model may include a three-channel, four-channel, or other multi-channel output image. For example, the first segmentation result output by the deep learning segmentation model includes not only the output image A and output image B described above, but also output image C and output image D. The pixel value of each pixel in output image C is used to represent the probability value of the corresponding pixel belonging to the white of the eye, and the pixel value of each pixel in output image D is used to represent the probability value of the corresponding pixel belonging to the eyelid.

[0086] Continuing with the first segmentation result as an example, the classification image in the first segmentation result is used to indicate which ocular structure each pixel in the reference image belongs to. The classification image can be determined based on a probability function. For example, the computer device can determine the ocular structure to which a pixel belongs based on the maximum probability value among pixels at the same location in output images A and B.

[0087] Since the first segmentation result includes at least one of a probability function and a classification image, and the second segmentation result includes at least one of a probability function and a classification image, the flexibility of the first segmentation result and the second segmentation result is improved.

[0088] FIG4 is a schematic diagram of a classification image in an embodiment of the present application. Taking a reference image as an example, if the reference image is input into a deep learning segmentation model, the first segmentation result determined by the deep learning segmentation model can be as shown in FIG4 , that is, the eyelid, pupil, iris, and white of the eye are segmented from the reference image. The target image is similarly processed and will not be further described here.

[0089] S202: Determine the displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result.

[0090] In this embodiment, after obtaining the first and second segmentation results, the computer device can determine the displacement information of the eye to be tracked based on the first and second segmentation results. The displacement information includes the displacement amount and displacement direction of the eye to be tracked. In some embodiments, the displacement information may also be the coordinates of the pupil center of the eye to be tracked after movement, which is not a limitation in this embodiment.

[0091] In some embodiments, the computer device may use the location of the pupil region to perform template matching on the first segmentation result and the second segmentation result to determine the displacement information of the eye to be tracked. Alternatively, the computer device may use the location of the iris region to perform template matching on the first segmentation result and the second segmentation result to determine the displacement information of the eye to be tracked. The first segmentation result and the second segmentation result may use a probability function or a classified image.

[0092] S203: Track the eyeball to be tracked according to the displacement information of the eyeball to be tracked.

[0093] Eye tracking can be applied to scanning imaging. The computer device controls the driving components of the scanning imaging to make corresponding compensation based on the displacement information of the eye to be tracked, so as to quickly track the eye to be tracked.

[0094] Taking OCT imaging as an example, at the initial moment, the OCT scans the eye to be tracked at position 1. If the displacement information determined by the computer device based on the target image at moment 2 indicates that the eye to be tracked moves upward by 1 cm, the computer device can send the displacement information to the OCT to control the OCT to move upward from position 1 to position 2 after 1 cm, and scan the eye to be tracked at position 2 to obtain scanning data of the eye to be tracked. The scanning data is used to obtain a scanning image of the eye to be tracked.

[0095] In some embodiments, eye tracking can also be applied to ophthalmic surgery, such as in the context of femtosecond surgery.

[0096] In the above-described eye tracking method, the reference image and the target image are images of the eye to be tracked, acquired using a visual sensor. A first segmentation result for the reference image and a second segmentation result for the target image are then determined based on a deep learning segmentation model. Because both the first segmentation result and the second segmentation result include the iris region and the pupil region, the deep learning segmentation model is used to output a pixel-level segmentation model that at least includes the pupil and iris. Furthermore, using the deep learning segmentation model, there is no need to track the eye based on reflective points on the cornea of ​​the eye to be tracked, thus avoiding the relative change in the position of the reflective points on the cornea. Based on the first and second segmentation results, the displacement information of the eye to be tracked can be more accurately determined, thereby improving the accuracy of tracking the eye to be tracked.

[0097] In an exemplary embodiment, the above S202 can be implemented as follows:

[0098] The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result; the boundary feature information is determined according to the boundary between the pupil area and the iris area in the first segmentation result.

[0099] In this embodiment, the boundary feature information is used to indicate the boundary of the eye structure obtained in the first segmentation result. The boundary feature information can be determined based on information such as the pixel points and curvature of the boundary of the eye structure, which is not limited in this embodiment.

[0100] The boundary feature information can be determined based on the boundary between the pupil region and the iris region in the first segmentation result. FIG5 is a schematic diagram of a boundary feature information in an embodiment of the present application. As shown in FIG5 , region 501 in FIG5 includes the boundary between the pupil region and the iris region in the first segmentation result.

[0101] Furthermore, after determining the boundary feature information in the first segmentation result, the computer device may determine the displacement information according to the boundary feature information in the first segmentation result and the second segmentation result.

[0102] In some embodiments, the computer device can determine the displacement information of the eye to be tracked based on the boundary feature information in the first segmentation result and the second segmentation result through template matching method, feature matching method, scale-invariant feature transform (SIFT) method, fast nearest neighbor search algorithm (Fast Library for Approximate Nearest Neighbors, FLANN) and the like. This embodiment is not limited to this.

[0103] In the above embodiment, since the boundary feature information is determined based on the boundary between the pupil area and the iris area in the first segmentation result, the displacement information can be determined more accurately based on the boundary feature information in the first segmentation result and the second segmentation result by using the boundary feature information in the first segmentation result.

[0104] In an exemplary embodiment, both the first segmentation result and the second segmentation result further include the white of the eye area; the above-mentioned S202 can also be implemented in the following manner:

[0105] The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result, wherein the boundary feature information is determined according to the boundary between the iris region and the eye white region in the first segmentation result.

[0106] In this embodiment, as shown in Figure 4 , both the first and second segmentation results also include the white of the eye region. Because the pupil scales during the movement of the eye being tracked, causing the boundary between the pupil and iris regions to change, the boundary between the iris and white of the eye regions does not change due to the movement of the eye being tracked. Therefore, in this embodiment, the boundary feature information is determined based on the boundary between the iris and white of the eye regions in the first segmentation result. As shown in Figure 5 , region 502 in Figure 5 includes the boundary between the iris and white of the eye regions in the first segmentation result.

[0107] After determining the boundary feature information in the first segmentation result, the computer device can determine the displacement information based on the boundary feature information in the first segmentation result and the second segmentation result. The method for determining the displacement information can refer to the above embodiment and will not be repeated here.

[0108] In the above embodiment, since both the first and second segmentation results also include the white of the eye region, the boundary feature information is determined based on the boundary between the iris region and the white of the eye region in the first segmentation result. Therefore, the boundary feature information does not change with the movement of the eye being tracked, resulting in higher accuracy. Consequently, displacement information can be more accurately determined based on the boundary feature information in the first segmentation result and the second segmentation result.

[0109] In some embodiments, as shown in FIG5 , the boundary feature information may also include the boundary between the pupil region and the iris region, as well as the boundary between the iris region and the white of the eye region according to the first segmentation result.

[0110] FIG6 is a flow chart of determining displacement information in an embodiment of the present application. In an exemplary embodiment, as shown in FIG6 , the above-mentioned “determining displacement information based on boundary feature information in the first segmentation result and the second segmentation result” includes S601 to S604.

[0111] S601: Determine a reference template from the first segmentation result according to boundary feature information in the first segmentation result.

[0112] For example, the computer device may determine a reference template from the first segmentation result based on the position of edge pixels at the boundary between the iris and white of the eye regions in the first segmentation result and a preset expansion distance. Continuing with FIG5 as an example, the computer device extends the edge pixels of the iris and white of the eye regions in the reference image up, down, left, and right by the preset expansion distances to obtain region 502, which is then used as the reference template.

[0113] It should be noted that the following description will be given using one reference template as an example. There may also be multiple reference templates, and this embodiment does not impose any limitation thereto.

[0114] S602: Determine a region to be matched according to a reference template.

[0115] In this embodiment, the size of the area to be matched can be determined based on the preset maximum displacement of the eye to be tracked and the size of the reference template. Figure 7 is a schematic diagram of the size of the area to be matched in an embodiment of this application. For example, the computer device can expand the size of the area to be matched in four directions, length and width, based on the size of the reference template, as shown in area 701 in Figure 7. The preset maximum displacement of the eye to be tracked can be preset in advance.

[0116] In some embodiments, the area to be matched may also be the entire image of the second segmentation result, which is not limited in this embodiment.

[0117] S603 : Based on the to-be-matched region and the second segmentation result, the to-be-matched region in the second segmentation result having the largest matching coefficient with the reference template is used as a target matching region.

[0118] In this embodiment, the matching coefficient includes but is not limited to a parameter such as a correlation coefficient for indicating the matching between the reference template and the to-be-matched region in the second segmentation result. It is understood that a larger matching coefficient indicates a closer match between the two regions.

[0119] After the computer device determines the to-be-matched region, it can slide the to-be-matched region within the second segmentation result and calculate the matching coefficient between the to-be-matched region and the reference template when the to-be-matched region slides to different positions within the second segmentation result. Furthermore, the computer device can select the to-be-matched region in the second segmentation result with the highest matching coefficient with the reference template as the target matching region.

[0120] S604 : Determine the displacement information of the eyeball to be tracked according to the first preset position in the reference template and the second preset position in the target matching area corresponding to the first preset position.

[0121] The first preset position may be the center position of the reference template, and the second preset position may be the center position of the target matching area.

[0122] After determining the reference template and the target matching area, the computer device can then determine the displacement information of the eye to be tracked based on the first preset position in the reference template and the second preset position in the target matching area corresponding to the first preset position. For example, the computer device can subtract the second preset position in the target matching area from the first preset position in the reference template to determine the displacement information of the eye to be tracked.

[0123] In the above embodiment, since it is possible to determine a reference template from the first segmentation result based on the boundary feature information in the first segmentation result, determine the area to be matched based on the reference template, and use the area to be matched in the second segmentation result with the largest matching coefficient with the reference template as the target matching area based on the area to be matched and the second segmentation result, it is possible to determine more accurate displacement information of the eyeball to be tracked based on the first preset position in the reference template and the second preset position in the target matching area corresponding to the first preset position.

[0124] FIG8 is a schematic diagram of another flow chart of determining displacement information in an embodiment of the present application. In an exemplary embodiment, as shown in FIG8 , S202 includes S801 to S803 .

[0125] S801: Determine a first centroid position of the pupil area in a first segmentation result.

[0126] In this embodiment, the first centroid position is used to indicate the centroid position of the pupil region in the first segmentation result. In some embodiments, the computer device may determine the first centroid position by a centroid method or an ellipse fitting method.

[0127] In one embodiment, the computer device may determine the first centroid position according to the coordinates of each pixel point of the first segmentation result multiplied by the corresponding weight.

[0128] When the first segmentation result is a classification image, the weight is either 0 or 1. That is, the weight of the pixels in the pupil area in the first segmentation result is 1, and the weight of each pixel in the area other than the pupil area is 0. When the first segmentation result is a probability function, the weight is determined based on the probability value of each pixel in the output image B corresponding to the pupil. That is, the weight is between 0 and 1.

[0129] Therefore, in order to improve the accuracy of the first centroid position, in this embodiment, the first segmentation result includes a probability function, and then the computer device can determine the first centroid position of the pupil area in the first segmentation result.

[0130] S802: Determine a second centroid position of the pupil area in the second segmentation result.

[0131] The principle of S802 is the same as that of S801, and the second centroid position is used to indicate the centroid position of the pupil region in the second segmentation result. In some embodiments, the computer device may determine the first centroid position by a centroid method or an ellipse fitting method.

[0132] Similarly, in one embodiment, the computer device may determine the second centroid position by multiplying the coordinates of each pixel point in the second segmentation result by the corresponding weight. Furthermore, to improve the accuracy of the second centroid position, in this embodiment, the second segmentation result includes a probability function, so that the computer device can determine the second centroid position of the pupil region in the second segmentation result.

[0133] S803: Determine the displacement information of the eyeball to be tracked according to the first center of mass position and the second center of mass position.

[0134] The first and second centroid positions take into account the distribution of the pupil area and are more accurate than calculating the center position of the pupil area. Furthermore, after determining the first and second centroid positions, the computer device can determine the displacement information of the eye to be tracked based on the first and second centroid positions. For example, the computer device can subtract the second centroid position from the first centroid position to determine the displacement information of the eye to be tracked.

[0135] In the above embodiment, the first centroid position of the pupil area in the first segmentation result is determined, and the second centroid position of the pupil area in the second segmentation result is determined. In this way, based on the first centroid position and the second centroid position, the displacement information of the eyeball to be tracked can be determined efficiently and accurately.

[0136] FIG9 is a flowchart of tracking an eyeball to be tracked in an embodiment of the present application. In an exemplary embodiment, as shown in FIG9 , S203 includes S901 to S902 .

[0137] S901 : Determine rotation information of the eyeball to be tracked according to the first segmentation result and the second segmentation result.

[0138] In this embodiment, to improve tracking accuracy, the computer device also needs to determine the rotation information of the eye to be tracked. In some embodiments, the computer device can determine the rotation information of the eye to be tracked based on the first segmentation result and the second segmentation result using a feature matching method, but this embodiment is not limited to this.

[0139] S902: Track the eyeball to be tracked according to the displacement information and the rotation information.

[0140] Furthermore, based on the displacement and rotation information, the eye to be tracked can be accurately tracked. Similar to the principle of S203, eye tracking can be used for scanning imaging. In some embodiments, eye tracking can also be applied to ophthalmic surgery, such as in femtosecond surgery scenarios.

[0141] Taking OCT imaging as an example, at the initial moment, the OCT scans the eye to be tracked at position 1. If, based on the target image at moment 2, the computer device determines that the eye to be tracked moves upward by 1 cm, and the determined rotation information indicates that the eye to be tracked rotates clockwise by 10°, then the computer device can send the displacement information and rotation information to the OCT to control the OCT to move upward by 1 cm from position 1, and rotate clockwise by 10° to reach position 2, and scan the eye to be tracked at position 2 to obtain a scanned image of the eye to be tracked.

[0142] In the above embodiment, since it is also necessary to determine the rotation information of the eye to be tracked based on the first segmentation result and the second segmentation result, and track the eye to be tracked based on the displacement information and the rotation information, both the displacement movement and the rotation movement of the eye to be tracked are taken into account, thereby further improving the accuracy of tracking the eye to be tracked.

[0143] FIG10 is a schematic diagram of a flow chart of determining rotation information in an embodiment of the present application. In an exemplary embodiment, as shown in FIG10 , S902 includes S1001 to S1002 .

[0144] S1001 , performing polar coordinate conversion processing on a first iris region in a first segmentation result and a second iris region in a second segmentation result, respectively, to obtain a first iris polar coordinate map corresponding to the first iris region and a second iris polar coordinate map corresponding to the second iris region.

[0145] In this embodiment, the rotation information can be determined using the texture information in the iris. The first segmentation result and the second segmentation result can be classified images. Thus, the iris region in the first segmentation result can be used as the first iris region, and the iris region in the second segmentation result can be used as the second iris region.

[0146] Furthermore, the computer device performs polar coordinate conversion processing on the first iris region, that is, converts the first iris region from a Cartesian coordinate system to a polar coordinate system, thereby determining a first iris polar coordinate map corresponding to the first iris region.

[0147] Similarly, the computer device performs polar coordinate conversion processing on the second iris area to determine a second iris polar coordinate map corresponding to the second iris area.

[0148] In some embodiments, to accurately calculate rotation information later, the first iris polar coordinate map and the second iris polar coordinate map need to be the same size. Therefore, in some embodiments, the computer device can linearly stretch the first iris polar coordinate map and the second iris polar coordinate map so that the first iris polar coordinate map and the second iris polar coordinate map are the same size.

[0149] S1002: Perform template matching on the first iris polar coordinate image and the second iris polar coordinate image to obtain rotation information of the eyeball to be tracked.

[0150] Furthermore, the computer device can perform template matching on the first iris polar coordinate image and the second iris polar coordinate image to obtain rotation information of the eyeball to be tracked.

[0151] Figure 11 is a schematic diagram of coordinate transformation in an embodiment of the present application. Figure 11(a) shows the first iris region of the eye to be tracked before rotational movement. The first iris region includes Texture 1, Texture 2, Texture 3, and Texture 4. Polar coordinate transformation is performed on the first iris region shown in Figure 11(a), resulting in the first iris polar coordinate diagram shown in Figure 11(b).

[0152] Figure 11(c) shows the second iris region after the eyeball to be tracked has rotated. The second iris region still includes Texture 1, Texture 2, Texture 3, and Texture 4. Performing polar coordinate transformation on the second iris region shown in Figure 11(c) yields the second iris polar coordinate diagram shown in Figure 11(d).

[0153] The first iris polar coordinate diagram and the second iris polar coordinate diagram have the same size. In the first iris polar coordinate diagram and the second iris polar coordinate diagram, the upper boundary represents the outer edge of the iris, and the lower boundary represents the inner edge of the iris.

[0154] As can be seen from Figure 11, since the eye to be tracked rotates, the position of the texture between the first iris polar coordinate map and the second iris polar coordinate map will also move. Therefore, the template matching method can be used to determine the rotation information of the eye to be tracked. Exemplarily, the computer device slides the first iris polar coordinate map and uses a cyclic boundary condition in the theta direction of the polar coordinates to determine the matching coefficient between the first iris polar coordinate map and the second iris polar coordinate map after each slide. The rotation information of the eye to be tracked is determined based on the sliding distance at which the matching coefficient is maximized. The theta direction is also the horizontal rightward direction in Figures 11(b) and 11(d).

[0155] In the above embodiment, polar coordinate conversion is performed on the first iris region in the first segmentation result and the second iris region in the second segmentation result, respectively, to obtain a first iris polar coordinate map corresponding to the first iris region and a second iris polar coordinate map corresponding to the second iris region. Therefore, after template matching is performed on the first iris polar coordinate map and the second iris polar coordinate map, the rotation information of the eye to be tracked can be obtained more accurately.

[0156] FIG12 is a schematic diagram of a process for determining a reference image in an embodiment of the present application. In an exemplary embodiment, as shown in FIG12 , the above-mentioned eye tracking method further includes S1201 to S1204 .

[0157] S1201: Acquire multiple first eye images of the eye to be tracked based on a visual sensor.

[0158] In this embodiment, the visual sensor can periodically capture multiple first eye images of the eye to be tracked at a certain capture frequency and send the first eye images to the computer device. For example, the computer device captures one first eye image every 50 milliseconds, and the computer device can capture 20 first eye images within the previous second. The first eye images can be seen in Figure 3.

[0159] S1202: Obtain segmentation results of each first eye image according to a deep learning segmentation model.

[0160] The computer device may directly input each first ocular image into the deep learning segmentation model to obtain a segmentation result for each first ocular image. The computer device may also pre-process each first ocular image and then input it into the deep learning segmentation model to obtain a segmentation result for each first ocular image.

[0161] S1203: Determine a quality detection result corresponding to the segmentation result of each first eye image.

[0162] Furthermore, the computer device performs a quality check on the segmentation results of each first ocular image to determine a quality check result corresponding to the segmentation result of each first ocular image. The quality check result may include a pass or fail, and the quality check result may also be represented by a quality score, where a higher score indicates better quality.

[0163] The quality detection result corresponding to the segmentation result of the first eye image can be used to characterize the occlusion of the pupil and / or iris in the segmentation result of the first eye image. For example, the computer device can determine that the quality detection result of the eye to be tracked passes if the area ratio of the iris region in the first eye image is greater than a preset value.

[0164] Of course, in some embodiments, the quality detection results can also be used to characterize the signal-to-noise ratio, resolution, and other aspects of the segmentation results. In some embodiments, the computer device can determine a quality score based on the weights of the various quality detection items. For example, the less occluded the first eye image is and the higher the signal-to-noise ratio is, the higher the quality score will be.

[0165] S1204: Determine a reference image from each first ocular image according to each quality detection result.

[0166] Furthermore, the computer device may determine a reference image from each first ocular image according to a quality detection result of each first ocular image.

[0167] In some embodiments, if the quality detection result includes a quality score, the computer device may use the first ocular image with the highest quality score as the reference image. If the quality detection result includes a pass or fail, the computer device may use the first ocular image with a pass quality detection result as the reference image, although this embodiment is not limited to this.

[0168] In the above embodiment, multiple first eye images of the eye to be tracked are obtained based on a visual sensor, and the segmentation results of each first eye image are obtained according to a deep learning segmentation model. Since the quality detection results corresponding to the segmentation results of each first eye image are determined, and a reference image is determined from each first eye image based on each quality detection result, the quality of the reference image is improved, thereby improving the accuracy of subsequent tracking.

[0169] FIG13 is a flowchart of a scanning device rescanning process in an embodiment of the present application. In an exemplary embodiment, as shown in FIG13 , before tracking the eye to be tracked, the above-mentioned eye tracking method further includes S1301 to S1304.

[0170] S1301, obtaining a second eye image of the eye to be tracked based on a visual sensor, and controlling a scanning device to obtain scanning data of the eye to be tracked.

[0171] In this embodiment, after determining the reference image, the computer device can continue to acquire a second ocular image of the eye to be tracked using the visual sensor and control the scanning device to acquire scan data of the eye to be tracked. Controlling the scanning device to acquire scan data of the eye to be tracked may include controlling the scanning device to acquire scan data at a first position. The second ocular image includes at least one frame of ocular images. The second ocular image can also be seen in Figure 3. For example, after determining the reference image at time 1, the computer device continues to acquire a second ocular image of the eye to be tracked at time 2.

[0172] Furthermore, after determining the reference image, the scanning device can start acquiring scanning data of the eye to be tracked. For example, the scanning device can continue to acquire scanning data of the eye to be tracked after time 1.

[0173] It should be noted that the visual sensor may collect eye images and the scanning device may obtain scanning data asynchronously.

[0174] S1302: Obtain a segmentation result of the second eye image according to the deep learning segmentation model.

[0175] Similarly, similar to the principle of S1202, the computer device can directly input each second ocular image into the deep learning segmentation model to obtain a segmentation result for each first ocular image. The computer device can also pre-process each second ocular image and then input it into the deep learning segmentation model to obtain a segmentation result for each first ocular image.

[0176] Continuing with the above example, the computer device determines the segmentation result of the second eye image based on the second eye image at time 2.

[0177] S1303: Determine a quality detection result corresponding to the segmentation result of the second eye image.

[0178] The principle of S1303 is similar to that of S1203. The quality test result can include pass or fail, and the quality test result can also be expressed as a quality score, where the larger the score, the better the quality.

[0179] The quality detection result corresponding to the segmentation result of the second eye image can be used to indicate occlusion of the pupil and / or iris in the segmentation result of the second eye image. For example, the computer device can determine that the quality detection result of the eye to be tracked passes if the area ratio of the iris region in the second eye image is greater than a preset value.

[0180] Of course, in some embodiments, the quality detection results can also be used to characterize the signal-to-noise ratio, resolution, and other aspects of the segmentation results. In some embodiments, the computer device can determine a quality score based on the weights of the various quality detection items. For example, the less occluded portions of the second eye image and the higher the signal-to-noise ratio, the higher the quality score.

[0181] S1304: Control the scanning device to reacquire scanning data of the eye to be tracked according to the quality detection result corresponding to the segmentation result of the second eye image.

[0182] In some embodiments, when the quality detection result includes a quality score, if the quality score corresponding to the second eye image is less than a preset score, the computer device can return to step S1301, that is, re-acquire the second eye image of the eye to be tracked based on the visual sensor, and control the scanning device to re-acquire the scanning data of the eye to be tracked.

[0183] In the case where the quality inspection result includes pass or fail, if the quality inspection result of the second eye image is fail, the computer device may also return to step S1301, re-acquire the second eye image of the eye to be tracked based on the visual sensor, and control the scanning device to re-acquire the scanning data of the eye to be tracked. This embodiment is not limited to this.

[0184] That is to say, when the quality of the second eye image is poor, the computer device can re-acquire the second eye image of the eye to be tracked based on the visual sensor, and control the scanning device to re-acquire the scanning data of the eye to be tracked until the quality of the second eye image is good. Eye tracking can then be performed based on the second eye image and the reference image.

[0185] In some embodiments, when the second eye image quality detection result is passed, or the quality score of the second eye image is not less than a preset score, the computer device can use the second eye image as the target image and then enter step S201.

[0186] In the above embodiment, since the second eye image of the eye to be tracked is obtained based on the visual sensor, and the scanning device is controlled to obtain the scanning data of the eye to be tracked, and the segmentation result of the second eye image is obtained according to the deep learning segmentation model, the quality detection result corresponding to the segmentation result of the second eye image is determined, and according to the quality detection result corresponding to the segmentation result of the second eye image, the scanning device is controlled to re-acquire the scanning data of the eye to be tracked. Therefore, through the quality detection result, the influence of large eye movements and blinking can be reduced, and the quality of the scanning data is improved.

[0187] In an exemplary embodiment, the image to be detected includes a segmentation result of the first eye image or a segmentation result of the second eye image; and the above-mentioned eye tracking method further includes:

[0188] If the image to be detected meets all the preset conditions, the quality test result of the image to be detected is determined to be passed; wherein the preset conditions include at least one of the following:

[0189] The ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is greater than a preset ratio;

[0190] The near-circularity index of the pupil area in the image to be detected is greater than a preset index;

[0191] The ratio of the minimum distance from the pupil center to the iris region boundary of the image to be detected to the maximum distance from the pupil center to the iris boundary is greater than a preset ratio.

[0192] In this embodiment, the quality detection result can be used to characterize the occlusion of the eye to be tracked. The quality detection result includes a pass or fail. The image to be detected includes the segmentation result of the first eye image or the segmentation result of the second eye image.

[0193] The system determines whether the ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is greater than a preset ratio to determine whether there is eyelid occlusion in the image to be detected. In other words, if the ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is not greater than the preset ratio, it indicates that there is eyelid occlusion in the image to be detected, and the pupil cannot be clearly seen.

[0194] Determining whether the pupil area's near-circularity index in the image being detected is greater than a preset index is also used to determine whether the pupil is obstructed. It is understood that without eyelid occlusion, the shape of the pupil being tracked is nearly circular. Therefore, if the pupil area's near-circularity index in the image being detected is not greater than a preset index, it indicates that there is occlusion in the image being detected.

[0195] The near-circularity index can be determined based on the shape of the pupil region of the image to be detected. For example, the near-circularity index of the pupil region can be determined based on the shape and perimeter of the pupil region. For example, the computer device can determine the near-circularity index of the pupil region based on W=, where W represents the near-circularity index, S represents the area of ​​the pupil region, and C represents the perimeter of the pupil region.

[0196] Similarly, whether the ratio of the minimum distance from the center of the pupil to the iris region boundary of the image to be detected to the maximum distance from the center of the pupil to the iris region boundary is greater than a preset ratio can also be used to determine whether the pupil is obstructed. Figure 14 is a schematic diagram of the pupil ratio in an embodiment of the present application. As shown in Figure 14, the minimum distance from the center of the pupil to the iris region boundary of the image to be detected is L2, and the maximum distance from the center of the pupil to the iris region boundary is L1. If the ratio between the minimum distance L2 and the maximum distance L1 is less than the preset ratio, it indicates that there is occlusion in the image to be detected.

[0197] In some embodiments, the upper edge of the iris region and the upper edge of the pupil region in the image to be detected may overlap. In this case, the distance between the pupil center and the upper edge of the iris region may be considered to be the minimum distance L2.

[0198] It should be noted that the image to be tested must meet all of the preset conditions before the computer device will determine that the quality inspection result is passed. For example, if the preset conditions include the ratio of the pupil area of ​​the image to be tested to the area of ​​the image to be tested being greater than a preset ratio, and the near-circularity index of the pupil area in the image to be tested being greater than a preset index, then the image to be tested must also meet the conditions of the pupil area being greater than the preset ratio and the near-circularity index of the pupil area being greater than the preset index for the computer device to determine that the quality inspection result of the image to be tested is passed.

[0199] In the above embodiment, since the preset conditions include at least one of the following: the ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is greater than a preset ratio, the near-circularity index of the pupil area in the image to be detected is greater than a preset index, and the ratio of the minimum distance from the pupil center to the iris area boundary of the image to be detected to the maximum distance from the pupil center to the iris boundary is greater than a preset ratio, and only when all the preset conditions are met will the quality detection result of the image to be detected be determined to be passed, thereby improving the accuracy of the quality detection result.

[0200] In an exemplary embodiment, the eye tracking method further includes the following steps:

[0201] If the area change between the pupil area in the reference image and the pupil area in the target image is greater than a first preset area difference, and / or the motion information corresponding to the target image is greater than a first preset difference, the scanning data obtained by the scanning device at the first scanning position is discarded, and the scanning device is controlled to re-acquire the scanning data of the eye to be tracked at the first scanning position; the motion information includes displacement information and / or rotation information; the first scanning position includes a scanning position that overlaps with the acquisition time of the target image.

[0202] In this embodiment, the first preset area difference and the first preset difference are used to limit the range of movement allowed for the eye to be tracked.

[0203] If the difference in area between the pupil region in the reference image and the pupil region in the target image is greater than a first predetermined area difference, it indicates that the pupil in the reference image and the target image has changed significantly. In this case, it can be considered that the pupil of the eye being tracked has moved significantly. The area difference may include the difference between the area of ​​the pupil region in the reference image and the area of ​​the pupil region in the target image.

[0204] If the motion information corresponding to the target image is greater than the first preset difference, it means that the displacement information and / or rotation information determined by the computer device based on the target image has changed too much relative to the reference image. In this case, it can also be considered that the movement of the eyeball to be tracked is too large.

[0205] Exemplarily, assuming that a reference image is determined at time 1 and a target image A is determined at time 2, the computer device determines whether the area change between the pupil area in the reference image and the pupil area in the target image A is greater than a preset area difference, and / or whether the motion information corresponding to the target image A is greater than a preset difference.

[0206] Furthermore, if the change in area between the pupil region in the reference image and the pupil region in the target image is greater than a preset area difference, and / or the motion information corresponding to the target image is greater than a first preset difference, the computer device may deem the previous motion of the eye to be tracked unreliable, discard the scan data acquired by the scanning device at the first scanning position, and control the scanning device to reacquire scan data of the eye to be tracked at the first scanning position. Furthermore, the computer device may continue to acquire the next frame of the target image to continue calculating the motion information of the eye to be tracked based on the next frame of the target image and the reference image.

[0207] The first scanning position includes a scanning position that overlaps with the acquisition time of the target image. When acquiring data, both the scanning device and the visual sensor establish a timestamp for the acquired data. It is understood that the data acquired by the scanning device refers to the scan data, while the data acquired by the visual sensor refers to the eye image. In other words, the computer device is capable of determining the acquisition time corresponding to the scan data and the acquisition time corresponding to the eye image.

[0208] In this way, when the motion information corresponding to the target image is greater than the first preset difference, the computer device can determine the scanning position that overlaps with the acquisition time of the target image, use the scanning position as the first scanning position, and control the scanning device to re-acquire the scanning data of the eye to be tracked at the first scanning position.

[0209] In other words, when the movement of the eye to be tracked in the target image is considered unreliable, the computer device can control the scanning device to start rescanning at the scanning position corresponding to the scanning data overlapping with the acquisition time of the target image.

[0210] In some embodiments, if the area change between the pupil area in the reference image and the pupil area in the target image is not greater than a first preset area difference, and / or the motion information corresponding to the target image is not greater than a first preset difference, the computer device can control the scanning device to move from the first scanning position to the third scanning position according to the calculated motion information, and control the scanning device to obtain scanning data of the eye to be tracked at the third scanning position, so as to achieve tracking of the eye to be tracked, thereby improving the accuracy of eye tracking.

[0211] In the above embodiment, since the area change between the pupil area in the reference image and the pupil area in the target image is greater than the first preset area difference, and / or the motion information corresponding to the target image is greater than the first preset difference, the scanning data acquired by the scanning device at the first scanning position is discarded, and the scanning device is controlled to reacquire the scanning data of the eye to be tracked at the first scanning position, and the first scanning position includes a scanning position that overlaps with the acquisition time of the target image. Therefore, the impact of excessive movement of the eye to be tracked is reduced, and the tracking accuracy is improved.

[0212] In an exemplary embodiment, the eye tracking method further includes the following steps:

[0213] If the area change between the pupil area in the first image and the pupil area in the second image is greater than a second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference, the scanning data obtained by the scanning device at the second scanning position is discarded, and the scanning device is controlled to re-acquire the scanning data of the eye to be tracked at the second scanning position; the second scanning position includes a scanning position that overlaps with the acquisition time of the second image; the first image and the second image are target images at two different time points, and the time point of the first image is earlier than the time point of the second image.

[0214] In this embodiment, the second preset area difference and the second preset difference are used to limit the maximum allowable movement speed of the eye to be tracked.

[0215] The first image and the second image are target images at two different time points, with the first image's time point being earlier than the second image's time point. The first image and the second image can be images whose scanning times are not adjacent or adjacent. For example, assuming that target image A is determined at time 2 and target image B is determined at time 3, target image A is the first image and target image B is the second image.

[0216] If the area change between the pupil region in the first image and the pupil region in the second image is greater than a second predetermined area difference, it indicates that the pupil in the reference image and the target image is changing too quickly. In this case, it can be considered that the pupil of the eye to be tracked is moving too quickly. The area change may include the difference between the area of ​​the pupil region in the first image and the area of ​​the pupil region in the second image.

[0217] Similarly, if the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference, it also indicates that the movement of the eye to be tracked is too fast. Similarly, the motion information includes displacement information and / or rotation information.

[0218] Continuing with the example of determining target image A at moment 2 and determining target image B at moment 3, the computer device determines whether the area change between the pupil area in target image A and the pupil area in target image B is greater than a second preset area difference, and / or whether the difference between the motion information corresponding to target image A and the motion information corresponding to target image B is greater than a second preset difference.

[0219] Furthermore, if the change in area between the pupil region in the first image and the pupil region in the second image is greater than a second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference, the computer device may determine that the motion of the eye to be tracked is unreliable, discard the scan data acquired by the scanning device at the second scanning position, and control the scanning device to reacquire scan data of the eye to be tracked at the second scanning position. Furthermore, the computer device may continue to acquire the next target image frame to continue calculating the movement information of the eye to be tracked based on the next target image frame and the reference image.

[0220] The second scanning position includes a scanning position that overlaps with the acquisition time of the second image. In this way, if the movement of the eye to be tracked in the second image is deemed unreliable, the computer device can control the scanning device to start rescanning at the scanning position corresponding to the scanning data that overlaps with the acquisition time of the second image.

[0221] In some embodiments, if the area change between the pupil area in the first image and the pupil area in the second image is not greater than a second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is not greater than a second preset difference, the computer device can control the scanning device to move from the second scanning position to the third scanning position according to the calculated motion information, and control the scanning device to obtain scanning data of the eye to be tracked at the third scanning position, so as to achieve tracking of the eye to be tracked, thereby improving the accuracy of eye tracking.

[0222] In the above embodiment, since the area change between the pupil area in the first image and the pupil area in the second image is greater than the second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than the second preset difference, the scanning data acquired by the scanning device at the second scanning position is discarded, and the scanning device is controlled to re-acquire the scanning data of the eye to be tracked at the second scanning position, and the second scanning position includes a scanning position that overlaps with the acquisition time of the second image. The first image and the second image are target images at two different time points, and the time point of the first image is earlier than the time point of the second image. Therefore, the impact of excessive movement of the eye to be tracked is reduced, and the tracking accuracy is improved.

[0223] To more clearly describe the eye tracking method of the present application, an explanation is provided herein in conjunction with Figure 15. Figure 15 is a schematic diagram of a process of an eye tracking method in an embodiment of the present application. As shown in Figure 15, a computer device may execute the eye tracking method according to the following process.

[0224] S1501: Acquire multiple first eye images of the eye to be tracked based on a visual sensor.

[0225] S1502: Obtain segmentation results of each first eye image according to a deep learning segmentation model.

[0226] S1503: Determine a quality inspection result corresponding to the segmentation result of each first eye image. If the ratio of the pupil area in the first eye image to the area of ​​the first eye image is greater than a preset ratio, the near-circularity index of the pupil area in the first eye image is greater than a preset index, and the ratio of the minimum distance from the center of the pupil to the boundary of the iris area to the maximum distance from the center of the pupil to the boundary of the iris area in the first eye image is greater than a preset ratio, then the quality inspection result of the first eye image is determined to pass.

[0227] S1504: Determine a reference image from each first ocular image according to each quality detection result.

[0228] S1505 , obtaining a second eye image of the eye to be tracked based on the visual sensor, and controlling the scanning device to obtain scanning data of the eye to be tracked.

[0229] S1506: Obtain a segmentation result of the second eye image according to the deep learning segmentation model.

[0230] S1507, determine the quality detection result corresponding to the segmentation result of the second eye image. If the quality detection result fails, return to execute step S1505 to control the scanning device to rescan. If the quality detection result passes, use the second eye image as the target image and enter step S1509. Among them, if the ratio of the pupil area in the second eye image to the area of ​​the second eye image is greater than a preset ratio, the near-circularity index of the pupil area in the second eye image is greater than a preset index, and the ratio of the minimum distance from the pupil center to the iris area boundary of the second eye image to the maximum distance from the pupil center to the iris boundary is greater than a preset ratio, then it is determined that the quality detection result of the second eye image passes.

[0231] S1508: Determine displacement information of the eye to be tracked based on the first segmentation result and the second segmentation result. The first segmentation result is a segmentation result determined based on the deep learning segmentation model and the reference image, and the second segmentation result is a segmentation result determined based on the deep learning segmentation model and the target image. In some embodiments, the computer device may determine the displacement information in accordance with S601 to S604, or in accordance with S801 to S803.

[0232] S1509: Perform polar coordinate conversion on the first iris region in the first segmentation result and the second iris region in the second segmentation result, respectively, to obtain a first iris polar coordinate map corresponding to the first iris region and a second iris polar coordinate map corresponding to the second iris region. The first iris polar coordinate map and the second iris polar coordinate map are of the same size.

[0233] S1510 , performing template matching on the first iris polar coordinate image and the second iris polar coordinate image to obtain rotation information of the eyeball to be tracked.

[0234] S1511: If a preset motion condition is met, the scanning data acquired by the scanning device at the first scanning position or the second scanning position is discarded, and the process returns to step S1505 to control the scanning device to rescan at the first scanning position or the second scanning position. The preset motion condition includes: a change in area between the pupil region in the reference image and the pupil region in the target image is greater than a first preset area difference; motion information corresponding to the target image is greater than a first preset difference; a change in area between the pupil region in the first image and the pupil region in the second image is greater than a second preset area difference; and a difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference.

[0235] S1512. If the preset motion condition is not met, the scanning device is controlled to move from the first scanning position or the second scanning position to the third scanning position according to the displacement information and the rotation information, and the scanning device is controlled to obtain scanning data of the eye to be tracked at the third scanning position to track the eye to be tracked.

[0236] S1501 to S1513 may refer to the above embodiment and will not be described in detail here.

[0237] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0238] Based on the same inventive concept, embodiments of the present application further provide an eye-tracking device for implementing the aforementioned eye-tracking method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more eye-tracking device embodiments provided below can be found in the above-described limitations of the eye-tracking method and will not be further elaborated here.

[0239] FIG16 is a block diagram of an eye tracking adjustment device according to an embodiment of the present application. In an exemplary embodiment, as shown in FIG16 , an eye tracking device 1600 is provided, comprising: a first determination module 1601, a second determination module 1602, and a tracking module 1603, wherein:

[0240] The first determination module 1601 is used to determine a first segmentation result of a reference image and a second segmentation result of a target image based on a deep learning segmentation model; the reference image and the target image are images of the eye to be tracked obtained using a visual sensor; the first segmentation result and the second segmentation result both include an iris area and a pupil area.

[0241] The second determining module 1602 is configured to determine the displacement information of the eye to be tracked according to the first segmentation result and the second segmentation result.

[0242] The tracking module 1603 is used to track the eye to be tracked according to the displacement information of the eye to be tracked.

[0243] In the above-mentioned eye-tracking device, the reference image and the target image are images of the eye to be tracked, acquired using a visual sensor. A first segmentation result for the reference image and a second segmentation result for the target image are then determined based on a deep learning segmentation model. Because both the first and second segmentation results include the iris and pupil regions, the deep learning segmentation model is used to output a pixel-level segmentation model that at least includes the pupil and iris. Furthermore, using the deep learning segmentation model, the eye to be tracked need not be tracked based on reflective points on the cornea of ​​the eye to be tracked, thus avoiding the relative change in the position of the reflective points on the cornea. Based on the first and second segmentation results, the displacement information of the eye to be tracked can be more accurately determined, thereby improving the accuracy of tracking the eye to be tracked.

[0244] In some embodiments, the second determination module 1602 is further configured to determine displacement information based on the boundary feature information in the first segmentation result and the second segmentation result; the boundary feature information is determined based on the boundary between the pupil area and the iris area in the first segmentation result.

[0245] In some embodiments, both the first segmentation result and the second segmentation result also include the white of the eye area; the second determination module 1602 is also used to determine the displacement information based on the boundary feature information in the first segmentation result and the second segmentation result, and the boundary feature information is determined based on the boundary between the iris area and the white of the eye area in the first segmentation result.

[0246] In some embodiments, the tracking module 1603 includes:

[0247] Used to determine the rotation information of the eye to be tracked according to the first segmentation result and the second segmentation result.

[0248] The tracking unit is used to track the eyeball to be tracked according to the displacement information and the rotation information.

[0249] FIG17 is a schematic diagram of the structure of a tracking system according to an embodiment of the present application. As shown in FIG17 , in one embodiment, a tracking system 1700 is provided. Tracking system 1700 includes a visual sensor 1701, a scanning device 1702, and any of the aforementioned tracking devices 1600. The visual sensor 1701 includes a pupil camera, and the scanning device 1702 includes an OCT.

[0250] In tracking system 1700, visual sensor 1701 acquires a reference image and a target image. Tracking device 1600 then determines a first segmentation result for the reference image and a second segmentation result for the target image based on a deep learning segmentation model. Based on the first and second segmentation results, it determines displacement information of the eye to be tracked. Based on the displacement information of the eye to be tracked, it controls scanning device 1702 to track the eye to be tracked. The tracking system 1700 can refer to the above-described embodiments and will not be described in detail here.

[0251] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0252] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0253] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An eye tracking method, wherein: The method comprises: Determine a first segmentation result of a reference image and a second segmentation result of a target image; the reference image and the target image are images of an eyeball to be tracked acquired by a visual sensor; the first segmentation result and the second segmentation result both include an iris area and a pupil area; Determining displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result; The eyeball to be tracked is tracked according to the displacement information of the eyeball to be tracked.

2. The method according to claim 1, wherein: The step of determining the displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result includes: The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result; the boundary feature information is determined according to the boundary between the pupil area and the iris area in the first segmentation result.

3. The method according to claim 1, wherein: The first segmentation result and the second segmentation result both further include a white eye area; and determining the displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result includes: The displacement information is determined according to the boundary feature information in the first segmentation result and the second segmentation result, wherein the boundary feature information is determined according to the boundary between the iris area and the white of the eye area in the first segmentation result.

4. The method according to claim 2 or 3, wherein: The determining the displacement information according to the boundary feature information in the first segmentation result and the second segmentation result includes: Determining a reference template from the first segmentation result according to the boundary feature information in the first segmentation result; Determine a to-be-matched area according to the reference template; According to the to-be-matched area and the second segmentation result, taking the to-be-matched area in the second segmentation result with the largest matching coefficient with the reference template as a target matching area; The displacement information of the eyeball to be tracked is determined according to a first preset position in the reference template and a second preset position in the target matching area corresponding to the first preset position.

5. The method according to any one of claims 1 to 4, wherein: The step of determining the displacement information of the eyeball to be tracked according to the first segmentation result and the second segmentation result includes: Determining a first centroid position of the pupil region in the first segmentation result; Determining a second centroid position of the pupil region in the second segmentation result; The displacement information of the eyeball to be tracked is determined according to the first center of mass position and the second center of mass position.

6. The method according to any one of claims 1 to 5, wherein: Tracking the eyeball to be tracked according to the displacement information of the eyeball to be tracked includes: Determining rotation information of the eyeball to be tracked according to the first segmentation result and the second segmentation result; The eyeball to be tracked is tracked according to the displacement information and the rotation information.

7. The method according to claim 6, wherein: The step of determining the rotation information of the eyeball to be tracked according to the first segmentation result and the second segmentation result includes: Performing polar coordinate conversion processing on the first iris region in the first segmentation result and the second iris region in the second segmentation result respectively, to obtain a first iris polar coordinate map corresponding to the first iris region and a second iris polar coordinate map corresponding to the second iris region; Template matching is performed on the first iris polar coordinate image and the second iris polar coordinate image to obtain rotation information of the eyeball to be tracked.

8. The method according to any one of claims 1 to 7, wherein: The determining of a first segmentation result of the reference image and a second segmentation result of the target image comprises: Based on the deep learning segmentation model, the first segmentation result and the second segmentation result are determined.

9. The method according to claim 8, wherein: The method further comprises: Acquire a plurality of first eye images of the eye to be tracked based on the visual sensor; Obtaining a segmentation result of each of the first eye images according to the deep learning segmentation model; Determine a quality detection result corresponding to the segmentation result of each of the first eye images; The reference image is determined from each of the first eye images according to each of the quality detection results.

10. The method according to claim 8, wherein: The method further comprises: Acquire a second eye image of the eye to be tracked based on the visual sensor, and control a scanning device to acquire scanning data of the eye to be tracked; Obtaining a segmentation result of the second eye image according to the deep learning segmentation model; Determining a quality detection result corresponding to the segmentation result of the second eye image; According to the quality detection result corresponding to the segmentation result of the second eye image, the scanning device is controlled to re-acquire the scanning data of the eye to be tracked.

11. The method according to any one of claims 1 to 10, wherein: The image to be detected includes a segmentation result of the first eye image or a segmentation result of the second eye image; the method further includes: If the image to be detected meets all the preset conditions, it is determined that the quality detection result of the image to be detected is passed; wherein the preset conditions include at least one of the following: The ratio of the pupil area of ​​the image to be detected to the area of ​​the image to be detected is greater than a preset ratio; The near-circularity index of the pupil area in the image to be detected is greater than a preset index; The ratio of the minimum distance from the pupil center to the iris region boundary of the image to be detected to the maximum distance from the pupil center to the iris boundary is greater than a preset ratio.

12. The method according to any one of claims 1 to 11, wherein: The method further comprises: If the area change between the pupil area in the reference image and the pupil area in the target image is greater than a first preset area difference, and / or the motion information corresponding to the target image is greater than a first preset difference, the scanning data acquired by the scanning device at the first scanning position is discarded, and the scanning device is controlled to re-acquire the scanning data of the eye to be tracked at the first scanning position; the motion information includes displacement information and / or rotation information; the first scanning position includes a scanning position that overlaps with the acquisition time of the target image.

13. The method according to claim 12, wherein: The method further comprises: If the area change between the pupil area in the reference image and the pupil area in the target image is not greater than the first preset area difference, and / or the motion information corresponding to the target image is not greater than the first preset difference, the scanning device is controlled to move from the first scanning position to the third scanning position according to the motion information, and the scanning device is controlled to obtain scanning data of the eye to be tracked at the third scanning position.

14. The method according to any one of claims 1 to 13, wherein: The method further comprises: If the area change between the pupil area in the first image and the pupil area in the second image is greater than a second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is greater than a second preset difference, then the scanning data acquired by the scanning device at the second scanning position is discarded, and the scanning device is controlled to reacquire the scanning data of the eye to be tracked at the second scanning position; the second scanning position includes a scanning position overlapping with the acquisition time of the second image; The first image and the second image are target images at two different time points, and the time point of the first image is earlier than the time point of the second image.

15. The method according to claim 14, wherein: The method further comprises: If the change in area between the pupil area in the first image and the pupil area in the second image is not greater than the second preset area difference, and / or the difference between the motion information corresponding to the second image and the motion information corresponding to the first image is not greater than the second preset difference, then the scanning device is controlled to move from the second scanning position to the third scanning position according to the motion information, and the scanning device is controlled to obtain scanning data of the eye to be tracked at the third scanning position.

16. The method according to any one of claims 1 to 15, wherein: The first segmentation result is in a form of at least one of a probability function and a classification image; the second segmentation result is in a form of at least one of a probability function and a classification image.

17. A tracking device, wherein: The device comprises: A first determination module is used to determine a first segmentation result of a reference image and a second segmentation result of a target image; the reference image and the target image are images of an eyeball to be tracked acquired by a visual sensor; the first segmentation result and the second segmentation result both include an iris area and a pupil area; A second determination module, configured to determine the displacement information of the eye to be tracked according to the first segmentation result and the second segmentation result; The tracking module is used to track the eyeball to be tracked according to the displacement information of the eyeball to be tracked.

18. A tracking system, wherein: The tracking system includes a visual sensor, a scanning device and a tracking device; the visual sensor includes a pupil camera; the scanning device includes OCT; The tracking device is used to execute the method according to any one of claims 1-16.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images