Light spot labeling method and system

Abstract

The application discloses a kind of light spot marking method and system, the light spot marking method includes: obtaining original image containing human eye;The pupil ellipse parameter of human eye in the original image is detected;Based on the pupil ellipse parameter, the number of each light spot is determined by the preset light spot estimation model, and the coarse position of each numbered light spot is estimated, and the light spot estimation model is a fully connected neural network model for outputting light spot position coordinates;The existence of each numbered light spot is judged by image processing process, and the image processing process at least includes clipping, threshold segmentation operation, corrosion operation, find contour operation;For the light spot that certainly exists, the light spot center coordinates are calculated according to coarse position.The application can solve the problem that the judgment of light spot existence in the prior art is not objective enough, it is difficult to accurately number light spot, the problem of insufficient labeling accuracy and low efficiency.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a method and system for marking light spots. Background Technology

[0002] Eye tracking (also known as eyeball tracking) is widely used not only in research fields such as psychology, transportation, and neuroscience, but also plays an important role in fields such as games, industrial simulation, and medical assistance. With the rapid development of virtual reality technology, eye tracking has gradually become one of the core technologies for interaction and foveated rendering in virtual reality. Eye tracking is achieved through eye tracking devices.

[0003] Eye-tracking algorithms that rely on spot information perform feature detection (including spot position and number information) on eye images captured by the camera, and then estimate the coordinates of the gaze point on the screen or the direction of the gaze.

[0004] Existing technologies mainly use manual annotation to obtain the location and numbering information of light spots, which has the following problems:

[0005] 1) The judgment of the existence of light spots is not objective enough; the size, shape and brightness of the candidate "light spots" vary. If the annotator judges whether the candidate "light spots" exist, there will be subjective bias. In particular, when different annotators annotate the same batch of light spot images, it is difficult to judge the existence of light spots with an absolutely uniform standard.

[0006] 2) Since the light emission properties of light sources with different numbers are the same, the size, shape, and brightness characteristics of the light spots cannot be used to distinguish them; in addition, light spots with similar numbers overlap in their positional variation range in the image. Therefore, it is difficult for annotators to accurately number the light spots simply by observing their characteristics and positions in the image.

[0007] 3) Manual annotation involves the coordination of human eyes and hands to annotate light spots, which is not accurate enough and has low efficiency. Summary of the Invention

[0008] To address these issues, one embodiment of the present invention proposes a spot marking method to solve the problems of insufficient objectivity in judging the existence of spot, difficulty in accurately numbering spot spots, insufficient marking accuracy, and low efficiency in the prior art.

[0009] A spot marking method according to an embodiment of the present invention includes:

[0010] Obtain the raw image containing the human eye;

[0011] Detect the pupil ellipse parameters of the human eye in the original image;

[0012] determine the number of each light spot and estimate the coarse position of each numbered light spot based on the pupil ellipse parameter through a preset light spot estimation model, the light spot estimation model being a fully connected neural network model for outputting light spot position coordinates;

[0013] judge the existence of each numbered light spot through an image processing process, the image processing process at least including cropping, threshold segmentation operation, erosion operation and contour finding operation;

[0014] For the light spot that does exist, calculate the light spot center coordinates thereof according to the coarse position.

[0015] According to the light spot labeling method, the number of each light spot is determined and the coarse position of each numbered light spot is estimated based on the pupil ellipse parameter through a preset light spot estimation model, the existence of each numbered light spot is then judged, that is, the existence of the light spot is judged through the image processing technology, the subjective interference of the human is excluded, the labeling efficiency is higher, the number of the light spot can be accurately determined through the light spot estimation model, and finally for the light spot that does exist, the light spot center coordinates thereof are calculated according to the coarse position, so that the labeling accuracy can be improved.

[0016] In addition, the light spot labeling method according to the above-mentioned embodiments of the present application can further have the following additional technical features:

[0017] Further, the light spot estimation model is obtained through the following steps:

[0018] Obtain pupil ellipse sample data and corresponding light spot position sample data, and make a data set according to the pupil ellipse sample data and the light spot position sample data;

[0019] Train the light spot estimation model based on the data set, the light spot estimation model being a fully connected neural network model, the input data being a vector [center, a_b, angle], the output data being a vector [x, y], wherein center is the center coordinates of the pupil ellipse, a_b is the semi-axis length of the pupil ellipse, angle is the angle of the short axis of the pupil ellipse rotating counterclockwise relative to the x-axis direction of the coordinate system, and [x, y] is the light spot position coordinates.

[0020] Further, judging the existence of each numbered light spot through the image processing process specifically includes:

[0021] For the target light spot of the target number, a rectangular image with a first preset size is cropped from the original image with the coarse position of the target light spot as the center;

[0022] Perform threshold segmentation operation on the rectangular image to obtain a first image;

[0023] performing an erosion operation on the first image to obtain a second image;

[0024] performing a contour finding operation on the second image to obtain a contour set;

[0025] if the contour set is not empty and an area of a contour with the largest area in the contour set is greater than a threshold value, determining that the target light spot exists; otherwise, determining that the target light spot does not exist.

[0026] Further, for the light spot that exists, calculating a light spot center coordinate according to the coarse position specifically comprises:

[0027] for the target light spot that exists, drawing a contour with the largest area in the contour set to a pure black background image with a second preset size and pixel values all being 0 to obtain an intermediate image;

[0028] calculating a centroid coordinate of the intermediate image;

[0029] calculating a light spot center coordinate in a coordinate system of the original image according to the centroid coordinate and the coarse position.

[0030] Further, the centroid coordinate of the intermediate image is calculated by using the following formula:

[0031]

[0032]

[0033] wherein, x c , y c are the centroid coordinates of the intermediate image, i is an x-axis pixel coordinate of the intermediate image, j is a y-axis pixel coordinate of the intermediate image, I ij is a pixel value corresponding to the pixel coordinate (i, j), m min , m max are start and end coordinates of summation calculation in the x-axis direction respectively, n min , n max are start and end coordinates of summation calculation in the y-axis direction respectively;

[0034] the light spot center coordinate in the coordinate system of the original image is calculated by using the following formula:

[0035] x p = x r + x c -w / 2

[0036] y p = y r + y c -h / 2

[0037] wherein, x px p x r x r x

[0038] Another embodiment of the present application provides a light spot labeling system to solve the problems of insufficient objectivity in judging the existence of light spots, difficulty in accurately numbering light spots, insufficient accuracy of labeling, and low efficiency in the prior art.

[0039] According to the light spot labeling system of an embodiment of the present application, the following technical effects can be achieved.

[0040] The acquisition module is configured to acquire an original image containing a human eye.

[0041] The detection module is configured to detect pupil ellipse parameters of the human eye in the original image.

[0042] The estimation module is configured to determine the number of each light spot and estimate the coarse position of each numbered light spot based on the pupil ellipse parameters through a preset light spot estimation model, wherein the light spot estimation model is a fully connected neural network model for outputting light spot position coordinates.

[0043] The judgment module is configured to judge the existence of each numbered light spot through an image processing process, wherein the image processing process at least includes cropping, threshold segmentation operation, erosion operation, and contour finding operation.

[0044] The calculation module is configured to calculate the light spot center coordinates of the light spot that actually exists according to the coarse position.

[0045] According to the light spot labeling system of the embodiment of the present application, the number of each light spot is determined and the coarse position of each numbered light spot is estimated based on the pupil ellipse parameters through a preset light spot estimation model, and then the existence of each numbered light spot is judged, that is, the judgment of the existence of light spots is realized through image processing technology, which excludes the subjective interference of human beings, has higher labeling efficiency, and can accurately determine the number of light spots through the light spot estimation model. Finally, the light spot center coordinates of the light spot that actually exists are calculated according to the coarse position, which can improve the accuracy of labeling.

[0046] In addition, the light spot labeling system according to the above-mentioned embodiments of the present application can have the following additional technical features.

[0047] Further, the light spot estimation model is obtained through the following steps:

[0048] Pupil ellipse sample data and corresponding light spot position sample data are acquired, and a data set is made according to the pupil ellipse sample data and the light spot position sample data.

[0049] Based on the data set, a light spot estimation model is trained, the light spot estimation model is a full connection neural network model, input data is a vector [center, a_b, angle], output data is a vector [x, y], wherein, center is the center coordinate of the pupil ellipse, a_b is the semi-axis length of the pupil ellipse, angle is the angle of the short axis of the pupil ellipse counterclockwise relative to the x-axis direction of the coordinate system, and [x, y] is the light spot position coordinate.

[0050] Further, the judging module is specifically used for:

[0051] For the target light spot with the target number, a rectangular image with a first preset size is cropped from the original image with the coarse position of the target light spot as the center;

[0052] A threshold segmentation operation is performed on the rectangular image to obtain a first image;

[0053] An erosion operation is performed on the first image to obtain a second image;

[0054] A contour finding operation is performed on the second image to obtain a contour set;

[0055] If the contour set is not an empty set, and the area of the largest contour in the contour set is greater than a threshold value, it is determined that the target light spot exists; otherwise, it is determined that the target light spot does not exist.

[0056] Further, the calculating module is specifically used for:

[0057] For the target light spot that exists, the largest contour in the contour set is drawn on a pure black background image with a second preset size and pixel value of 0 to obtain an intermediate image;

[0058] The centroid coordinates of the intermediate image are calculated;

[0059] According to the centroid coordinates and the coarse position, the light spot center coordinates in the coordinate system of the original image are calculated.

[0060] Further, the calculating module is specifically used for calculating the centroid coordinates of the intermediate image by the following formula:

[0061]

[0062]

[0063] Wherein, x c , y c are the centroid coordinates of the intermediate image, i is the x-axis pixel coordinate of the intermediate image, j is the y-axis pixel coordinate of the intermediate image, I ijis a pixel value corresponding to a pixel coordinate (i, j), m min , m max are start and end coordinates of summation calculation in x-axis direction, n min , n max are start and end coordinates of summation calculation in y-axis direction;

[0064] The spot center coordinates in the coordinate system of the original image are calculated by using the following formula:

[0065] x p = x r + x c - w / 2

[0066] y p = y r + y c - h / 2

[0067] wherein x p , y p are the spot center coordinates in the coordinate system of the original image, x r , y r are the coordinates of the coarse position of the target spot, and w, h are the width and height of the pure black background image respectively. BRIEF DESCRIPTION OF DRAWINGS

[0068] The above and / or additional aspects and advantages of the embodiments will become apparent and be readily understood from the following description, taken in conjunction with the drawings, in which:

[0069] Figure 1 is a flow chart of a spot labeling method according to an embodiment of the present application;

[0070] Figure 2 is an exemplary original image;

[0071] Figure 3 is an exemplary estimated coarse position of a spot;

[0072] Figure 4 is an exemplary final result of spot labeling;

[0073] Figure 5 is a structural schematic diagram of a spot labeling system according to another embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0075] Referring to Figure 1 The light spot labeling method provided by an embodiment of the present application comprises steps S101-S105.

[0076] S101, an original image containing a human eye is acquired.

[0077] The original image M0 containing a human eye can be acquired by photographing or manual inputting. When photographing, the human eye should be in an open-eye state, and the pupil is not blocked by the eyelid. An exemplary original image M0 is shown in FIG. 1. Figure 2

[0078] S102, pupil ellipse parameters of the human eye in the original image are detected.

[0079] The pupil ellipse parameters specifically can include a center coordinate, a semi-axis length and a rotation angle. The specific detection method can be a method based on traditional image processing or a pupil positioning method based on deep learning.

[0080] The method based on traditional image processing for detecting the pupil ellipse parameters of the human eye in the original image specifically comprises the following steps: coarse positioning of the pupil, and then sequentially performing steps of clipping, threshold segmentation, opening operation, contour finding, contour merging, convex hull operation and ellipse fitting, to finally obtain the ellipse parameter information.

[0081] The pupil positioning method based on deep learning for detecting the pupil ellipse parameters of the human eye in the original image specifically comprises the following steps: acquiring an eye map, labeling pupil position information, making a pupil positioning data set, training a pupil positioning neural network, and finally performing pupil positioning on a new eye map by using the trained pupil positioning neural network.

[0082] S103, based on the pupil ellipse parameters, the number of each light spot is determined by a preset light spot estimation model, and the coarse position of each numbered light spot is estimated, the light spot estimation model being a fully connected neural network model for outputting light spot position coordinates.

[0083] ​wherein, based on the pupil ellipse parameters, the coarse positions of the numbered glints are estimated (at this time, it is assumed that the glints all exist). The glint numbering and the coarse position estimation of the glints are realized by a glint estimation model, the input of the model being the pupil ellipse parameters, and the output being the coarse position coordinates [(x1, y1), (x2, y2),..., (xn, yn)] of the numbered glints. Wherein, (x1, y1), (x2, y2), (xn, yn) are the normalized coordinates of the glints numbered 1, 2, n in the original image M0, respectively. An exemplary estimated coarse position of the glints is shown in FIG. 2. Figure 3 Figure 3 There are a total of 1-8 numbered glints in FIG. 2.

[0084] The steps for establishing the glint estimation model include:

[0085] Pupil ellipse sample data and corresponding glint position sample data are obtained, and a data set is made according to the pupil ellipse sample data and the glint position sample data, wherein the pupil ellipse sample data and the corresponding glint position sample data can be represented as {pupil ellipse sample data, glint position sample data}, and can be obtained by simulation. The method is: establishing an eyeball, camera, and light source model; adjusting the position and posture of the eyeball to obtain pupil contour data and glint position data at different eyeball poses. Specifically, based on the light source position, camera position, sensor parameters, and eyeball parameters, a 3D eyeball model is constructed. In the 3D eyeball model, the eyeball and cornea are set as spherical surfaces, and the mechanical center of the eyeball is fixed. In the forward 3D eyeball model, the kappa angle is ignored, and the optical axis is considered as the visual axis. According to N gaze points on the screen of the image acquisition device and the default mechanical center position of the eyeball, N optical axis data can be obtained, and then combined with the already established 3D eyeball model, the imaging positions of the glint and pupil ellipse key points in the camera can be obtained through the laws of reflection and refraction of light.

[0086] Based on the data set, a glint estimation model is trained, the glint estimation model being a fully connected neural network model, the input data being a vector [center, a_b, angle], the output data being a vector [x, y], wherein center is the center coordinate of the pupil ellipse, a_b is the semi-axis length of the pupil ellipse, angle is the angle of the short axis of the pupil ellipse rotating counterclockwise relative to the x-axis direction of the coordinate system, and [x, y] is the glint position coordinate.

[0087] The glint estimation model is trained by a large number of {pupil ellipse sample data, glint position sample data}, and for each image, the corresponding coarse position and number of glints can be given.

[0088] In S104, the existence of each numbered glint is determined by an image processing process, which at least includes cropping, threshold segmentation operation, erosion operation, and contour finding operation. ​

[0089] Specifically, for a target spot numbered as a target number, a rectangular image with a first preset size, for example, a rectangular image with a width of 40 pixels and a height of 40 pixels, is cropped from the original image, with a coarse position of the target spot as a center;

[0090] A threshold segmentation operation is performed on the rectangular image to obtain a first image. In a specific implementation, a threshold segmentation process is as follows: pixel values of pixel points belonging to a spot are set to 255, and pixel values of other pixel points are set to 0. A dynamic threshold segmentation method is used, and specific steps include: a 21x21 mean filter is used to process the original image to obtain a reference image; a value g of each pixel in the original image is compared with a value r of a corresponding pixel in the reference image, if g-r>40, the value of the pixel in the original image is set to 255, otherwise, the value is set to 0;

[0091] An erosion operation is performed on the first image to obtain a second image, and the erosion operation eliminates miscellaneous points in the image;

[0092] A contour finding operation is performed on the second image to obtain a contour set;

[0093] If the contour set is not an empty set, and an area of a largest contour in the contour set is greater than a threshold, it is determined that the target spot indeed exists; otherwise, it is determined that the target spot does not exist.

[0094] In this way, existence of each numbered spot can be determined.

[0095] S105, for the spot that indeed exists, a spot center coordinate of the spot is calculated according to the coarse position.

[0096] Specifically, for the target spot that indeed exists, a largest contour in the contour set is drawn on a pure black background image with a second preset size and pixel values of 0 to obtain an intermediate image.

[0097] A centroid coordinate of the intermediate image is calculated.

[0098] In this embodiment, the centroid coordinate of the intermediate image is calculated by using the following formula:

[0099]

[0100]

[0101] wherein, x c , y c are the centroid coordinates of the intermediate image, specifically, the spot center coordinates in an intermediate image coordinate system; i is an x-axis pixel coordinate of the intermediate image, j is a y-axis pixel coordinate of the intermediate image, and I ijpixel value corresponding to pixel coordinate (i, j), m min , m max start and end coordinates of summation calculation in x-axis direction, n min , n max start and end coordinates of summation calculation in y-axis direction;

[0102] According to the centroid coordinates and the coarse position, calculate the spot center coordinates in the coordinate system of the original image.

[0103] In this embodiment, the spot center coordinates in the coordinate system of the original image are calculated by the following formula:

[0104] x p = x r + x c -w / 2

[0105] y p = y r + y c -h / 2

[0106] wherein x p , y p are the spot center coordinates in the coordinate system of the original image, x r , y r are the coordinates of the coarse position of the target spot, and w, h are the width and height of the pure black background image, respectively.

[0107] An exemplary final result of spot labeling is shown in Figure 4 Compared with Figure 3 , in Figure 4 , the 1st, 6th, 7th and 8th spots do not actually exist.

[0108] In summary, according to the spot labeling method of the present embodiment, based on the pupil ellipse parameters, the number of each spot is determined through a preset spot estimation model, and the coarse position of each numbered spot is estimated, and then the existence of each numbered spot is judged, that is, the existence of the spot is judged through image processing technology, which eliminates the subjective interference of human beings and is more efficient. The number of the spot can also be accurately determined through the spot estimation model, and finally for the spot that actually exists, the spot center coordinates are calculated according to the coarse position, which can improve the accuracy of labeling.

[0109] Please refer to Figure 5 , another embodiment of the present application provides a spot labeling system, which comprises:

[0110] An acquisition module is configured to acquire an original image containing a human eye.

[0111] A detection module is configured to detect the pupil ellipse parameters of the human eye in the original image.

[0112] an estimation module configured to determine a number of each light spot and estimate a coarse position of each numbered light spot based on the pupil ellipse parameter and through a preset light spot estimation model, the light spot estimation model being a fully connected neural network model for outputting light spot position coordinates;

[0113] a judgment module configured to judge existence of each numbered light spot through an image processing process, the image processing process at least including cropping, threshold segmentation operation, erosion operation and contour finding operation;

[0114] a calculation module configured to calculate a light spot center coordinate of a light spot that is determined to exist according to the coarse position.

[0115] In the embodiment, the light spot estimation model is obtained through the following steps:

[0116] obtaining pupil ellipse sample data and corresponding light spot position sample data, and making a data set according to the pupil ellipse sample data and the light spot position sample data;

[0117] training a light spot estimation model based on the data set, the light spot estimation model being a fully connected neural network model, input data being a vector [center, a_b, angle], output data being a vector [x, y], wherein center is a center coordinate of the pupil ellipse, a_b is a semi-axis length of the pupil ellipse, angle is an angle of a short axis of the pupil ellipse relative to a direction of an x-axis of a coordinate system counterclockwise, and [x, y] is a light spot position coordinate.

[0118] In the embodiment, the judgment module is specifically configured to:

[0119] cropping a rectangular image with a first preset size from the original image, with a coarse position of a target light spot as a center, the target light spot being a target numbered light spot;

[0120] performing threshold segmentation operation on the rectangular image to obtain a first image;

[0121] performing erosion operation on the first image to obtain a second image;

[0122] performing contour finding operation on the second image to obtain a contour set;

[0123] if the contour set is not an empty set and an area of a contour with a largest area in the contour set is greater than a threshold value, it is determined that the target light spot exists; otherwise, it is determined that the target light spot does not exist.

[0124] In the embodiment, the calculation module is specifically configured to:

[0125] For the target light spot that does exist, the largest profile in the profile set is drawn on a pure black background image with a second preset size and pixel values of 0 to obtain an intermediate image;

[0126] The centroid coordinates of the intermediate image are calculated;

[0127] According to the centroid coordinates and the coarse position, the light spot center coordinates in the coordinate system of the original image are calculated.

[0128] In the embodiment, the calculation module is specifically configured to calculate the centroid coordinates of the intermediate image by using the following formula:

[0129]

[0130]

[0131] wherein x c , y c are the centroid coordinates of the intermediate image, i is the x-axis pixel coordinate of the intermediate image, j is the y-axis pixel coordinate of the intermediate image, I ij is the pixel value corresponding to the pixel coordinate (i, j), m min , m max are the start and end coordinates of the summation calculation in the x-axis direction, n min , n max are the start and end coordinates of the summation calculation in the y-axis direction.

[0132] The light spot center coordinates in the coordinate system of the original image are calculated by using the following formula:

[0133] x p = x r + x c - w / 2

[0134] y p = y r + y c - h / 2

[0135] wherein x p , y p are the light spot center coordinates in the coordinate system of the original image, x r , y r are the coordinates of the coarse position of the target light spot, and w, h are the width and height of the pure black background image, respectively.

[0136] According to the light spot labeling system of the embodiment, based on the pupil ellipse parameters, the numbering of each light spot is determined through a preset light spot estimation model, the coarse position of each numbered light spot is estimated, and the existence of each numbered light spot is judged, that is, the existence of the light spot is judged through the image processing technology, the subjective interference of the human is excluded, the labeling efficiency is higher, the numbering of the light spot can be accurately determined through the light spot estimation model, and finally for the light spot that actually exists, the light spot center coordinates are calculated according to the coarse position, and the labeling accuracy can be improved.

[0137] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of both. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0138] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber (optical devices), and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a manner that is suitable for use by the instruction execution system, apparatus, or device.

[0139] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0140] In the description of the specification, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the above expressions in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0141] Although embodiments of the application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application, which is defined by the following claims and their equivalents.

Claims

1. A method for annotating a light spot, characterized in that, The method comprises: acquiring an original image containing a human eye; detecting a pupil ellipse parameter of the human eye in the original image; based on the pupil ellipse parameter, determining the number of each glint and estimating the coarse position of each numbered glint through a preset glint estimation model, the glint estimation model being a fully connected neural network model for outputting glint position coordinates; judging the existence of each numbered glint through an image processing process, the image processing process at least including cropping, threshold segmentation operation, erosion operation and contour finding operation; for the glint that exists, calculating the glint center coordinates according to the coarse position.

2. The light spot annotation method of claim 1, wherein, The glint estimation model is obtained through the following steps: acquiring pupil ellipse sample data and corresponding glint position sample data, and making a data set according to the pupil ellipse sample data and the glint position sample data; based on the data set, training the glint estimation model, the glint estimation model being a fully connected neural network model, the input data being a vector [center, a_b, angle], the output data being a vector [x, y], wherein center is the center coordinates of the pupil ellipse, a_b is the semi-axis length of the pupil ellipse, angle is the angle of the short axis of the pupil ellipse relative to the x-axis direction of the coordinate system counterclockwise, and [x, y] is the glint position coordinates.

3. The light spot annotation method of claim 1, wherein, The judgment of the existence of each numbered glint through the image processing process specifically includes: for a target glint with a target number, cropping a rectangular image with a first preset size from the original image with the coarse position of the target glint as the center; performing threshold segmentation operation on the rectangular image to obtain a first image; performing erosion operation on the first image to obtain a second image; performing contour finding operation on the second image to obtain a contour set; if the contour set is not an empty set and the area of the largest contour in the contour set is greater than a threshold, it is determined that the target glint exists; otherwise, it is determined that the target glint does not exist.

4. The light spot annotation method of claim 3, wherein, For the glint that exists, calculating the glint center coordinates according to the coarse position specifically includes: for the target glint that exists, drawing the largest contour in the contour set to a pure black background image with a second preset size and pixel value of 0 to obtain an intermediate image; calculating the centroid coordinates of the intermediate image; calculating the glint center coordinates in the coordinate system of the original image according to the centroid coordinates and the coarse position.

5. The light spot annotation method of claim 4, wherein The centroid coordinates of the intermediate image are calculated by the following formula: wherein x c , y c are the centroid coordinates of the intermediate image, i is the x-axis pixel coordinate of the intermediate image, j is the y-axis pixel coordinate of the intermediate image, I ij is the pixel value corresponding to the pixel coordinate (i, j), m min , m max are the start and end coordinates of the summation calculation in the x-axis direction, respectively, n min , n max are the start and end coordinates of the summation calculation in the y-axis direction, respectively. The glint center coordinates in the coordinate system of the original image are calculated by the following formula: x p = x r + x c - w / 2 y p = y r + y c - h / 2 wherein x p , y p are the coordinates of the center of the spot in the coordinate system of the original image, x r , y r are the coordinates of the coarse position of the target spot, and w, h are the width and height of the pure black background image, respectively.

6. A light spot annotation system, characterized by The system comprises: an acquisition module for acquiring an original image containing a human eye; a detection module for detecting a pupil ellipse parameter of the human eye in the original image; an estimation module for determining the number of each glint and estimating the coarse position of each numbered glint based on the pupil ellipse parameter through a preset glint estimation model, the glint estimation model being a fully connected neural network model for outputting glint position coordinates; a judging module configured to determine existence of each numbered light spot through an image processing procedure, the image processing procedure comprising at least clipping, threshold segmentation operation, erosion operation, and contour finding operation; a calculating module configured to calculate light spot center coordinates of a light spot that is determined to exist according to the coarse position.

7. The light spot annotation system of claim 6, wherein, The light spot estimation model is established through the following steps: obtain pupil ellipse sample data and corresponding light spot position sample data, and make a data set according to the pupil ellipse sample data and the light spot position sample data; train a light spot estimation model based on the data set, the light spot estimation model being a full connection neural network model, input data being a vector [center, a_b, angle], output data being a vector [x, y], wherein center is a center coordinate of a pupil ellipse, a_b is a semi-axis length of the pupil ellipse, angle is an angle of a short axis of the pupil ellipse relative to a direction of an x-axis of a coordinate system counterclockwise, and [x, y] is a light spot position coordinate.

8. The light spot annotation system of claim 7, wherein, The judging module is specifically configured to: clip a rectangular image with a first preset size from the original image with the coarse position of a target light spot as a center, the target light spot being a target numbered light spot; perform threshold segmentation operation on the rectangular image to obtain a first image; perform erosion operation on the first image to obtain a second image; perform contour finding operation on the second image to obtain a contour set; if the contour set is not an empty set and an area of a largest contour in the contour set is greater than a threshold, it is determined that the target light spot exists; otherwise, it is determined that the target light spot does not exist.

9. The light spot annotation system of claim 8, wherein, The calculating module is specifically configured to: for a target light spot that is determined to exist, draw a largest contour in the contour set to a pure black background image with a second preset size and pixel values of 0 to obtain an intermediate image; calculate a centroid coordinate of the intermediate image; calculate light spot center coordinates in a coordinate system of the original image according to the centroid coordinate and the coarse position.

10. The light spot annotation system of claim 9, wherein, The calculating module is specifically configured to calculate the centroid coordinate of the intermediate image using the following formula: wherein x c , y c are the centroid coordinates of the intermediate image, i is the x-axis pixel coordinate of the intermediate image, j is the y-axis pixel coordinate of the intermediate image, I ij is the pixel value corresponding to the pixel coordinate (i, j), m min , m max are the start and end coordinates of the summation calculation in the x-axis direction, n min , n max are the start and end coordinates of the summation calculation in the y-axis direction. calculate the light spot center coordinates in the coordinate system of the original image using the following formula: x p = x r + x c - w / 2 y p = y r + y c - h / 2 wherein x p , y p are the coordinates of the center of the spot in the coordinate system of the original image, x r , y r are the coordinates of the coarse position of the target spot, and w, h are the width and height of the pure black background image, respectively.

Images