Eye image processing method and device, eye movement tracking system and electronic device

Abstract

The application relates to an eye image processing method and device, an eye movement tracking system and an electronic device, wherein the eye image processing method comprises the following steps: acquiring eye image information; performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a coarse pupil positioning image; performing edge detection processing on the first preprocessed image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the coarse pupil positioning image; and generating an eye movement feature extraction result for the eye image information according to the target pupil positioning result. Through the application, the problems of low accuracy and low efficiency of eye image processing are solved, and efficient and accurate eye image processing is realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to eye image processing methods, apparatus, eye-tracking systems and electronic devices. Background Technology

[0002] Eye tracking, also known as gaze tracking, is a technique that uses camera equipment to observe the movement of a person's eyes in real time and estimates the direction and coordinates of the gaze's focal point using certain methods. The core data of eye tracking algorithms are the positional information of the pupil and the light spot; for example, quantitative information about eye fixation movement is estimated by calculating the relative offset of the pupil.

[0003] However, the inherent physiological mechanisms of the human eye and the nonlinearity, randomness, and complexity of eye movements increase the difficulty of pupil center localization. Related technologies typically rely on template matching and other methods to quickly locate the pupil region, but this results in low accuracy, and the template parameters cannot adapt to individual pupil size variations, affecting the accuracy of pupil localization. Alternatively, deep learning-based object detection methods can be used to locate the pupil, but obtaining accurate pupil edge information requires fine-tuning based on the object detection results. If the pupil localization task is viewed as object segmentation, a large number of sample annotations are needed, which are difficult and inefficient, leading to low accuracy and efficiency in eye image processing.

[0004] Currently, no effective solution has been proposed to address the issues of low accuracy and efficiency in eye image processing within related technologies. Summary of the Invention

[0005] This application provides an eye image processing method, apparatus, eye tracking system, and electronic device to at least address the problems of low accuracy and efficiency in eye image processing in related technologies.

[0006] In a first aspect, embodiments of this application provide an eye image processing method, the method comprising:

[0007] Acquire eye image information;

[0008] The eye image information is debinarized to obtain a first preprocessed image, and the first preprocessed image is subjected to contour extraction to obtain a coarse pupil localization image.

[0009] The first preprocessed image is subjected to edge detection processing to obtain an edge image, and the target pupil localization result is obtained based on the edge image and the pupil coarse localization image.

[0010] Based at least on the target pupil localization result, an eye movement feature extraction result is generated for the eye image information.

[0011] In some embodiments, the step of performing debinarization on the eye image information to obtain the first preprocessed image includes:

[0012] An integral image is calculated based on the eye image information. Haar features are calculated based on the integral image. The maximum Haar response value and Haar response radius are obtained based on the Haar features. The first region of interest image is obtained based on the maximum Haar response value and Haar response radius.

[0013] The first region of interest image is debinarized to obtain the first preprocessed image.

[0014] In some embodiments, the step of performing contour extraction processing on the first preprocessed image to obtain a coarse pupil localization image includes:

[0015] The first preprocessed image is subjected to contour extraction processing to obtain a first optimal contour, and an eyelid positioning sub-image of the eye image information is determined based on the first optimal contour; a pixel mean image is calculated based on the eyelid positioning sub-image, and the gradient information of the pixel mean image is calculated.

[0016] The pixel mean image is clustered using a clustering algorithm to obtain clustering results, and the skin color brightness threshold is obtained based on the clustering results;

[0017] The pixel values ​​obtained by traversing the pixel mean image are compared with the skin brightness threshold, and the target eyelid localization result is determined based on the comparison result and the gradient information.

[0018] If the eyelid distance in the target eyelid positioning result is greater than a preset eyelid distance threshold, the pupil coarse positioning image is obtained based on the eyelid positioning sub-image; if the eyelid distance is less than or equal to the preset eyelid distance threshold, the blink determination result is obtained.

[0019] In some embodiments, obtaining the coarse pupil localization image based on the eyelid localization sub-image includes:

[0020] The eyelid positioning sub-image is debinarized to obtain a second preprocessed image, and the second preprocessed image is subjected to contour extraction to obtain a second optimal contour.

[0021] Based on the first optimal contour, the pupil center position and pupil radius information are obtained, and the second optimal contour is filtered according to the pupil center position and pupil radius information to obtain the first contour point set;

[0022] If the number of the first contour point set is greater than the preset number of contour points, the first contour point set is iteratively fitted to obtain a fitted circle result; the pupil contour point result is obtained based on the first contour point set and the fitted circle result, and the pupil coarse positioning image is obtained based on the pupil contour point result.

[0023] When the number of contour points is less than or equal to the preset number of contour points, the blink determination result is obtained.

[0024] In some embodiments, obtaining the pupil contour point result based on the first contour point set and the fitted circle result includes:

[0025] The fitted circle results are then scaled up and scaled down to obtain the annular template results.

[0026] The pupil contour point result is obtained based on the first contour point set and the circular template result.

[0027] In some embodiments, obtaining the target pupil localization result based on the edge image and the pupil coarse localization image includes:

[0028] The edge image and the coarse pupil localization image are enlarged proportionally and subjected to an AND operation to obtain a first overlapping point set. The first overlapping point set is then fitted to obtain a first pupil fitting result.

[0029] The edge image and the first pupil fitting result are ANDed to obtain a second overlapping point set. The second overlapping point set is fitted to obtain a second pupil fitting result. The second pupil fitting result is then highlighted and filtered to obtain a second contour point set.

[0030] The gradient direction results of the second contour point set traversed are calculated sequentially. The second contour point set is then filtered based on the gradient direction results to obtain a third contour point set. The target pupil positioning result is then generated based on the third contour point set.

[0031] In some embodiments, generating eye movement feature extraction results for the eye image information based at least on the target pupil localization result includes:

[0032] The topological vector information of the supplementary light source is obtained, and the supplementary light source is used to provide supplementary lighting during the acquisition of the eye image information;

[0033] A second region of interest image is extracted from the coarse pupil localization image. A contour extraction process is performed on the second region of interest image to obtain a contour sequence. A bright spot contour sequence is obtained based on the contour sequence.

[0034] The target spot localization result is obtained based on the topological vector information and the bright spot contour sequence, and the eye movement feature extraction result is generated based on the target pupil localization result and the target spot localization result.

[0035] In some embodiments, obtaining the target spot localization result based on the topology vector information and the bright spot contour sequence includes:

[0036] The bright spot contour points in the traversed bright spot contour sequence are used as reference points in turn, and the vector information between other bright spot contour points in the bright spot contour sequence and the reference points is calculated.

[0037] A template vector is obtained based on the topological vector information and the vector information. A light source spot sequence is obtained by combining the topological vector information and the bright spot contour sequence. A corneal reflection spot sequence is obtained by matching the light source spot sequence and the template vector.

[0038] Traverse the corneal reflective spot sequence to obtain the third region of interest image corresponding to each corneal reflective spot in the sequence, and perform contour extraction processing on each third region of interest image to obtain the target spot;

[0039] The target spot localization result is obtained based on the target spot.

[0040] In some embodiments, the edge detection processing of the first preprocessed image to obtain an edge image includes:

[0041] The first preprocessed image is segmented using the Otsu's algorithm to obtain a segmentation threshold, and edge detection is performed on the first preprocessed image based on the segmentation threshold to obtain the edge image.

[0042] Secondly, embodiments of this application provide an eye image processing device, the device comprising: an acquisition module, a coarse positioning module, a target module, and a generation module;

[0043] The acquisition module is used to acquire eye image information;

[0044] The coarse positioning module is used to perform inverse binarization processing on the eye image information to obtain a first preprocessed image, and to perform contour extraction processing on the first preprocessed image to obtain a pupil coarse positioning image.

[0045] The target module is used to perform edge detection processing on the first preprocessed image to obtain an edge image, and to obtain a target pupil localization result based on the edge image and the pupil coarse localization image.

[0046] The generation module is used to generate eye movement feature extraction results for the eye image information based at least on the target pupil localization result.

[0047] Thirdly, embodiments of this application provide an eye-tracking system, which includes an image acquisition device and a control device;

[0048] The image acquisition device is used to acquire eye image information;

[0049] The control device is configured to perform the eye image processing method as described in the first aspect above on the eye image information to obtain the eye movement feature extraction result, and generate an eye movement tracking result based on the eye movement feature extraction result.

[0050] Fourthly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the eye image processing method as described in the first aspect above.

[0051] Fifthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the eye image processing method as described in the first aspect above.

[0052] Compared to related technologies, the eye image processing method, apparatus, eye-tracking system, and electronic device provided in this application, by acquiring eye image information; performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a coarse pupil localization image; performing edge detection processing on the first preprocessed image to obtain an edge image, and obtaining a target pupil localization result based on the edge image and the coarse pupil localization image; and generating an eye movement feature extraction result for the eye image information based at least on the target pupil localization result, solves the problems of low accuracy and efficiency in eye image processing, and achieves efficient and accurate eye image processing.

[0053] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is an application environment diagram of an eye image processing method according to an embodiment of this application;

[0056] Figure 2 This is a flowchart of an eye image processing method according to an embodiment of this application;

[0057] Figure 3 This is a flowchart of another eye image processing method according to an embodiment of this application;

[0058] Figure 4A This is a schematic diagram of an eyelid positioning sub-image according to an embodiment of this application;

[0059] Figure 4B This is a schematic diagram of a target eyelid localization result according to an embodiment of this application;

[0060] Figure 5 This is a flowchart of an eyelid localization algorithm according to an embodiment of this application;

[0061] Figure 6A This is a schematic diagram of a third contour point set according to an embodiment of this application;

[0062] Figure 6B This is a schematic diagram of a target pupil localization result according to an embodiment of this application;

[0063] Figure 7 This is a flowchart of a pupil localization algorithm according to an embodiment of this application;

[0064] Figure 8 This is a flowchart of another eye image processing method according to an embodiment of this application;

[0065] Figure 9 This is a schematic diagram of a target spot positioning result according to an embodiment of this application;

[0066] Figure 10 This is a flowchart of a corneal reflective spot localization algorithm according to an embodiment of this application;

[0067] Figure 11 This is a flowchart of an eye image processing method according to a preferred embodiment of this application;

[0068] Figure 12 This is a structural block diagram of an eye image processing device according to an embodiment of this application;

[0069] Figure 13 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0071] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0072] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0073] The eye image processing method provided in this application can be applied to, for example... Figure 1In the application environment shown, the image acquisition device 12 communicates with the server device 14 via a network. The server device 14 acquires eye image information through the image acquisition device 12, performs debinarization processing on the eye image information to obtain a first preprocessed image, performs contour extraction processing on the first preprocessed image to obtain a coarse pupil localization image, and performs edge detection processing on the first preprocessed image to obtain an edge image. Based on the edge image and the coarse pupil localization image, a target pupil localization result is obtained, and at least based on the target pupil localization result, an eye movement feature extraction result is generated for the eye image information. The image acquisition device 12 can be, but is not limited to, various binocular cameras, PTZ cameras, or other devices used for image acquisition, and the server device 14 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0074] This embodiment provides a method for processing eye images. Figure 2 This is a flowchart of an eye image processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0075] Step S220: Obtain eye image information.

[0076] The aforementioned eye image information refers to near-eye images to be processed, such as pupil localization; preferably, the eye image information can be infrared near-eye images. This eye image information can be obtained by extracting consecutive frame images from the video stream acquired by the aforementioned image acquisition device using the aforementioned server equipment or processing chip and other control devices.

[0077] Step S240: Perform inverse binarization processing on the eye image information to obtain a first preprocessed image, and perform contour extraction processing on the first preprocessed image to obtain a coarse pupil localization image.

[0078] Specifically, in step S240 above, the extracted pupil region image can first be binarized to obtain a preprocessed pupil region image. The purpose of this is to extract the pupil contour, preparing for subsequent image processing steps such as pupil center localization. Furthermore, to improve the accuracy and efficiency of eye image processing, the control device can also perform scaling, sub-image extraction, or filtering on the eye image information, ultimately obtaining the first preprocessed image. Then, the control device extracts the contours of connected regions in the first preprocessed image to obtain a set of contour points for the pupil, thereby obtaining the coarse pupil localization image.

[0079] Step S260: Perform edge detection processing on the first preprocessed image to obtain an edge image, and obtain the target pupil localization result based on the edge image and the pupil coarse localization image.

[0080] Specifically, the control device can use an edge extraction algorithm to detect the pupil contour in the first preprocessed image. For example, the grayscale value of the pixels in the first preprocessed image is set to 0 or 255. All pixels with a grayscale value not less than a selected threshold are considered to be the target pupil region, and their grayscale value is represented by 255. Otherwise, the grayscale value is 0, representing the background or other object region, thereby extracting the edge image. Then, the control device performs an AND operation on the edge image and the coarse pupil localization image to obtain a set of overlapping points. Based on this set of overlapping points, ellipse fitting and other processing are performed to finally obtain the target pupil localization result.

[0081] In some embodiments, the above-mentioned edge detection processing of the first preprocessed image to obtain an edge image further includes the following steps: performing image segmentation processing on the first preprocessed image using the OTSU algorithm to obtain a segmentation threshold, and performing edge detection on the first preprocessed image according to the segmentation threshold to obtain the edge image.

[0082] Step S280: At least based on the target pupil localization result, generate eye movement feature extraction results for the eye image information.

[0083] Among them, the above-mentioned eye movement feature extraction results are data analysis results reflecting eye movement behavior; based on the above-mentioned target pupil localization results, the eye movement feature extraction results including the movement trajectory, orientation and other data of eye movement behavior are determined, thereby enabling human-computer interaction based on eye movement control.

[0084] Through steps S220 to S280, a first preprocessed image is obtained by inverse binarizing the eye image information. A coarse pupil localization image is obtained based on the first preprocessed image. The target pupil localization result is obtained based on the coarse pupil localization image and the edge image obtained by edge detection. This realizes an eye image processing method that obtains the pupil region by binarizing the image and locates the pupil by combining edge information. The algorithm is simple and computationally efficient. At the same time, the accuracy of pupil detection is improved by two pupil localizations, thus solving the problems of low accuracy and efficiency in eye image processing and realizing an efficient and accurate eye image processing method.

[0085] In some embodiments, the above-described debinarization of the eye image information to obtain the first preprocessed image further includes the following steps:

[0086] Step S241: Calculate an integral image based on the eye image information, calculate Haar features based on the integral image, obtain the maximum Haar response value and Haar response radius based on the Haar features, and obtain the first region of interest image based on the maximum Haar response value and Haar response radius.

[0087] Specifically, the control device calculates an integral image of the eye image information and uses this integral image to find the location with the largest Haar response value; the largest Haar response value indicates that the image information at that location is most similar to a preset Haar template. Then, the control terminal uses the location with the largest Haar response value as the center and twice the Haar response radius as the side length to extract the initial pupil localization sub-image from the eye image information, which is the first region of interest image roiImage.

[0088] Step S242: Perform inverse binarization on the first region of interest image to obtain the first preprocessed image.

[0089] Specifically, the first region of interest image, i.e., the roiImage, is first subjected to inverse binarization, so that the low-brightness areas of the roiImage are displayed as white and the high-brightness areas are displayed as black. Morphological filtering is then performed to eliminate isolated noise points, and finally the first preprocessed image is obtained.

[0090] Through the above steps S241 to S242, the region of interest image is obtained through the Haar response, and the obtained region of interest image is debinarized to obtain the first preprocessed image, thereby realizing the preprocessing before eye image processing, which can help improve the accuracy of eye image processing.

[0091] In some embodiments, an eye image processing method is provided. Figure 3 This is a flowchart of another eye image processing method according to an embodiment of this application, such as... Figure 3 As shown, the process includes Figure 2 The steps S220, S260, and S280 shown herein also include the following steps:

[0092] Step S320: The eye image information is debinarized to obtain a first preprocessed image, and the first preprocessed image is contour extracted to obtain a first optimal contour. Based on the first optimal contour, the eyelid positioning sub-image of the eye image information is determined. The pixel mean image is calculated based on the eyelid positioning sub-image, and the gradient information of the pixel mean image is calculated.

[0093] After obtaining the first preprocessed image through step S320 or steps S241 to S242, the control device can perform contour extraction processing on the connected regions in the first preprocessed image. Based on the prior information that the pupil shape is approximately circular and the pupil region accounts for the largest proportion in the first preprocessed image, the first optimal contour is obtained. Based on the first optimal contour, the control device can roughly calculate the pupil center position and pupil radius information. Based on the pupil center position, the control device defaults to taking the x-coordinate of the pupil center position as the x-coordinate of the upper and lower eyelid centers, thereby using the minimum bounding rectangle to extract the eyelid positioning sub-image in the eye image information to achieve vertical positioning. Specifically, Figure 4A The area marked by the black outline is the extracted eyelid localization sub-image.

[0094] After obtaining the above-mentioned eyelid positioning sub-image, the control device can first filter the eyelid positioning sub-image to improve the accuracy of image processing, and then statistically analyze the average pixel value avgImage of each row in the processed eyelid positioning sub-image to obtain the pixel average image, and calculate the gradient information gradient of the pixel average image in the y direction.

[0095] Step S340: Cluster the pixel mean image using a clustering algorithm to obtain clustering results, and obtain the skin color brightness threshold based on the clustering results.

[0096] The clustering algorithm mentioned above can be the k-means clustering algorithm. Its steps are: randomly select K pixels as initial cluster centers; calculate the distance between each pixel and each initial cluster center; and assign each pixel to the nearest initial cluster center. Each initial cluster center and the pixels assigned to it represent a cluster. For each assigned sample, the cluster centers are recalculated based on the existing pixels in that cluster. This process is repeated until no (or a minimum number) pixels are reassigned to different clusters, or no (or a minimum number) cluster centers change, thus finally generating the classification results for all pixels in the pixel mean image. This clustering algorithm can also be DBSCAN (Density-Based Spatial Clustering of Applications with Noise), etc., which will not be elaborated upon here.

[0097] The aforementioned clustering algorithm can then divide the pixel mean image into different skin color regions, such as the periorbital region, pupil region, and iris region, based on the clustering results of the pixels. The control device can then determine the pixel values ​​of the upper and lower eyelids based on all the divided skin color regions, and use these pixel values ​​as the skin color brightness threshold to determine the eyelid positions in subsequent steps of the eye image information.

[0098] Step S360: Compare the pixel values ​​obtained by traversing the pixel mean image with the skin brightness threshold, and determine the target eyelid localization result based on the comparison result and the gradient information.

[0099] In the aforementioned pixel mean image, the control device can first search from bottom to top for the row where the pixel value first satisfies the condition of being less than the skin color brightness threshold, and use this row as the initial coordinate in the vertical direction of the lower eyelid. The control device uses coordinates on the pixel mean image. Within the central region, the maximum gradient point is searched based on the gradient information calculated above, and the y-coordinate of this point is used as the final position of the lower eyelid. Then, the control device can locate the upper eyelid by searching from top to bottom in the pixel mean image to find the row where the pixel value is first less than the skin tone brightness threshold. Then, starting from that row, the device searches downwards for the row with the largest gradient, and uses the y-value of that row as the final location of the upper eyelid. For example, in... Figure 4B The detected positions of the upper and lower eyelids are marked with white lines.

[0100] Step S380: If the eyelid distance in the target eyelid positioning result is greater than the preset eyelid distance threshold, obtain the pupil coarse positioning image based on the eyelid positioning sub-image; if the eyelid distance is less than or equal to the preset eyelid distance threshold, obtain the blink determination result.

[0101] The control device can calculate the eyelid distance between the upper and lower eyelid positions based on the target eyelid positioning results. If the eyelid distance is greater than a preset eyelid distance threshold, it indicates that the user is not blinking, and subsequent pupil positioning steps can be performed based on the eyelid positioning sub-image. If the eyelid distance is less than or equal to the preset eyelid distance threshold, it indicates that the eyelid distance is too small, meaning the user is blinking. Since pupil detection would be inaccurate in this case, the blink determination result can be obtained directly, and the eye image processing flow can be terminated to improve the accuracy and efficiency of eye image processing.

[0102] Through steps S320 to S380, an eyelid localization process is added before pupil localization. This allows for the method of first determining whether blinking is occurring using the interpupillary distance, and then using the determination result to achieve pupil localization. This enhances the robustness of pupil detection and effectively improves the accuracy and efficiency of eye image processing. Furthermore, in the subsequent corneal reflection spot localization node, the eyelid position information in this process can also serve as a supplementary condition for filtering out the high-brightness information formed by secondary reflections of the light spot in the fundus region.

[0103] It should be noted that this embodiment also provides an eyelid localization algorithm. Figure 5 This is a flowchart of an eyelid localization algorithm according to an embodiment of this application, such as... Figure 5 As shown, the process includes the following steps:

[0104] Step S501: Obtain infrared eye-tracking images as eye image information; to reduce computational load, first perform 4x downsampling on the eye image information to obtain a smaller image.

[0105] Step S502: Calculate the integral image on the reduced image smallImage, find the position with the maximum Haar response, take the position with the maximum Haar response value as the center, and take twice the Haar response radius as the side length, extract the initial localization sub-image of the pupil from the reduced image smallImage, that is, the first region of interest image roiImag.

[0106] Step S503: First, perform debinarization on the roiImage to obtain a binary image, so that the low-brightness area of ​​the image is displayed as white and the high-brightness area is displayed as black, and perform morphological filtering to eliminate isolated noise. Then, extract the contours of the connected regions in the processed binary image. Based on the prior information that the pupil shape is approximately circular and the pupil area accounts for the largest proportion in the roiImage image, the first optimal contour is selected.

[0107] Step S504: Based on the first optimal contour, roughly obtain the pupil center position ptCenter and pupil radius fRadius; based on the pupil center position ptCenter, extract the following from the eye image information: Figure 4A The eyelid localization sub-image pullil_strip in the area indicated by the black box in the middle.

[0108] Step S505: The eyelid positioning sub-image pullil_strip is filtered, and the average pixel value avgImage of each row is calculated. The gradient information gradient of the average pixel value image avgImage in the y direction is calculated.

[0109] Step S506: Perform k-means clustering calculation on the pixel average image avgImage to obtain the skin color brightness threshold skinVal.

[0110] Step S507: In the pixel average image avgImage, search from bottom to top for the row where the pixel value first satisfies the condition of < skinVal, and use it as the coordinate in the vertical direction of the preliminary lower eyelid.

[0111] Step S508: Centered on in the vertical direction of the image, within the radius delta region, search for the point with the maximum gradient based on the gradient information gradient, and use the y coordinate of this point as the final position of the lower eyelid.

[0112] Step S509: Then locate the upper eyelid. In the pixel average image avgImage, search from top to bottom: first search for the row information where the pixel value first satisfies < skinVal, and then search downward from this row for the row with the maximum gradient, and use the y value of this row as the final position of the upper eyelid.

[0113] Step S510: Calculate the eyelid distance and determine whether the eyelid distance is too small; if not, output the target eyelid localization result; if so, determine it as a blink and end the eye image processing flow.

[0114] In some embodiments, obtaining the rough pupil localization image based on the eyelid localization sub-image further includes the following steps:

[0115] Step S381: Perform inverse binaryzation processing on the eyelid localization sub-image to obtain a second preprocessing image, and perform contour extraction processing on the second preprocessing image to obtain a second optimal contour.

[0116] It should be added that since the upper eyelid naturally forms a shadow on the eyeball, in order to improve the pupil detection rate, the control device can first perform contrast enhancement processing on the eyelid localization sub-image to strengthen the edge information. The control device performs inverse binaryzation operation on the contrast-processed eyelid localization sub-image, and removes noise through morphological filtering to obtain the above-mentioned second preprocessing image. The control device searches for the contours of all connected regions in the second preprocessing image. Since it can be approximately considered that the pupil is located in the central region of the image in the second preprocessing image, all contours are traversed based on this prior condition, and the contour with the largest area and the ratio of the x coordinate to the y coordinate of the region center point close to 1 is found as the above-mentioned second optimal contour pupil_contour; it can be understood that if the control device fails to find the second optimal contour that meets the conditions, it is considered that there is no pupil in the current second preprocessing image, and the eye image processing flow can be directly ended to improve the eye image processing efficiency.

[0117] Step S382: Based on the first optimal contour, obtain the pupil center position and pupil radius information, and filter the second optimal contour according to the pupil center position and pupil radius information to obtain the first contour point set.

[0118] Specifically, based on the pupil center position ptCenter and pupil radius information fRadius obtained during the eyelid localization stage, contour points that are too close to the upper and lower eyelid regions in the second optimal contour are filtered to obtain the first contour point set clean_pupil_contour after filtering. This eliminates the interference of factors such as eyelashes, which helps to improve the accuracy of pupil localization.

[0119] Step S383: If the number of the first contour point set is greater than the preset number of contour points, iteratively fit the first contour point set to obtain a fitting circle result; obtain the pupil contour point result based on the first contour point set and the fitting circle result, and obtain the pupil coarse positioning image based on the pupil contour point result.

[0120] The number of preset contour points can be set by the user, for example, to 5. When the control device detects that the number of the first contour point set is greater than the preset number of contour points, it indicates that the detection is effective. The least squares method can be used for iterative fitting to obtain the fitted circle result with the minimum loss. The first contour point set is then matched with the fitted circle result to obtain the matched contour points as the pupil contour point result. Finally, the pupil coarse localization image is obtained by fitting the pupil contour point result.

[0121] Step S384: If the number of contour points is less than or equal to the preset number of contour points, the blink determination result is obtained.

[0122] Through the above steps S381 to S384, the method of first obtaining the first contour point set by filtering eyelid information and then determining whether blinking occurs based on the first contour point set is realized in the coarse pupil localization stage. This avoids the error in detection results caused by pupil detection when blinking, enhances the robustness of pupil detection, and further improves the accuracy and efficiency of eye image processing.

[0123] In some embodiments, obtaining the pupil contour point result based on the first contour point set and the fitted circle result further includes the following steps: scaling up and scaling down the fitted circle result to obtain a circular template result; and obtaining the pupil contour point result based on the first contour point set and the circular template result.

[0124] Specifically, the radius of the fitted circle is enlarged and reduced according to a preset ratio. This preset ratio can be set by the user, for example, to 1.01 and 0.99. That is, the control device enlarges the fitted circle to 1.01 times its radius by a ratio of 1.01, and reduces it to 0.99 times its radius by a ratio of 0.99, thus forming a ring based on the enlargement and reduction results, resulting in the aforementioned ring template. Then, the control device selects contour points that fall within this ring template from the first contour point set as pupil contour points. This achieves a filtering method that automatically removes noise points while avoiding the omission of contour points due to the first contour point set not perfectly matching the fitted circle.

[0125] Through the above embodiments, the fitting circle is enlarged and reduced proportionally to obtain the annular template result. Based on the annular template result, the first contour point set is further filtered to finally obtain the pupil contour point result. This realizes the method of filtering isolated points (noise points) in the coarse pupil positioning process and improves the accuracy of pupil ellipse fitting.

[0126] In some embodiments, obtaining the target pupil localization result based on the edge image and the pupil coarse localization image further includes the following steps:

[0127] Step S261: Enlarge the edge image and the coarse pupil localization image proportionally and perform an AND operation to obtain a first overlapping point set. Then, perform fitting processing on the first overlapping point set to obtain the first pupil fitting result.

[0128] It should be further explained that, in order to improve the accuracy of image processing, the control device can first enlarge the edge image and the pupil coarse positioning image by the same proportion, and perform morphological dilation on the elliptical region in the pupil coarse positioning image. Then, the control device performs a bitwise AND operation on the processed edge image and the pupil coarse positioning image to obtain a first set of overlapping points, and performs ellipse fitting based on the first set of overlapping points to obtain the first pupil fitting result.

[0129] Step S262: Perform an AND operation on the edge image and the first pupil fitting result to obtain a second overlapping point set; perform fitting processing on the second overlapping point set to obtain a second pupil fitting result; and perform highlight filtering processing on the second pupil fitting result to obtain a second contour point set.

[0130] Specifically, the control device performs a bitwise AND operation on the edge image and the first pupil fitting result to obtain a second overlapping point set for secondary refinement, and then performs ellipse fitting to obtain a secondary refined ellipse. Based on the secondary refined ellipse, the control device obtains its pixel set on the first preprocessed image, then filters out points with high grayscale values ​​(i.e., bright points) based on the brightness information of the first preprocessed image, and finally uses the remaining contour points after filtering as the second contour point set.

[0131] Step S263: Calculate the gradient direction of the second contour point set traversed sequentially, filter the second contour point set based on the gradient direction result to obtain the third contour point set, and generate the target pupil positioning result based on the third contour point set.

[0132] The control device traverses the second contour point set, searching for the location of the maximum gradient within a local region along the line connecting the center of the ellipse and the current contour point. Based on this location, the current contour point is fine-tuned and updated to ensure accurate pupil localization. Simultaneously, the control device sequentially calculates the gradient direction for each traversed contour point. Using the line connecting the center of the ellipse and the current contour point as the base_angle, the device compares the degree of overlap between the equation of the line represented by base_angle and the gradient direction. If the equation of the line approximately overlaps with the gradient direction, the corresponding current contour point is saved; otherwise, if there is a significant difference, the current contour point is deleted. After the traversal is complete, a filtered third contour point set is obtained, and the target pupil localization result is finally generated based on this third contour point set. Specifically, Figure 6A The white point set in the middle is based on the third contour point set, and multiple white point sets constitute an approximately elliptical pupil region; Figure 6B The white line marked in the figure represents the final fitted approximate ellipse of the target pupil localization result.

[0133] Through steps S261 to S263, a second contour point set is obtained by performing two AND operations on the edge image and the coarse pupil localization image. This effectively improves the accuracy of fitting the pupil ellipse using the contour point set. At the same time, by traversing the second contour point set to obtain the gradient direction results of each point, the second contour point set is filtered based on the gradient direction results, which further improves the accuracy of pupil localization and thus improves the accuracy of eye image processing.

[0134] It should be noted that this embodiment also provides a pupil localization algorithm. Figure 7 This is a flowchart of a pupil localization algorithm according to an embodiment of this application, such as... Figure 7 As shown, the process includes the following steps:

[0135] Step S701: Obtain infrared eye movement image as eye image information; during the preprocessing process, based on steps S501 to S504 in the above eyelid localization algorithm, obtain low-brightness area information, and then obtain the region of interest image pupilImage for pupil localization, i.e., the above eyelid localization sub-image.

[0136] In step S702, since the upper eyelid naturally forms a shadow on the eyeball, in order to improve the pupil detection rate, the pupilImage image is first subjected to contrast enhancement processing to strengthen edge information.

[0137] Step S703: Based on the initial positioning results, set the upper and lower bound constraints of the contour points and end the preprocessing process.

[0138] Step S704: Start coarse pupil localization; perform inverse binarization and morphological filtering on the pupilImage image to remove noise, and obtain the second preprocessed image.

[0139] Step S705: Find the contours of all connected components in the second preprocessed image to obtain the contours. Since the pupilImage image is obtained based on the Haar response, it can be approximated that the pupil is located in the central region of the image. Using this as a priori condition, traverse all contours and find the contour with the largest area and the ratio of the x-coordinate to the y-coordinate of the center point of the region is close to 1 as the second optimal contour pupil_contour.

[0140] Step S706: If a second optimal contour that meets the conditions is found, based on the pupil coarse positioning center and radius information obtained in the eyelid positioning stage, filter the contour points in the second optimal contour pupil_contour that are too close to the upper and lower eyelid regions, that is, filter the contour points based on the upper and lower bound constraints to eliminate the interference of eyelashes, etc., and obtain the filtered first contour point set clean_pupil_contour; otherwise, it is considered that there is no pupil in the current image, and the pupil positioning algorithm ends.

[0141] Step S707: Determine if the number of clean_pupil_contour points is greater than 5; if not, it indicates coarse localization recognition, end the pupil localization algorithm and output the blink judgment result that is determined to be blinking; if yes, iteratively fit the circle with the minimum loss based on the least squares method.

[0142] Step S708: Enlarge and reduce the radius of the circle to obtain the circular template mask, and select the pupil_contour contour points that fall within the circle of the mask as the pupil contour points filteredPoints.

[0143] Step S709: An ellipse is obtained by fitting the pupil contour points filteredPoints to locate the pupil, thereby obtaining the pupil coarse localization image pupil_ellipse and ending the pupil coarse localization process.

[0144] Step S710: Start pupil fine localization; use the OTSU algorithm to obtain the segmentation threshold th for the pupilImage image, and perform Canny edge detection based on the segmentation threshold th to obtain the edge image edgeImage.

[0145] Step S711: The edge image edgeImage and the pupil coarse localization image pupil_ellipse are enlarged proportionally to obtain the enlarged edge image scale_edgeImage and the enlarged pupil coarse localization image scale_pupil_ellipse_image, respectively; the ellipse described by scale_pupil_ellipse_image is morphologically dilated, and then the image is ANDed with scale_edgeImage to obtain the first overlapping point set boundaryPoints, and the first pupil fitting result pupil_ellipse1 is obtained based on the first overlapping point set.

[0146] Step S712: Perform an AND operation on the edge image edgeImage of the original image size and the first pupil fitting result pupil_ellipse1 to obtain the contour edge point set again, that is, the second overlapping point set boundaryPoints, and perform ellipse fitting to obtain the secondary refined ellipse pupil_ellipse2.

[0147] Step S713: Based on the secondary refined ellipse pullil_ellipse2, obtain its pixel point set boundaryPoints on the pullilImage image. Then, filter the highlighted points based on the highlight information, and then use the remaining points after filtering as the ellipse contour points, that is, the second contour point set filtered_boundaryPoints.

[0148] Step S714: Traverse the contour points, search for the location of the maximum gradient in the local region along the line connecting the ellipse center and the current contour point pi, and fine-tune and update the contour point pi based on the location of the maximum gradient.

[0149] Step S715: Calculate the gradient direction angle for each point in sequence, and let the direction of the line connecting the ellipse center center and the current contour point pi be base_angle. If the equations of the lines represented by angle and base_angle are approximately coincident, save the current contour point; if there is a large difference between the equations of the lines represented by the two, delete the current contour point; end the pupil fine positioning process; obtain the contour point information and corresponding ellipse information that fit the edge of the pupil and output them.

[0150] In some embodiments, an eye image processing method is provided. Figure 8 This is a flowchart of another eye image processing method according to an embodiment of this application, such as... Figure 8 As shown, the process includes Figure 2 Steps S220 to S260 shown herein also include the following steps:

[0151] Step S820: Obtain the topology vector information of the supplementary light source, which is used to provide supplementary lighting during the acquisition of the eye image information.

[0152] Understandably, eye-tracking schemes based on the pupil-corneal reflection method typically use infrared LEDs as supplementary lighting sources, generating a reference spot on the near-spherical cornea. Changes in the eye's gaze movement are reflected in the infrared image as changes in the pupil position. Before detecting the corneal reflection spot, the user can input pre-deployed LED hardware structure design information into the aforementioned control device. Based on this information, the control device constructs and generates a set of topological vectors for each supplementary lighting source, i.e., the aforementioned topological vector information.

[0153] Step S840: Extract the second region of interest image from the coarse pupil localization image, perform contour extraction processing on the second region of interest image to obtain a contour sequence, and obtain a bright spot contour sequence based on the contour sequence.

[0154] Specifically, the control device sequentially extracts the second region of interest (ROI) image for each light spot from the coarse pupil localization image. It is understood that the control device can also perform processing such as size enlargement on the second ROI image to improve the accuracy of light spot localization. The control device also performs binarization processing on the second ROI image to obtain the bright information connected components, and extracts the contours of these connected components to obtain a contour sequence. For contours with excessively large areas, the control device can also use morphological methods to divide them into multiple smaller regions. Then, the control device sequentially calculates the minimum bounding rectangle for each contour and, combined with the aforementioned eyelid information, filters out contours that are significantly off-center from the image. The contour set is updated, and a list of bright spot contour centers is calculated, i.e., the aforementioned bright spot contour sequence.

[0155] Step S860: Obtain the target spot localization result based on the topological vector information and the bright spot contour sequence, and generate the eye movement feature extraction result based on the target pupil localization result and the target spot localization result.

[0156] The control device can calculate the sequence with the highest matching degree with the topological vector in the bright spot contour sequence based on the topological vector information, and use it as the corneal reflective spot sequence. Based on the corneal reflective spot sequence, the target spot localization result is obtained, and finally the eye movement feature extraction result is generated.

[0157] The corneal reflective spot serves as reference information for pupil movement, and its positioning accuracy directly affects the accuracy of fixation point calculation. Common algorithms used in related technologies to calculate the spot center include the centroid method, fitting method, Hough transform method, and radial symmetry method. The latter two methods involve large computational loads and are unsuitable for low-power devices. The centroid method is simple to implement and fast, but has poor anti-interference capabilities and lower accuracy. The fitting method relies on the extraction of edge contour points; if the spot is too small, resulting in fewer contour points than required for fitting the general equation of an ellipse / circle, the method will fail. Furthermore, the LED infrared light source in the eye-tracking system's hardware structure design contains structural information. The methods described above do not incorporate this structural design information to match the hardware ID number corresponding to each spot, lacking a mapping relationship from image coordinates to corresponding 3D coordinates in the world coordinate system. This will affect the subsequent calculation of eye-tracking results.

[0158] In this embodiment, by obtaining the topological vector information of the supplementary light source through steps S820 to S860, a bright spot contour sequence is obtained based on the coarse pupil positioning image. Then, the target spot positioning result is obtained according to the topological vector information and the bright spot contour sequence. This provides a spot positioning method that combines hardware LED light distribution design, realizes the mapping from image coordinates to world coordinate system, and is simple to calculate. Therefore, it further improves the accuracy and efficiency of eye image processing.

[0159] In some embodiments, obtaining the target spot localization result based on the topological vector information and the bright spot contour sequence further includes the following steps:

[0160] Step S861: Take the bright spot contour points in the traversed bright spot contour sequence as reference points in turn, and calculate the vector information between the other bright spot contour points in the bright spot contour sequence and the reference points.

[0161] Step S862: Obtain a template vector based on the topological vector information and the vector information; combine the topological vector information and the bright spot contour sequence to obtain a light source spot sequence; and match the light source spot sequence and the template vector to obtain a corneal reflection spot sequence.

[0162] Specifically, after obtaining the above vector information, a topological vector that is approximately collinear with the above vector information is searched from the above topological vector information as a template vector. Since the hardware information of the supplementary light source is referenced in the embodiments of this application, each spot in each sequence has a corresponding hardware ID. The above bright spot contour sequences can be combined and merged in the same way according to the hardware ID. For example, the spots corresponding to hardware ID 1 are merged into one set of spot sequences, and the spots corresponding to hardware ID 2 are merged into another set of spot sequences, thereby obtaining the above light source spot sequence.

[0163] The control device calculates the sum of distances from each group of light spots in the light spot sequence to the center of the pupil, and takes the target light spot sequence with the smallest sum of distances and the highest matching degree with the template vector as the corneal reflection light spot sequence, and calculates the center of each light spot in the corneal reflection light spot sequence as the coarse localization result of the light spot.

[0164] Step S863: Traverse the corneal reflective spot sequence and obtain the third region of interest image corresponding to each corneal reflective spot in the sequence. Perform contour extraction processing on each third region of interest image to obtain the target spot.

[0165] Specifically, the control device sequentially extracts the third region of interest (ROI) image of each light spot from the aforementioned eye image information, performs size enlargement and binarization processing, and then extracts the contour and finds the connected component with the largest area. It iterates through all light spots. If the number of contour points of the largest connected component corresponding to the current light spot is less than or equal to a preset number, the current bright spot information is discarded, and the third ROI images corresponding to other light spots are extracted again, until all light spots have been traversed. If the number of contour points of the largest connected component is greater than the preset number, the corresponding current light spot is taken as the target light spot, ellipse fitting is performed on the target light spot, the geometric center of the fitted ellipse is calculated and saved.

[0166] Step S864: Obtain the target spot positioning result based on the target spot.

[0167] After traversing all light spots through step S863, the number of target light spots is counted. If the number of target light spots is less than a preset threshold, the requirement for subsequent corneal center calculation is not met, and the algorithm can be directly terminated in processing the current eye image information. If the number of target light spots is greater than or equal to the preset threshold, the corneal reflection light spot sequence information is saved in the form of (hardware ID, ellipse geometric center) data pairs and used as the target light spot localization result for subsequent eye-tracking calculation and analysis. Specifically, Figure 9 The four white dots marked around the pupil represent the detected LED light spot information.

[0168] Through steps S861 to S864 above, template vectors are obtained by using topological vector information and vector information in the bright spot contour sequence. Thus, a template matching algorithm based on the relative position vectors of any two LEDs is designed, which reduces the interference of other bright noise points in the eye image information on the spot extraction and helps to improve the accuracy of spot extraction.

[0169] It should be noted that this embodiment also provides a corneal reflection spot localization algorithm. Figure 10 This is a flowchart of a corneal reflective spot localization algorithm according to an embodiment of this application, such as... Figure 10 As shown, the process includes the following steps:

[0170] Step S1001: Obtain the input information of the corneal reflection spot localization algorithm, including: infrared near-eye image, pupil coarse localization image extracted from pupil localization (pupil_ellipse), eyelid information (eyelieds), and LED hardware structure design information.

[0171] Step S1002: Start coarse localization of the light spot; based on the coarse localization image of the pupil obtained by the pupil localization algorithm, the region of interest image roiImage extracted from the corneal reflected light spot is set.

[0172] Step S1003: Binarize the roiImage image to obtain the highlighted connected components.

[0173] Step S1004: Extract contours from the above connected components to obtain contour sequences.

[0174] Step S1005: Traverse contours. For contours with excessively large areas, use morphological methods to process them into multiple smaller areas to achieve the segmentation of the adhered contour regions; update the contour set contours.

[0175] Step S1006: Traverse contours, calculate the minimum bounding rectangle of each contour in turn, and filter contours that are significantly off-center from the image center by combining eyelid information to achieve high-brightness area filtering; update the contour set contours and calculate the bright spot contour center list facula_center_list.

[0176] Step S1007: Determine whether the number of candidate combinations of bright spots in the above bright spot contour center list facula_center_list is less than or equal to 0; if yes, end the corneal reflection spot localization process; if no, continue to execute the subsequent steps.

[0177] Step S1008: Based on the hardware structure information, construct a set of topological vectors templates for each LED light source; sequentially using the facula_center of each bright spot contour as the base, calculate the vector v of the remaining bright spot contour centers relative to the base, and search for approximately collinear template vectors from the templates; and merge the bright spots in the bright spot contour center list according to the hardware ID indicated by the hardware structure information.

[0178] Step S1009: Determine whether the number of the above bright spot combinations is greater than 0; if yes, then select the optimal light spot combination based on the sum of the distances to the center of the pupil and the matching degree with the hardware template; if no, then end the corneal reflection light spot positioning process.

[0179] Step S1010: Calculate the center of the light spot and end the coarse positioning process of the light spot.

[0180] Step S1011: Start fine localization of light spots; read the information of the i-th light spot and scale up the light spot image to obtain the magnified image scale_roi.

[0181] Step S1012: Perform binarization on the scale_roi image, extract the contour, and find the connected component with the largest area. The contour of the connected component with the largest area is the optimal contour.

[0182] Step S1013: Determine whether the number of points of the above optimal contour is greater than 5; if not, discard the current spot information and re-execute step S1011; if yes, execute the subsequent steps.

[0183] Step S1014: Perform ellipse fitting on the current spot information, calculate and record the geometric center of the spot; end the spot fine positioning process.

[0184] Step S1015: Determine if the number of light spots is less than 2; if yes, the corneal reflection light spot localization process ends directly; if no, the corneal reflection light spot sequence information is saved in the form of (ID, center) data pairs and output to the subsequent analysis module.

[0185] The embodiments of this application will be described in detail below with reference to practical application scenarios. Figure 11 This is a flowchart of an eye image processing method according to a preferred embodiment of this application, such as... Figure 11 As shown, the process includes the following steps:

[0186] Step S1101: Acquire infrared eye-tracking images as eye image information.

[0187] Step S1102: Start the pupil localization process; extract the region of interest (ROI) image from the eye image information based on the Haar response.

[0188] Step S1103: Perform eyelid localization based on the ROI image, and determine whether the eyelid distance is too small based on the eyelid localization result; if so, proceed to step S1107.

[0189] Step S1104: If the judgment result of step S1103 is negative, then the pupil region is coarsely located, the pupil region is extracted from the eye image information, and inverse binarization is performed to generate a coarse pupil location image based on the coarsely located pupil outline.

[0190] Step S1105: Determine whether the coarse positioning was successful; if not, proceed to step S1107.

[0191] Step S1106: If the judgment result of step S1105 is yes, then perform Canny edge detection based on the adaptive threshold, and iteratively correct the pupil contour based on the pupil coarse localization image and the edge image to obtain the target pupil localization result; end the pupil localization process.

[0192] Step S1107: Determine if it is blinking; end the eye image processing flow.

[0193] Step S1108: Start corneal reflective spot localization; extract the ROI image, perform binarization and highlight region contour extraction.

[0194] Step S1109: Based on topological prior information, select LED light spots.

[0195] Step S1110: Based on the LED spot, refine the LED outline to obtain the target spot positioning result.

[0196] Through the above steps S1101 to S1110, a robust feature extraction method is provided, achieving a detection rate of over 99% on infrared near-eye images in eye-tracking projects, with detection accuracy reaching sub-pixel level. Furthermore, when tested on a PC, the time overhead is less than 30ms, meeting real-time requirements, thus ensuring the accuracy and efficiency of the eye image processing method.

[0197] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0198] This embodiment also provides an eye image processing apparatus for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0199] Figure 12 This is a structural block diagram of an eye image processing device according to an embodiment of this application, such as... Figure 12 As shown, the device includes: an acquisition module 122, a coarse localization module 124, a target module 126, and a generation module 128; the acquisition module 122 is used to acquire eye image information; the coarse localization module 124 is used to perform inverse binarization processing on the eye image information to obtain a first preprocessed image, and perform contour extraction processing on the first preprocessed image to obtain a pupil coarse localization image; the target module 126 is used to perform edge detection processing on the first preprocessed image to obtain an edge image, and obtain a target pupil localization result based on the edge image and the pupil coarse localization image; the generation module 128 is used to generate an eye movement feature extraction result for the eye image information based at least on the target pupil localization result.

[0200] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0201] This embodiment also provides an eye-tracking system, which includes an image acquisition device and a control device. The image acquisition device is used to acquire eye image information. The control device is used to execute the steps in any of the above method embodiments based on the eye image information to obtain eye movement feature extraction results, and to generate eye-tracking results based on the eye movement feature extraction results. The control device includes, but is not limited to, various server devices, processing chips, microcontrollers, personal computers, or other devices used to control the jump rope counting process to generate target jump rope detection results. Specifically, this eye-tracking system can be applied to augmented reality (AR) or virtual reality (VR) devices. Taking AR glasses as an example, the image acquisition device of the AR glasses acquires eye image information. The pupil detection unit and LED reflective spot detection unit of the control device in the AR glasses obtain the target pupil positioning result and the target light spot positioning result respectively through the above method embodiment. The corneal center calculation unit in the control device calculates the corneal center position based on the target light spot positioning result, and the pupil center calculation unit in the control device calculates the pupil center position based on the target pupil positioning result. Then, the gaze point calculation unit in the control device calculates the gaze point based on the corneal center position and the gaze point calculation unit. The AR glasses finally realize interaction based on the gaze point calculation result. Through the above embodiment, a method for pupil positioning and corneal reflective spot detection in an end-to-end eye-tracking system is provided, providing reliable feature information for eye-tracking schemes based on the pupil-corneal reflection method.

[0202] In some embodiments, a computer device is provided, which may be a server. Figure 13 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application, such as... Figure 13 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores eye-tracking feature extraction results. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned eye image processing method.

[0203] Those skilled in the art will understand that Figure 13The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0204] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0205] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0206] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0207] S1, acquire eye image information.

[0208] S2, the eye image information is debinarized to obtain a first preprocessed image, and the first preprocessed image is contour extracted to obtain a coarse pupil localization image.

[0209] S3, perform edge detection processing on the first preprocessed image to obtain an edge image, and obtain the target pupil localization result based on the edge image and the pupil coarse localization image.

[0210] S4, at least based on the target pupil localization result, generate eye movement feature extraction results for the eye image information.

[0211] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0212] Furthermore, in conjunction with the eye image processing methods in the above embodiments, this application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any one of the eye image processing methods in the above embodiments.

[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0214] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0215] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An eye image processing method characterized by, The method comprises: obtaining eye image information; performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a coarse pupil positioning image; performing edge detection processing on the first preprocessed image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the coarse pupil positioning image, comprising: performing same-scale magnification and intersection operation on the edge image and the coarse pupil positioning image to obtain a first overlapping point set, and performing fitting processing on the first overlapping point set to obtain a first pupil fitting result; performing intersection operation on the edge image and the first pupil fitting result to obtain a second overlapping point set, performing fitting processing on the second overlapping point set to obtain a second pupil fitting result, and performing highlight filtering processing on the second pupil fitting result to obtain a second contour point set; sequentially calculating gradient direction results of the second contour point set that is traversed, performing screening processing on the second contour point set based on the gradient direction results to obtain a third contour point set, and generating the target pupil positioning result according to the third contour point set; generating an eye movement feature extraction result for the eye image information according to at least the target pupil positioning result, comprising: obtaining topology vector information of a light supplement source, the light supplement source being used for light supplement during acquisition of the eye image information; extracting a second region of interest image from the coarse pupil positioning image, performing contour extraction processing on the second region of interest image to obtain a contour sequence, and obtaining a glint contour sequence based on the contour sequence; based on the topology vector information, calculating a sequence with the highest topology vector matching degree in the glint contour sequence as a corneal reflection glint sequence, obtaining a target glint positioning result based on the corneal reflection glint sequence, and generating the eye movement feature extraction result according to the target pupil positioning result and the target glint positioning result.

2. The eye image processing method of claim 1, wherein, The inverse binarization processing on the eye image information to obtain the first preprocessed image comprises: calculating an integral image based on the eye image information, calculating a Haar feature based on the integral image, obtaining a Haar response maximum value and a Haar response radius based on the Haar feature, and obtaining a first region of interest image based on the Haar response maximum value and the Haar response radius; performing inverse binarization processing on the first region of interest image to obtain the first preprocessed image.

3. The eye image processing method of claim 1, wherein, The contour extraction processing on the first preprocessed image to obtain the coarse pupil positioning image comprises: performing contour extraction processing on the first preprocessed image to obtain a first optimal contour, determining an eyelid positioning sub-image of the eye image information based on the first optimal contour, calculating a pixel mean value image based on the eyelid positioning sub-image, and calculating gradient information of the pixel mean value image; performing clustering on the pixel mean value image by using a clustering algorithm to obtain a clustering result, and obtaining a skin color brightness threshold value based on the clustering result; The pixel value obtained by traversing the pixel mean image is compared with the skin color brightness threshold value respectively, and a target eyelid positioning result is determined according to the comparison result and the gradient information; In a case where an eyelid distance in the target eyelid positioning result is greater than a preset eyelid distance threshold, a pupil rough positioning image is obtained based on the eyelid positioning sub-image; in a case where the eyelid distance is less than or equal to the preset eyelid distance threshold, a blink determination result is obtained.

4. The eye image processing method of claim 3, wherein, The pupil rough positioning image based on the eyelid positioning sub-image includes: The eyelid positioning sub-image is subjected to inverse binarization processing to obtain a second preprocessed image, and the second preprocessed image is subjected to contour extraction processing to obtain a second optimal contour; Pupil center position and pupil radius information are obtained based on the first optimal contour, and the second optimal contour is subjected to screening processing according to the pupil center position and the pupil radius information to obtain a first contour point set; In a case where the number of the first contour point set is greater than a preset contour point number, the first contour point set is subjected to iterative fitting to obtain a fitting circle result; a pupil contour point result is obtained according to the first contour point set and the fitting circle result, and the pupil rough positioning image is obtained according to the pupil contour point result; In a case where the number of the contour point set is less than or equal to the preset contour point number, the blink determination result is obtained.

5. The eye image processing method of claim 4, wherein, The pupil contour point result obtained according to the first contour point set and the fitting circle result includes: The fitting circle result is subjected to proportional amplification and proportional reduction processing respectively to obtain a circular ring template result; The pupil contour point result is obtained according to the first contour point set and the circular ring template result.

6. The eye image processing method of claim 1, wherein, The topological vector information is used to calculate a sequence with the highest matching degree with the topological vector in the bright spot contour sequence as a corneal reflection light spot sequence, and a target light spot positioning result is obtained based on the corneal reflection light spot sequence, including: The bright spot contour points in the bright spot contour sequence traversed in sequence are used as reference points, and vector information between other bright spot contour points in the bright spot contour sequence and the reference points is calculated; A template vector is obtained according to the topological vector information and the vector information, and a light source light spot sequence is obtained according to the topological vector information and the bright spot contour sequence combination; A corneal reflection light spot sequence is matched according to the light source light spot sequence and the template vector, and a light spot rough positioning result is obtained according to the corneal reflection light spot sequence; The corneal reflection light spot sequence is traversed, and a third region of interest image corresponding to each corneal reflection light spot in the corneal reflection light spot sequence is obtained in sequence, and a target light spot is obtained by performing contour extraction processing on each third region of interest image; The target light spot positioning result is obtained based on the target light spot and the light spot rough positioning result.

7. The eye image processing method according to any one of claims 1 to 6, characterized in that, The edge detection processing of the first preprocessed image includes: OTSU algorithm is used for image segmentation processing of the first preprocessed image to obtain a segmentation threshold, and the first preprocessed image is subjected to edge detection according to the segmentation threshold to obtain the edge image.

8. An ocular image processing apparatus characterized by comprising: The device comprises an acquisition module, a coarse positioning module, a target module and a generation module; The acquisition module is configured to acquire eye image information; The coarse positioning module is configured to perform inverse binarization processing on the eye image information to obtain a first preprocessed image, and perform contour extraction processing on the first preprocessed image to obtain a coarse pupil positioning image; The target module is configured to perform edge detection processing on the first preprocessed image to obtain an edge image, and obtain a target pupil positioning result according to the edge image and the coarse pupil positioning image, including: performing same-scale magnification and intersection operation on the edge image and the coarse pupil positioning image to obtain a first overlapping point set, performing fitting processing on the first overlapping point set to obtain a first pupil fitting result; performing intersection operation on the edge image and the first pupil fitting result to obtain a second overlapping point set, performing fitting processing on the second overlapping point set to obtain a second pupil fitting result, and performing highlight filtering processing on the second pupil fitting result to obtain a second contour point set; sequentially calculating gradient direction results of the second contour point set that is traversed, performing screening processing on the second contour point set based on the gradient direction results to obtain a third contour point set, and generating the target pupil positioning result according to the third contour point set; The generation module is configured to generate an eye movement feature extraction result for the eye image information according to at least the target pupil positioning result, including: acquiring topology vector information of a light supplement source, the light supplement source being used for light supplement during acquisition of the eye image information; extracting a second region of interest image according to the coarse pupil positioning image, performing contour extraction processing on the second region of interest image to obtain a contour sequence, and obtaining a glint contour sequence based on the contour sequence; based on the topology vector information, calculating a sequence in the glint contour sequence that has the highest topology vector matching degree as a corneal reflection glint sequence, and obtaining a target glint positioning result based on the corneal reflection glint sequence, and generating the eye movement feature extraction result according to the target pupil positioning result and the target glint positioning result.

9. An eye tracking system, characterized by The eye movement tracking system comprises an image acquisition device and a control device; The image acquisition device is configured to acquire eye image information; The control device is configured to execute the eye image processing method according to any one of claims 1 to 7 for the eye image information to obtain the eye movement feature extraction result, and generate an eye movement tracking result based on the eye movement feature extraction result. 10.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to execute the eye image processing method according to any one of claims 1 to 7.

11. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the eye image processing method according to any one of claims 1 to 7 when executed.

Images