Marker-based sight line estimation method and device

Abstract

The invention discloses a marker-based sight line estimation method and device. The method comprises the following steps: acquiring to-be-processed eye image information; the to-be-processed eye image information comprises two pieces of to-be-estimated eye image information; performing feature information extraction on the to-be-processed eye image information to obtain target marker feature information; the target marker feature information comprises first target feature information and second target feature information; and performing sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for estimating sight lines based on markers. Background Art

[0002] Gaze estimation refers to estimating a person's gaze direction through image or video data. Traditional gaze estimation algorithms are mainly based on computer vision and image processing technology. They use eye feature points (such as pupils, eye corners, etc.) for tracking through feature point tracking methods. By analyzing the movement of these feature points in continuous image frames, the eyeball movement trajectory is estimated, and the gaze direction is obtained. When extracting features based on the threshold method, it is impossible to accurately separate feature points and non-feature point areas due to uneven illumination or grayscale distribution, and the use of adaptive thresholds has a high computational overhead; traditional feature point detection algorithms are affected by pupils being blocked by eyelashes or eyelids, partially closed eyes, and pupil areas not fully exposed, and cannot effectively detect the pupil contour area through contour detection; traditional feature points include pupils and eye corners or Purkinje spots, etc., and environmental factors such as changes in lighting conditions, reflections, and shadows may cause problems in the detection of pupils and eye corners. These factors may cause unstable feature point detection, thereby affecting the accuracy of gaze estimation. Therefore, a marker-based line of sight estimation method and device are provided to solve the problem that the traditional feature point detection algorithm is unable to effectively detect the contour area of ​​the pupil through contour detection due to the influence of the pupil being blocked by eyelashes or eyelids, partial closure of the eyes, and incomplete exposure of the pupil area, so as to accurately locate the eye position, improve the alignment and tracking of virtual objects, and enhance the user's eye device usage experience. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a marker-based line of sight estimation method and device that is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0004] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a sight line estimation method based on a marker, the method comprising:

[0005] Acquire eye image information to be processed; the eye image information to be processed includes 2 eye image information to be estimated;

[0006] Extracting feature information from the eye image information to be processed to obtain target marker feature information; the target marker feature information includes first target feature information and second target feature information;

[0007] The target marker feature information is processed for line of sight estimation to obtain target eye line of sight estimation information.

[0008] A second aspect of an embodiment of the present invention discloses a sight line estimation device based on a marker, the device comprising:

[0009] An acquisition module, used for acquiring eye image information to be processed; the eye image information to be processed includes two pieces of eye image information to be estimated;

[0010] A first processing module is used to extract feature information from the eye image information to be processed to obtain target marker feature information; the target marker feature information includes first target feature information and second target feature information;

[0011] The second processing module is used to perform sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information.

[0012] The third aspect of the present invention discloses another device for estimating sight lines based on a marker, the device comprising:

[0013] A memory storing executable program code;

[0014] a processor coupled to the memory;

[0015] The processor calls the executable program code stored in the memory to execute part or all of the steps in the marker-based line of sight estimation method disclosed in the first aspect of the embodiment of the present invention.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the marker-based line of sight estimation method disclosed in the first aspect of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flowchart of a method for estimating sight lines based on a marker disclosed in an embodiment of the present invention;

[0019] Figure 2 is a structural schematic diagram of a sight line estimation device based on a marker disclosed in an embodiment of the present invention;

[0020] Figure 3 It is a structural schematic diagram of another landmark-based sight line estimation device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0023] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] The present invention discloses a method and device for estimating sight lines based on a marker, which is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience. The following are detailed descriptions.

[0025] Embodiment 1

[0026] See also Figure 1 , Figure 1 is a flow chart of a method for estimating sight lines based on landmarks disclosed in an embodiment of the present invention. Figure 1 The described landmark-based sight line estimation method is applied to a management system, such as a local server or a cloud server for management, and the embodiments of the present invention do not limit this. Figure 1 As shown, the landmark-based sight line estimation method may include the following operations:

[0027] 101. Obtain eye image information to be processed.

[0028] In the embodiment of the present invention, the eye image information to be processed includes two eye image information to be estimated.

[0029] 102. Extract feature information from the eye image information to be processed to obtain target marker feature information.

[0030] In an embodiment of the present invention, the target marker characteristic information includes first target characteristic information and second target characteristic information.

[0031] 103. Perform sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information.

[0032] It should be noted that the eye image information to be processed can be obtained through an image acquisition device. Taking a common image acquisition device as an example, a built-in infrared light source of a head-mounted device is used as illumination, and data is collected through a near-eye infrared camera. The collected image is used for feature extraction in a subsequent feature extraction module.

[0033] It should be noted that the marker is pre-set in the smart contact lens, and the marker is in the form of three rings, that is, three rings are designed outside the pupil outline, which are white-black-white rings. Based on the fit between the contact lens and the human eye, the black ring marker part in the smart contact lens can be used as a new eye feature to be extracted, because the grayscale distribution of the black ring and the inner and outer white rings is uniform and the difference is obvious, which is more conducive to detection than using ordinary pupils as features, and can locate the center of the ring more quickly and accurately.

[0034] It should be noted that the marker-based line of sight estimation method of the present application develops a unique, easy-to-identify, and anti-interference marker design on smart glasses. By utilizing the fitting characteristics of contact lenses, markers can replace the traditional features of the human eye and are easier to detect; and compared to adding sensor modules, the application development cost is greatly reduced. By adopting an improved adaptive edge detection algorithm based on gradient entropy, after detecting edge segments of different shapes, the edge segment closest to the marker contour can be selected according to the entropy value in the gradient direction, so that even if the smart glasses marker is blocked by eyelashes, eyelids, etc. and is only partially exposed, the final ellipse can restore the most approximate contour by fitting. The combination of head-mounted devices and smart contact lenses, the combination of traditional eye feature detection algorithms and smart contact lenses.

[0035] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0036] In an optional embodiment, the above-mentioned step of extracting feature information from the eye image information to be processed to obtain target marker feature information includes:

[0037] For any to-be-estimated eye image information in the to-be-processed eye image information, performing feature region selection processing on the to-be-estimated eye image information to obtain target marker feature region information;

[0038] Feature information is extracted from the characteristic region information of the target marker to obtain the target marker characteristic information.

[0039] It should be noted that when the pupil is fully visible, pupil detection is a simple problem, but in the case of occlusion and tilt angle, pupil contour detection becomes difficult. In order to solve the problem of difficult detection, the present application first performs feature area selection processing on the estimated eye image information to replace the outer contour of the original pupil with the black ring of the smart glasses as the outer contour, and then extracts feature information from the target marker feature area information, thereby solving the problem that the traditional feature point detection algorithm is affected by the pupil being blocked by eyelashes or eyelids, the eyes being partially closed, and the pupil area not being fully exposed, and cannot effectively detect the pupil contour area through contour detection.

[0040] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0041] In another optional embodiment, the feature region selection process is performed on the eye image information to be estimated to obtain the target marker feature region information, including:

[0042] Pre-processing the eye image information to be estimated to obtain target eye image information;

[0043] The target eye image information is optimized and selected to obtain the target marker feature area information.

[0044] It should be noted that pre-processing the eye image information to be estimated can significantly reduce the computational burden and processing time, especially when the processed image resolution is high or a complex algorithm is required, so that information about the area can be analyzed and extracted more accurately.

[0045] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0046] In yet another optional embodiment, pre-processing the eye image information to be estimated to obtain target eye image information includes:

[0047] Performing grayscale processing on the eye image information to be estimated to obtain grayscale eye image information;

[0048] Binarizing the grayscale eye image information using a binarization threshold to obtain binarized eye image information;

[0049] Performing image dilation processing on the binary eye image information to obtain dilated eye image information;

[0050] The feature region frame information is used to perform cropping and selection processing on the expanded eye image information to obtain the target eye image information.

[0051] It should be noted that the grayscale processing of the eye image information to be estimated is to convert the image from a color image to a grayscale image.

[0052] It should be noted that the above binarization threshold can be any value between 100 and 225. Furthermore, after the binarization process, most of the interference information such as the white of the eye, iris, eyelashes, etc. is removed, and the most obvious feature of the black ring, that is, the pseudo pupil outline, can be highlighted.

[0053] It should be noted that the above-mentioned image dilation processing of the binary eye image information uses the dilation operation of OpenCV to enhance the characteristic area in the image, thereby preventing the threshold from being too low, resulting in a small or diluted grayscale value density in the characteristic area.

[0054] It should be noted that the above-mentioned cropping and selection processing of the expanded eye image information using the feature area frame information uses a frame of a specified area size (feature area frame information, such as a square frame of 6 pixels * 6 pixels) to cut out the area on the binarized image instead of the adaptive area frame, so as to more quickly calculate its position coordinates in the original image (target eye line of sight estimation information).

[0055] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0056] In yet another optional embodiment, the target eye image information is optimized and selected to obtain target marker feature area information, including:

[0057] Performing contour detection on the target eye image information to obtain image contour information; the image contour information includes a plurality of contour information to be processed;

[0058] For any to-be-processed contour information in the image contour information, confirm that the to-be-processed contour information is standby contour information;

[0059] Determine whether the spare contour information has parent contour information, and obtain a contour determination result; the parent contour information represents the smallest contour information to be processed that can cover the spare contour information;

[0060] When the contour judgment result is yes, determining that the parent contour information corresponding to the spare contour information is new spare contour information, and triggering execution to judge whether the spare contour information has parent contour information, to obtain the contour judgment result;

[0061] When the contour judgment result is no, determining the backup contour information as a first image contour information;

[0062] Repeatedly performing contour elimination on all first image contour information to obtain target image contour information; the target image contour information includes a plurality of second image contour information; the number of the second image contour information is not greater than the number of the first image contour information;

[0063] Perform contour area calculation processing on all second image contour information in the target image contour information to obtain image contour area information; the image contour area information includes a plurality of image contour area values;

[0064] Determine that the image area corresponding to the second image contour information corresponding to the largest image contour area value in the image contour area information is the target marker characteristic area information.

[0065] It should be noted that the above-mentioned contour detection of the target eye image information is implemented using findContours of OpenCV, thereby obtaining all contour areas distributed in a hierarchical manner, that is, image contour information.

[0066] It should be noted that the above-mentioned parent contour information represents the smallest contour information to be processed that can cover the spare contour information, and the area of ​​the parent contour information is the smallest contour information to be processed among all the spare contour information. For example, when the spare contour information is a circle with a center of (0,0) and a radius of 1, the contour information to be processed covering the spare contour information is, in sequence, a first circle with a center of (0,0) and a radius of 2, and a second circle with a center of (0,0) and a radius of 3, and the first circle is the parent contour information.

[0067] It should be noted that the above-mentioned contour area calculation processing for all second image contour information in the target image contour information can be implemented based on integration, which can be implemented according to any algorithm in the prior art, such as the cutting and patching method, and the embodiment of the present invention does not limit this.

[0068] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0069] In an optional embodiment, the above-mentioned extraction of characteristic information of the target marker characteristic region information to obtain the target marker characteristic information includes:

[0070] Performing edge segment detection on target marker feature area information to obtain marker edge feature information;

[0071] Perform near-circular segment detection and screening on the edge feature information of the marker to obtain the near-circular point set information of the marker;

[0072] Perform ellipse fitting on the information of the near-circular point set of the landmark to obtain the standard ellipse point set information;

[0073] Based on the standard ellipse point set information and the eye image information to be processed, the target marker feature information is determined.

[0074] It should be noted that the above-mentioned edge segment detection of the target marker feature area information is performed using a parameter-free edge segment detector. Furthermore, the parameter-free edge segment detector is a combination of ED and a reverse edge verification mechanism. Furthermore, since common edge detectors such as ED, EDlines, and EDCircle have several parameters, they must be adjusted by the user for different tasks and application scenarios, making the process more complicated. The parameter edge segment detector can achieve any type of image using a set of fixed parameters.

[0075] It should be noted that the above-mentioned detection and screening of near-circular segments of the edge feature information of the marker is implemented using a method based on gradient direction analysis. The direction of the gradient provides information about the direction of the edge. By calculating the gradient, the position and direction information of the edge can be better presented. First, the gradient direction and amplitude of each point in the edge segment are calculated, and 22.5° is used as the quantization angle. In order to calculate discrete symbols, the unit circle is divided into 16 regions and 8 different directions. Then, each gradient direction value in the calculation result of the previous step is looped through, and the histogram count of the corresponding direction is added by 1, and then the count value in the histogram is normalized to probability (that is, the frequency of each direction) to ensure that the sum of the histogram is 1. In order to calculate the entropy value of the histogram, the entropy function is used on the quantized gradient distribution of the segment: Where i is the sequence number of each direction, n is the total number of directions, and p is the probability of each direction. Since the entropy value of a perfect circular edge segment is the largest, as the segment shape is different from a circle (ellipse, etc.), the entropy of the final straight line becomes zero, thereby filtering out edge segments that produce small entropy values. And there may be near-circular light spots with large entropy values ​​as interference elements. We then calculate the length of each edge segment and filter out edge segments that are less than a given length threshold, and finally obtain the required contour edge segment of the marker, which is the desired feature contour. Further, the detected contour edge segments are sequentially used to obtain the marker near-circular point set information corresponding to the minimum interval point, that is, the marker near-circular point set information is composed of the minimum interval points corresponding to the detected contour edge segments. It should be noted that the distance of the above-mentioned minimum interval point can be set by the user or it can be a default value given by the system. Further, it can be a unit distance composed of millimeters.

[0076] It should be noted that the above-mentioned ellipse fitting of the marker near-circle point set information is to fit the minimum interval points corresponding to the detected contour edge segments into an ellipse, and then calculate the center coordinates and radius of the ellipse, and finally generate a complete circle, and then slice the standard ellipse points according to the distance of the minimum interval points to obtain the final standard ellipse point set information.

[0077] It should be noted that the above-mentioned method based on the standard ellipse point set information and the eye image information to be processed is to merge the black spots corresponding to the black circles in the eye image information to be estimated and the standard ellipse point set information to generate a new point set, and then fit a new ellipse again according to the new point set, and then obtain the first target feature information and the second target feature information corresponding to the center of the ellipse, thereby realizing the fitting of the point set and the updating of the shape, and more comprehensively describing the shape characteristics of the original data set.

[0078] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0079] In another optional embodiment, performing sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information includes:

[0080] Get estimated model coefficient information;

[0081] The sight line estimation model is used to calculate and process the estimation model coefficient information and the target landmark feature information to obtain the target eye sight line estimation information;

[0082] Among them, the line of sight estimation model is:

[0083]

[0084] Where X and Y are the estimated values ​​of horizontal and vertical sight lines in the target eye sight line estimation information, respectively; c0, c1, c2, c3, c4, d0, d1, d2, d3 and d4 are the estimated model coefficient values ​​in the estimated model coefficient information; A x and A y are respectively the first horizontal characteristic value and the first vertical characteristic value in the first target characteristic information in the target marker characteristic information; B x B y and are respectively the second transverse characteristic value and the second longitudinal characteristic value in the second target characteristic information in the target marker characteristic information.

[0085] In this optional embodiment, as an optional embodiment, the obtaining of the estimation model coefficient information includes:

[0086] Acquire sample set information; the sample set information includes a number of training sample information; each training sample information includes sample estimation value information and sample feature value information;

[0087] The training estimation model and the sample set information are fitted using the least squares method to obtain the target training estimation model; the expressions of the training estimation model and the sight estimation model are consistent;

[0088] For any training sample information, the target training estimation model is used to perform line of sight estimation processing on the sample feature value information in the training sample information to obtain training estimation value information;

[0089] Calculate the distance between the training estimated value information and the sample estimated value information to obtain the training estimated value;

[0090] Determine whether there is a training estimation value greater than a training estimation threshold among all training estimation values, and obtain a training estimation result;

[0091] When the training judgment result is yes, the training sample information corresponding to the training estimation value whose training estimation value is greater than the training estimation threshold is deleted from the sample set information, and the least square method is triggered to perform fitting processing on the training estimation model and the sample set information to obtain the target training estimation model;

[0092] When the training judgment result is no, the model coefficient value corresponding to the target training estimation model is determined as the estimation model coefficient information.

[0093] It should be noted that the above training estimation threshold may be input by the user or may be a default value automatically generated by the system. Further, the training estimation threshold is 0.01.

[0094] It should be noted that the above sample estimation value information is the horizontal sample estimation value and the vertical sample estimation value corresponding to the horizontal sight estimation value and the vertical sight estimation value in the target eye sight estimation information. Further, calculating the distance value between the training estimation value information and the sample estimation value information is to directly calculate the distance between the coordinates corresponding to the sample estimation value information and the coordinates corresponding to the training estimation value information.

[0095] It should be noted that the above training sample information is obtained by users wearing personalized smart contact lenses with added feature points and using head-mounted AR devices to calibrate the calibration points. These calibration points are distributed in different areas of the screen, covering the entire possible field of view of the user. The user is guided to look at each calibration point in turn. When the user looks at each calibration point, the system collects the corresponding eye movement data, that is, the multi-frame eye diagrams of the user in the corresponding time period of each calibration point. The sample feature value information can be obtained by estimating the position of the center point of the multi-frame eye diagram, and the position information of the calibration point is the sample estimated value information.

[0096] It should be noted that the training sample information corresponding to the training estimation value whose training estimation value is greater than the training estimation threshold is deleted from the sample set information, and the least squares method is used to fit the training estimation model and the sample set information, so that the target training estimation model is obtained by eliminating some non-ideal samples, such as the user's distraction during the calibration process, thereby further improving the accuracy of line of sight estimation.

[0097] It can be seen that implementing the marker-based line of sight estimation method described in the embodiment of the present invention is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0098] Embodiment 2

[0099] See also Figure 2 , Figure 2 is a schematic diagram of a structure of a sight line estimation device based on a landmark disclosed in an embodiment of the present invention. Figure 2 The described device can be applied to a warehouse management system, such as a local server or cloud server for warehouse logistics management based on line of sight estimation of landmarks, etc., and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the device may include:

[0100] An acquisition module 201 is used to acquire eye image information to be processed; the eye image information to be processed includes two eye image information to be estimated;

[0101] The first processing module 202 is used to extract feature information from the eye image information to be processed to obtain target marker feature information; the target marker feature information includes first target feature information and second target feature information;

[0102] The second processing module 203 is used to perform sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information.

[0103] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0104] In another optional embodiment, Figure 2 As shown, the first processing module 202 extracts feature information from the eye image information to be processed to obtain target marker feature information, including:

[0105] For any to-be-estimated eye image information in the to-be-processed eye image information, performing feature region selection processing on the to-be-estimated eye image information to obtain target marker feature region information;

[0106] Feature information is extracted from the characteristic region information of the target marker to obtain the target marker characteristic information.

[0107] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0108] In yet another optional embodiment, Figure 2 As shown, the first processing module 202 performs feature region selection processing on the eye image information to be estimated to obtain target marker feature region information, including:

[0109] Pre-processing the eye image information to be estimated to obtain target eye image information;

[0110] The target eye image information is optimized and selected to obtain the target marker feature area information.

[0111] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0112] In yet another optional embodiment, Figure 2 As shown, the first processing module 202 performs pre-processing on the eye image information to be estimated to obtain target eye image information, including:

[0113] Performing grayscale processing on the eye image information to be estimated to obtain grayscale eye image information;

[0114] Binarizing the grayscale eye image information using a binarization threshold to obtain binarized eye image information;

[0115] Performing image dilation processing on the binary eye image information to obtain dilated eye image information;

[0116] The feature region frame information is used to perform cropping and selection processing on the expanded eye image information to obtain the target eye image information.

[0117] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0118] In yet another optional embodiment, Figure 2 As shown, the first processing module 202 optimizes and selects the target eye image information to obtain target marker feature area information, including:

[0119] Performing contour detection on the target eye image information to obtain image contour information; the image contour information includes a plurality of contour information to be processed;

[0120] For any to-be-processed contour information in the image contour information, confirm that the to-be-processed contour information is standby contour information;

[0121] Determine whether the spare contour information has parent contour information, and obtain a contour determination result; the parent contour information represents the smallest contour information to be processed that can cover the spare contour information;

[0122] When the contour judgment result is yes, determining that the parent contour information corresponding to the spare contour information is new spare contour information, and triggering execution to judge whether the spare contour information has parent contour information, to obtain the contour judgment result;

[0123] When the contour judgment result is no, determining the backup contour information as a first image contour information;

[0124] Repeatedly performing contour elimination on all first image contour information to obtain target image contour information; the target image contour information includes a plurality of second image contour information; the number of the second image contour information is not greater than the number of the first image contour information;

[0125] Perform contour area calculation processing on all second image contour information in the target image contour information to obtain image contour area information; the image contour area information includes a plurality of image contour area values;

[0126] Determine that the image area corresponding to the second image contour information corresponding to the largest image contour area value in the image contour area information is the target marker characteristic area information.

[0127] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0128] In yet another optional embodiment, Figure 2 As shown, the first processing module 202 extracts characteristic information from the characteristic region information of the target marker to obtain characteristic information of the target marker, including:

[0129] Performing edge segment detection on target marker feature area information to obtain marker edge feature information;

[0130] Perform near-circular segment detection and screening on the edge feature information of the marker to obtain the near-circular point set information of the marker;

[0131] Perform ellipse fitting on the information of the near-circular point set of the landmark to obtain the standard ellipse point set information;

[0132] Based on the standard ellipse point set information and the eye image information to be processed, the target marker feature information is determined.

[0133] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0134] In yet another optional embodiment, Figure 2 As shown, the second processing module 203 performs sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information, including:

[0135] Get estimated model coefficient information;

[0136] The sight line estimation model is used to calculate and process the estimation model coefficient information and the target landmark feature information to obtain the target eye sight line estimation information;

[0137] Among them, the line of sight estimation model is:

[0138]

[0139] Where X and Y are the estimated values ​​of horizontal and vertical sight lines in the target eye sight line estimation information, respectively; c0, c1, c2, c3, c4, d0, d1, d2, d3 and d4 are the estimated model coefficient values ​​in the estimated model coefficient information; A x and A y are respectively the first horizontal characteristic value and the first vertical characteristic value in the first target characteristic information in the target marker characteristic information; Bx B y and are respectively the second horizontal characteristic value and the second vertical characteristic value in the second target characteristic information in the target marker characteristic information.

[0140] It can be seen that implementation Figure 2 The described landmark-based line of sight estimation device is conducive to accurately locating the eye position, improving the alignment and tracking of virtual objects, and enhancing the user's eye device usage experience.

[0141] Embodiment 3

[0142] See also Figure 3 , Figure 3 is a schematic diagram of the structure of another device for estimating sight lines based on a marker disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a warehouse management system, such as a local server or cloud server for warehouse logistics management based on line of sight estimation of landmarks, etc., and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include:

[0143] A memory 301 storing executable program codes;

[0144] a processor 302 coupled to the memory 301;

[0145] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the marker-based line of sight estimation method described in the first embodiment.

[0146] Embodiment 4

[0147] An embodiment of the present invention discloses a computer-readable storage medium, which stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps in the landmark-based line of sight estimation method described in the first embodiment.

[0148] Embodiment 5

[0149] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the marker-based line of sight estimation method described in Example 1.

[0150] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0151] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0152] Finally, it should be noted that the method and device for line of sight estimation based on markers disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical schemes described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the embodiments of the present invention.

Claims

1. A method for estimating sight lines based on a landmark, characterized in that: The method comprises: Acquire eye image information to be processed; the eye image information to be processed includes 2 eye image information to be estimated; Extracting feature information from the eye image information to be processed to obtain target marker feature information; the target marker feature information includes first target feature information and second target feature information; The target marker feature information is processed for line of sight estimation to obtain target eye line of sight estimation information.

2. The method for estimating sight lines based on landmarks according to claim 1, characterized in that: The step of extracting characteristic information from the eye image information to be processed to obtain characteristic information of the target marker includes: For any of the eye image information to be estimated in the eye image information to be processed, performing feature region selection processing on the eye image information to be estimated to obtain target marker feature region information; Feature information is extracted from the target marker feature area information to obtain target marker feature information.

3. The method for estimating sight lines based on landmarks according to claim 2, characterized in that: The step of performing feature region selection processing on the eye image information to be estimated to obtain target marker feature region information includes: Pre-processing the eye image information to be estimated to obtain target eye image information; The target eye image information is optimized and selected to obtain target marker feature area information.

4. The method for estimating sight lines based on landmarks according to claim 3, characterized in that: The pre-processing of the eye image information to be estimated to obtain target eye image information includes: Performing grayscale processing on the eye image information to be estimated to obtain grayscale eye image information; Binarizing the grayscale eye image information using a binarization threshold to obtain binarized eye image information; Performing image dilation processing on the binary eye image information to obtain dilated eye image information; The expanded eye image information is cropped and selected using the feature region frame information to obtain target eye image information.

5. The method for estimating sight lines based on landmarks according to claim 3, characterized in that: The step of optimizing and selecting the target eye image information to obtain target marker feature area information includes: Performing contour detection on the target eye image information to obtain image contour information; the image contour information includes a plurality of contour information to be processed; For any of the to-be-processed contour information in the image contour information, confirm that the to-be-processed contour information is standby contour information; Determine whether the spare contour information has parent contour information, and obtain a contour determination result; the parent contour information represents the minimum contour information to be processed that can cover the spare contour information; When the contour judgment result is yes, determining that the parent contour information corresponding to the standby contour information is the new standby contour information, and triggering the execution of the judgment of whether the standby contour information has parent contour information to obtain a contour judgment result; When the contour judgment result is no, determining that the backup contour information is a first image contour information; Repeatedly performing contour elimination on all the first image contour information to obtain target image contour information; the target image contour information includes a plurality of second image contour information; the number of the second image contour information is not greater than the number of the first image contour information; Performing contour area calculation processing on all the second image contour information in the target image contour information to obtain image contour area information; the image contour area information includes a plurality of image contour area values; Determine that the image area corresponding to the second image contour information corresponding to the largest image contour area value in the image contour area information is the target marker characteristic area information.

6. The method for estimating sight lines based on landmarks according to claim 2, characterized in that: The step of extracting characteristic information from the characteristic region information of the target marker to obtain characteristic information of the target marker includes: Performing edge segment detection on the target marker feature area information to obtain marker edge feature information; Performing near-circular segment detection and screening on the edge feature information of the marker to obtain near-circular point set information of the marker; Performing ellipse fitting on the near-circle point set information of the marker to obtain standard ellipse point set information; Based on the standard ellipse point set information and the eye image information to be processed, target marker feature information is determined.

7. The method for estimating sight lines based on landmarks according to claim 1, characterized in that: The performing sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information includes: Get estimated model coefficient information; Using a sight line estimation model to calculate and process the estimation model coefficient information and the target marker feature information to obtain target eye sight line estimation information; Wherein, the line of sight estimation model is: Wherein, X and Y are the estimated value of the horizontal sight line and the estimated value of the vertical sight line in the target eye sight line estimation information respectively; c0, c1, c2, c3, c4, d0, d1, d2, d3 and d4 are the estimated model coefficient values ​​in the estimated model coefficient information; A x and A y are respectively the first transverse characteristic value and the first longitudinal characteristic value in the first target characteristic information in the target marker characteristic information; B x B y and are respectively the second transverse characteristic value and the second longitudinal characteristic value in the second target characteristic information in the target marker characteristic information.

8. A sight line estimation device based on a landmark, characterized in that: The device comprises: An acquisition module, used for acquiring eye image information to be processed; the eye image information to be processed includes two pieces of eye image information to be estimated; A first processing module is used to extract feature information from the eye image information to be processed to obtain target marker feature information; the target marker feature information includes first target feature information and second target feature information; The second processing module is used to perform sight line estimation processing on the target marker feature information to obtain target eye sight line estimation information.

9. A sight line estimation device based on a landmark, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the marker-based line of sight estimation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the landmark-based line-of-sight estimation method as described in any one of claims 1-7.

Images