A target positioning method and device based on eye movement data

Abstract

The application discloses a target positioning method and device based on eye movement data, and the method comprises the following steps: acquiring information of all eye movement data points in an observation interface; classifying the eye movement data points according to the position information and collection time of each eye movement data point to obtain target eye movement data points belonging to a fixation behavior; determining a color region corresponding to the target eye movement data points in the observation interface by using a preset Gaussian function, and determining a first target region according to the distance of the color region; inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; comparing the first target region with the second target regions contained in the observation interface, and determining a target positioning region of the observation interface according to the comparison result; and the application can accurately and efficiently position a target region in the observation interface based on eye movement data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a target positioning method and device based on eye movement data. BACKGROUND

[0002] In recent years, neural networks have been widely used in target detection, but they face challenges such as occlusion, incomplete targets, and insufficient clarity in complex environments. In contrast, the human visual system has strong perception and cognition capabilities, which can quickly adjust the focal length and adapt to complex backgrounds, effectively making up for the missed and false detections of mechanical equipment. Researchers have gradually introduced the advantages of human vision into computer vision to improve the accuracy of target positioning by combining eye movement information.

[0003] The existing target positioning method generally includes two kinds. The first kind is to combine eye movement features and electroencephalogram to identify the electroencephalogram features induced by the user's visual attention area, and to obtain target position information based on eye movement features to improve the user's efficiency in collecting information in complex backgrounds. However, the electroencephalogram device not only needs to be installed and adjusted by professionals, but also has a complex data acquisition process and is easily affected by external environmental interference, resulting in a decline in signal quality and poor accuracy of target detection. The second kind is a non-verbal communication framework based on eye gaze, in which the user's eye gaze movement is actively tracked and analyzed to infer the user's intention in daily life activities based on the relationship between visual behavior and psychological processes. However, this method is easily affected by environmental factors and user subjective intentions.

[0004] Therefore, how to improve the accuracy of target positioning has become a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a target positioning method and device based on eye movement data, which can accurately and efficiently locate the target position of the gaze in the observation interface based on eye movement data.

[0006] According to one aspect of the present application, a target positioning method based on eye movement data is provided, which comprises:

[0007] Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and collection time;

[0008] Classifying and processing the eye movement data points based on the position information and collection time of each eye movement data point to obtain target eye movement data points belonging to gaze behavior;

[0009] determining a color region corresponding to the target eye movement data point in the observation interface by a preset Gaussian function, and determining a first target region according to a distance of the color region; wherein the first target region represents a region of interest of the observation interface for the user, and each first target region contains different amounts of target eye movement data points;

[0010] inputting the observation interface into the trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target region represents a potential region of interest of the observation interface for the user;

[0011] comparing the first target region and the second target region contained in the observation interface, and determining a target positioning region of the observation interface through the comparison result.

[0012] Optionally, the target eye movement data points belonging to the fixation behavior are obtained by classifying the eye movement data points according to the position information and the collection time of each eye movement data point, including:

[0013] calculating the Euclidean distance between the eye movement data point and the previous eye movement data point in time sequence, and determining whether the Euclidean distance is greater than a preset distance threshold;

[0014] if yes, determining the eye movement data point in the fixation event as a fixation end point, and determining the eye movement data point not in the fixation event as a saccade point;

[0015] if no, determining whether there is a fixation event that has started at the collection time of the eye movement data point, and determining the eye movement data point corresponding to the fixation event that has not started as a fixation start point;

[0016] determining the fixation start point, the fixation end point, and the eye movement data point at the fixation start point and the fixation end point as a fixation point;

[0017] determining the fixation point and the fixation start point of the fixation event as a target eye movement data point if the time interval between the fixation point and the fixation start point is greater than a preset time threshold.

[0018] Optionally, the color region corresponding to the target eye movement data point in the observation interface is determined by a preset Gaussian function, including:

[0019] creating a white graph with the same size as the observation interface, and marking each target eye movement data point on the white graph according to the position information;

[0020] sequentially traversing the target eye movement data points on the white graph, and creating a two-dimensional Gaussian template corresponding to the target eye movement data point by a preset Gaussian function, wherein the two-dimensional Gaussian template contains a plurality of weight values;

[0021] Superimpose the two-dimensional Gaussian template on the corresponding target eye movement data point according to the position information, perform pseudo-color processing on the white diagram through the weight value, superimpose the processed image on the observation interface, and obtain a color region corresponding to the target eye movement data point in the observation interface.

[0022] Optionally, the determining the first target region according to the distance of the color region comprises:

[0023] respectively calculating a boundary distance between different color regions, and determining whether the boundary distance is less than or equal to a preset threshold value;

[0024] If yes, it is determined that the corresponding color region belongs to the same first target region, and a rectangular frame is used to frame the first target region;

[0025] If no, it is determined that the corresponding color region belongs to different first target regions, and a rectangular frame is used to frame the first target regions respectively.

[0026] Optionally, the comparing the first target region and the second target region included in the observation interface, and determining the target positioning region of the observation interface through the comparison result comprises:

[0027] determining whether an intersection-union ratio of the first target region and the second target region is greater than a preset similarity threshold value;

[0028] If yes, the first target region and the second target region in the observation interface are both taken as the target positioning region of the observation interface;

[0029] If no, the second target region in the observation interface is taken as the target positioning region of the observation interface.

[0030] Optionally, the method further comprises:

[0031] calculating the target positioning region through a preset attention degree formula to obtain an importance index of the target eye movement data point; wherein the preset attention degree formula is: R represents all target positioning regions, represents a fixation number of the i th target positioning region, represents an eye movement feature index of the i th target positioning region, represents a heat map value of the i th target positioning region, represents a proactive interaction value of the i th target positioning region.

[0032] Optionally, the method further comprises:

[0033] In response to an updating operation of the eye movement data in the observation interface, a new eye movement data point is obtained, and a target eye movement data point and a color region corresponding to the target eye movement data point are determined through the new eye movement data point;

[0034] A boundary distance of a color region corresponding to the current target eye movement data point and a color region of a previous target positioning region is calculated, and a common display radius of a corresponding color region with a boundary distance less than a first preset threshold is determined;

[0035] A target positioning region to which a corresponding color region with a common display radius greater than a second preset threshold belongs is determined as a target positioning region with a maximum attention degree.

[0036] To achieve the above-mentioned purpose, the present application further provides a target positioning device based on eye movement data, the device comprising:

[0037] An acquisition module is configured to acquire information of all eye movement data points in an observation interface; wherein the information of the eye movement data points comprises position information and acquisition time;

[0038] A classification module is configured to perform classification processing on the eye movement data points through the position information and the acquisition time of each eye movement data point, to obtain target eye movement data points belonging to a fixation behavior;

[0039] A determination module is configured to determine a color region corresponding to a target eye movement data point in the observation interface through a preset Gaussian function, and determine a first target region according to a distance of the color region; wherein the first target region represents an attention region of the observation interface of a user, and each first target region contains different numbers of target eye movement data points;

[0040] A detection module is configured to input the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target region represents a potential region of interest of the observation interface of the user;

[0041] A comparison module is configured to compare the first target region and the second target region contained in the observation interface, and determine a target positioning region of the observation interface through a comparison result.

[0042] To achieve the above-mentioned purpose, the present application further provides a computer device, which specifically comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the target positioning method based on eye movement data when executing the computer program.

[0043] In order to achieve the above object, the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the target positioning method based on eye movement data.

[0044] The target positioning method and device based on eye movement data provided by the application can obtain target eye movement data points by classifying eye movement data points, determine the color region corresponding to the target eye movement data points by using a Gaussian function, compare the target region formed by the color region with the target region obtained by a target detection model, and determine the final target positioning region. The detection result of the target detection algorithm is combined with the region obtained by the eye movement data to compensate for the missed detection problem of the target detection algorithm in a complex scene, improve the accuracy of target positioning, and significantly reduce the missed detection rate. BRIEF DESCRIPTION OF DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the application. Moreover, the same reference numerals are used throughout the various drawings to designate identical parts. In the drawings:

[0046] Figure 1 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0047] Figure 2 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0048] Figure 3 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0049] Figure 4 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0050] Figure 5 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0051] Figure 6 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0052] Figure 7 An optional flowchart of the target positioning method based on eye movement data provided for example one;

[0053] Figure 8 An optional hardware structure diagram of the computer device provided for example three. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0055] Embodiment one

[0056] The embodiment of the present application provides a target positioning method based on eye movement data, as shown in the figure, the method specifically comprises the following steps: Figure 1

[0057] S101: acquiring information of all eye movement data points in an observation interface; wherein the information of the eye movement data points comprises position information and acquisition time.

[0058] In the embodiment, the observation interface is a continuously changing interface in any video observed by a user through a screen; the user uses a high-precision eye movement tracking device to capture the pixel position of the line of sight falling in the observation interface in real time, and records the change of the pupil size of the user at the time of observation. The eye movement data point is the pixel position seen by the user's eyes on the observation interface. The information of each eye movement data point comprises position information, acquisition time and pupil size.

[0059] Specifically, after acquiring the information of all eye movement data points in the observation interface in step S101, the method further comprises:

[0060] Step A1: mapping the pixel values of all eye movement data points into angle values;

[0061] Specifically, the eye movement data captured by the eye movement tracking device is usually quantified in pixels, but the pixel value itself does not have physical meaning, and due to the difference in the size of the observation interface and the distance of the observer, it is necessary to convert the information of all eye movement data points from pixel coordinates to visual angle degrees. Thus, through the mapping process, the unit inconsistency between the pixel value and the actual physical coordinates can be eliminated.

[0062] Step A2: inputting the information of all eye movement data points into a Kalman filtering algorithm for smoothing processing to obtain the smoothed information of all eye movement data points.

[0063] In the embodiment, the eye movement tracking device is affected by noise in the environment when capturing data in real time, and through the use of the Kalman filtering algorithm, the problem of jitter of eye movement data is eliminated, and the processing efficiency is improved.

[0064] ​S102: Classify the eye movement data points according to the position information and the collection time of each eye movement data point to obtain target eye movement data points belonging to the gaze behavior.

[0065] When the line of sight moves on the observation interface, there is a period of time in which the line of sight is relatively stable, and the eye movement behavior in this period of time is referred to as a gaze behavior, and the eye movement data points in the gaze behavior are referred to as gaze points. There is also a behavior in which the line of sight rapidly jumps, and the eye movement data points in this behavior are referred to as saccade points. The gaze points and the saccade points in the eye movement data points can be distinguished by a preset classification algorithm.

[0066] It should be noted that the classification operation of the eye movement data points can be implemented by a preset classification algorithm, and the preset classification algorithm is an I-VT algorithm (Velocity-Threshold Identification) or an I-DT algorithm (Dispersion-Threshold Identification).

[0067] Specifically, the classification of the eye movement data points in the step S102 according to the position information and the collection time of each eye movement data point to obtain target eye movement data points belonging to the gaze behavior includes:

[0068] Step B1: Calculate the Euclidean distance between the eye movement data points and the previous eye movement data points in time sequence, and determine whether the Euclidean distance is greater than a preset distance threshold;

[0069] The Euclidean distance between the eye movement data points and the previous eye movement data points is the angle difference between the two eye movement data points. In particular, when the first eye movement data point is input, the Euclidean distance between the first eye movement data point and the previous eye movement data point is 0.

[0070] Step B2: If yes, determine that the eye movement data point in the gaze event is a gaze end point, and determine that the eye movement data point not in the gaze event is a saccade point;

[0071] The gaze event represents a gaze behavior, and includes a gaze start point, a gaze end point, and eye movement data points between the gaze start point and the gaze end point.

[0072] Step B3: If no, determine whether there is a started gaze event at the collection time of the eye movement data point, and determine that the eye movement data point corresponding to the non-started gaze event is a gaze start point;

[0073] Step B4: Determine that the gaze start point, the gaze end point, and the eye movement data points at the collection time of the gaze start point and the gaze end point are gaze points;

[0074] Furthermore, the step S102 of classifying the eye movement data points according to the position information and the acquisition time of each eye movement data point to obtain the target eye movement data point belonging to the gaze behavior further includes:

[0075] Step B5: Determine whether there is a gaze event that has started at the time of collecting the last eye movement data point;

[0076] Step B6: If yes, then determine the last eye movement data point as the gaze ending point; if no, then determine the last eye movement data point as the saccade point.

[0077] In this embodiment, if Figure 2 As shown, the location information and acquisition time of eye movement data points are input into the system of the preset classification algorithm in chronological order. Each eye movement data point is then evaluated sequentially. First, the system determines whether the currently analyzed eye movement data point is the last one. If not, the Euclidean distance between the current eye movement data point and the previous eye movement data point is calculated and whether this distance is greater than a preset distance threshold. If it is greater than the preset distance threshold and the user is in the process of fixating, the current eye movement data point's information is added to the "End" (Efix) list and processing continues with the next eye movement data point. If it is greater than the preset distance threshold and the user is not in the process of fixating, processing continues with the next eye movement data point. If it is less than or equal to the preset distance threshold and the user is not in the process of fixating, processing continues with the next eye movement data point. If it is less than or equal to the preset distance threshold and the user is in the process of fixating, processing continues with the next eye movement data point. When the last eye movement data point is input, if the user is in the process of fixating, the last eye movement data point's information is added to the "End" (Efix) list. Finally, the start Sfix list and the end Efix list are output.

[0078] The Sfix list and Efix list record multiple eye movement data points that represent fixation behaviors, and the position of the fixation start point of each fixation behavior in the Sfix list corresponds to the position of the fixation end point of each fixation behavior in the Efix list. The fixation start point, fixation end point, and eye movement data points between the fixation start point and the fixation end point of each fixation behavior are all fixation points. Eye movement data points that are not in the fixation behavior are saccade points, and the types of all eye movement data points are labeled.

[0079] In addition, the I-VT algorithm can also be used to identify eye movement data points representing gaze behavior and eye movement data points representing scan behavior through a speed threshold, where the speed threshold is the quotient of the Euclidean distance and the time difference.

[0080] Step B7: taking a gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold as a target eye movement data point.

[0081] In this embodiment, if Figure 3 As shown, all classified eye movement data points are filtered again: input eye movement data points one by one to determine whether the previous eye movement data point of the currently analyzed eye movement data point is a fixation point. If not, input the next eye movement data point; if so, calculate whether the time difference between the current eye movement data point and the fixation start point of the current fixation behavior is greater than the preset time threshold. When the time difference is greater than the preset time threshold, the current eye movement data point is added to the heat map drawing queue; when the time difference is less than or equal to the preset time threshold, input the next eye movement data point. The eye movement data points in the heat map drawing queue are the target eye movement data points. By classifying the eye movement data points and filtering out the scan points and short fixation points, the user's focus is more accurately reflected, highlighting the importance of the fixation point.

[0082] Preferably, the preset distance threshold may be 30°, and the preset time threshold may be 200 milliseconds.

[0083] S103: Determine the color area corresponding to the target eye movement data point in the observation interface through a preset Gaussian function, and determine the first target area based on the distance of the color area; wherein the first target area represents the user's focus area on the observation interface, and each first target area contains a different number of target eye movement data points.

[0084] In this embodiment, a two-dimensional Gaussian function is used to calculate the target eye movement data points in the heat map drawing queue. Each target eye movement data point corresponds to a color region. The color regions corresponding to different target eye movement data points may overlap. When the distance between different color regions is less than a preset value, a rectangular frame is used to select the corresponding color regions to obtain a first target region. This first target region represents the user's focus area on the viewing interface.

[0085] Specifically, the determining of the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function in step S103 includes:

[0086] Step C1: creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to position information;

[0087] Step C2: traversing the target eye movement data points on the white image in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes multiple weight values;

[0088] Step C3: superimpose the two-dimensional Gaussian template on the corresponding target eye movement data point according to the position information, pseudo-color process the white graph through the weight value, and superimpose the processed image on the observation interface to obtain the color region corresponding to the target eye movement data point in the observation interface.

[0089] In this embodiment, the color region generated according to the target eye movement data point is a fixation heat map. A white graph with the same size as the observation interface and a gray value of 0 is created, and the target eye movement data points in the heat map drawing queue are marked on the white graph according to the position information of the target eye movement data points in the heat map drawing queue. A two-dimensional Gaussian function can calculate a two-dimensional Gaussian template corresponding to each target eye movement data point and assign a weight value to the Gaussian template. Each Gaussian template is superimposed on the corresponding target eye movement data point, and the center of each Gaussian template is at the position of the corresponding target eye movement data point, that is, the weight at the target eye movement data point is the highest, and the weight value gradually decreases on the Gaussian template as the distance from the target eye movement data point increases. The white graph is pseudo-color processed, and the region where each target eye movement data point corresponds to the Gaussian template will form a colored region, and the color will change with the different weight values. By assigning weights to the region corresponding to the target eye movement data point through the Gaussian template and forming a color region according to the weight value, the accuracy of target positioning is improved.

[0090] Specifically, the step S103 includes:

[0091] Step D1: calculate the boundary distance between different color regions respectively, and determine whether the boundary distance is less than or equal to a preset threshold value;

[0092] Step D2: if yes, it is determined that the corresponding color region belongs to the same first target region, and a rectangular frame is used to frame the first target region;

[0093] Step D3: if no, it is determined that the corresponding color region belongs to different first target regions respectively, and a rectangular frame is used to frame the first target regions respectively.

[0094] In this embodiment, the color regions formed by adjacent target eye movement data points may intersect, the boundary distance between different color regions is calculated, when the boundary distance is less than or equal to a preset threshold value, a large region formed by multiple different intersecting color regions is divided into the same first target region and framed by a rectangular frame; when the boundary distance is greater than the preset threshold value, multiple different color regions are respectively divided into separate first target regions, and each first target region is framed by a rectangular frame.

[0095] Further, after the step D3, it further includes:

[0096] Step E1: frame the color region with a rectangular frame to obtain a detection frame;

[0097] Step E2: calculate the mean value of all pixel values in the detection frame, and determine whether the mean value is greater than a preset boundary threshold;

[0098] Step E3: if yes, set the region in the detection frame as a high attention region, and mark the high attention region as a first target region;

[0099] Step E4: if no, set the region in the detection frame as a low attention region.

[0100] In this embodiment, the color region is framed with a rectangular frame to obtain a detection frame. To reduce the amount of calculation, the sum of non-zero pixel values in the detection frame is calculated and the mean value is calculated, that is, the sum of colored pixels in the detection frame is calculated and the mean value is calculated. In particular, the boundary threshold is twice the mean value of the current detection frame. Determine whether the mean value in the detection frame is greater than the boundary threshold, if no, the detection frame is a low attention region; if yes, the detection frame is a high attention region, and the high attention region is marked as a first target region.

[0101] S104: input the observation interface into the trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target region represents a potential region of interest of the user for the observation interface.

[0102] In this embodiment, the entire observation interface is detected using the trained target detection model to obtain an observation interface containing a target detection frame. The second target region is the region in the detected target detection frame, which represents the user's potential region of interest.

[0103] Preferably, the trained target detection model is a YOLOv5 detection model (You Only Look Once version 5).

[0104] S105: compare the first target region and the second target region contained in the observation interface to determine the target positioning region of the observation interface through the comparison result.

[0105] Specifically, the comparison of the first target region and the second target region contained in the observation interface in step S105 to determine the target positioning region of the observation interface through the comparison result comprises:

[0106] Step F1: determine whether the intersection over union of the first target region and the second target region is greater than a preset similarity threshold;

[0107] Step F2: If yes, both the first target region and the second target region in the observation interface are positioned as the target positioning region of the observation interface.

[0108] Step F3: If no, the second target region in the observation interface is positioned as the target positioning region of the observation interface.

[0109] In this embodiment, the first target region formed by the eye movement data in the observation interface is compared with the second target region obtained by the YOLOv5 detection model, and the similarity is measured by the intersection over union. If the intersection over union is greater than a preset similarity threshold, the second target region replaces the first target region and is recorded as the target positioning region. If the intersection over union is less than the similarity threshold, the second target region is retained while the first target region is also retained, and the final target positioning region is obtained. The high attention region positioned by the eye movement data is retained by the intersection over union, which makes up for the risk of missed detection.

[0110] Preferably, the method further comprises:

[0111] The importance index of the target eye movement data point is obtained by calculating the target positioning region by a preset attention degree formula, wherein the preset attention degree formula is: R represents all target positioning regions, characterizes the number of gazes of the i-th target positioning region, characterizes the eye movement feature index of the i-th target positioning region, characterizes the heat map value of the i-th target positioning region, characterizes the active interaction value of the i-th target positioning region.

[0112] In this embodiment, the attention degree of the target positioning region detected in the observation interface is calculated to obtain the importance degree of different target positioning regions. The statistical indicators such as gaze, saccade and pupil change significantly under different emotional states, and 29 eye movement features covering gaze, saccade and pupil change are selected to calculate the target attention degree, including: number of gazes, gaze frequency, average gaze duration, gaze duration standard deviation, first gaze time, first gaze duration, number of saccades, saccade frequency, average saccade duration, saccade duration standard deviation, saccade duration kurtosis, saccade duration skewness, average saccade amplitude, saccade amplitude standard deviation, saccade amplitude kurtosis, saccade amplitude skewness, average saccade speed, saccade speed standard deviation, saccade speed kurtosis, saccade speed skewness, saccade amplitude peak value, saccade speed peak value, average pupil diameter, pupil diameter standard deviation, pupil diameter skewness, pupil diameter kurtosis, pupil activity index (IPA).

[0113] The target attention degree calculation formula is: , R represents all target positioning areas, and i represents the current target positioning area.

[0114] Specifically, , Indicates the number of times the current target is fixed on, which increases cumulatively. When the fixation point enters and leaves the target area, the number of times the current target is fixed on is accumulated plus 1. For normalization, the number of times the current target is fixed on is divided by the number of times all targets are fixed on as part of the attention degree score.

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122] in, Is a stable updated value. Only when a gaze point enters the target area and leaves, the current target It will be updated, otherwise it will remain unchanged. It includes two parts. The first part is the inner activity results calculated based on the emotion and cognitive load theory. The importance of random forest prediction is obtained when the subject looks at the target. There are three results: 1, 2, 3, The score representing the cognitive load of the subject's gaze on the target is mainly represented by the pupil activity index. For normalization, it is processed using the sigmoid function. The second part is a characteristic index calculated based on the eye movement characteristics. The fixation index, saccade index, and pupil index are calculated separately, and the sum of the three is used as the characteristic index of the second part; represents the mean, Represents variance.

[0123] Calculated from the heat map, since the heat map updates quickly, The time changes, and close to the current time, the greater the value, the latest attention area can have a greater attention score, facilitating the update of the attention degree of each target. In order to obtain a more accurate potential sensitive area index, a quantitative method is adopted. First, the heat map data in each rectangular frame is summed, and then divided by the number of all pixel points contained in the frame, thereby obtaining the hotspot mean value of the frame. Secondly, the hotspot mean value of each frame is divided by the mean value of the entire heat map at the current time, to serve as an index for quantifying the attention degree of the area in the frame. The specific calculation is shown in the following calculation formula:

[0124]

[0125]

[0126]

[0127]

[0128] wherein k represents a pixel in the current target positioning area, j represents a Gaussian function, represents the intensity of the entire interface heat map, represents the heat map intensity of each outlined potential sensitive area, represents the heat map score of the potential sensitive area.

[0129]

[0130] Only the current target is actively selected to be set to the maximum attention, and the rest of the targets are all 0.

[0131] After obtaining the final importance index, all target positioning areas are sorted. According to the sorting result, the target with the highest importance is set to 1, and the importance of the remaining targets decreases in turn, and is set according to the relative gap with the target with the maximum importance.

[0132]

[0133] Through calculation, the attention value for measuring the relative importance of each target positioning area in the entire observation interface can be obtained. And according to the calculated attention value, all target positioning areas are sorted. As shown in Figure 4 Figure 5 As shown in the figure, each target positioning area in the observation interface is labeled with attention.

[0134] Preferably, the method further comprises:

[0135] ​Step G1: in response to the updating operation of the eye movement data in the observation interface, obtaining a new eye movement data point, and determining a target eye movement data point and a color region corresponding to the target eye movement data point through the new eye movement data point;

[0136] Step G2: calculating the boundary distance of the color region corresponding to the current target eye movement data point and the color region of the previous target positioning region, and determining the common display radius of the corresponding color region with the boundary distance less than a first preset threshold;

[0137] Step G3: determining the target positioning region to which the corresponding color region with the common display radius greater than a second preset threshold belongs as the target positioning region with the maximum attention.

[0138] In the embodiment, the user observes the constantly changing interface in the video through the screen, which will constantly generate new eye movement data points. When the new eye movement data points are transmitted, whether the newly obtained eye movement data point is the target eye movement data point and the color region corresponding to the target eye movement data point is determined through the contents in steps S101 to S105. The boundary distance of the color region corresponding to the current target eye movement data point and the color region of the previous target positioning region is calculated. If the boundary distance is less than a first preset threshold, the radius of the color region corresponding to the current target eye movement data point and the radius of the color region of the previous target positioning region are summed and the average value is calculated, which is taken as the common display radius. When the common display radius is greater than a second preset threshold, the target positioning region to which the corresponding color region belongs is determined as the target positioning region with the maximum attention. In particular, the maximum attention of the target positioning region is 1.

[0139] As Figure 6As shown, in the present embodiment, the user obtains all eye movement data points of the observation interface and performs smoothing processing according to a Kalman filtering algorithm, inputs information of the eye movement data points after the smoothing processing into an I-DT algorithm, identifies eye movement data points with fixation behavior and eye movement data points with saccade behavior, and performs filtering processing to obtain target eye movement data points; a first target region is obtained according to the target eye movement data points; a second target region is obtained by identifying the observation interface through a traditional target detection model, and the first target region and the second target region are compared to obtain a target positioning region. When a tracking algorithm is continuously used on the target positioning region, the target positioning region is updated according to the result of the tracking algorithm, and a heat map value of the updated target positioning region is calculated; information of the target eye movement data points in the target positioning region is obtained; whether a region is continuously fixated is judged according to whether the color regions coincide, if a region is continuously fixated and the fixation cumulative time exceeds a threshold value, the maximum attention degree of the current target positioning region is set to one; pupil features are calculated according to the information of the target eye movement data points, and an eye movement feature index is calculated through a random forest algorithm for emotion recognition; when a new target eye movement data point enters the target positioning region, the fixation number of the current target positioning region is increased by one, otherwise the fixation number of the target positioning region where the new target eye movement data point is located is cleared. The attention degree of the target positioning region is calculated according to the heat map value, the maximum attention degree, the fixation number and the eye movement feature index of the target positioning region; the above steps are repeated when new eye movement data points are continuously obtained; and finally an observation interface of the target positioning region with the attention degree is output.

[0140] In the present embodiment, through the target positioning method based on the eye movement data points, the following beneficial effects are achieved: (1) the first target region generated by introducing the eye movement data points compensates for the missed detection problem of the traditional target detection algorithm in the morphing target, the occluded target or the complex background scene. Based on the intersection over union analysis, the first target region focused by the human eye is compared with the second target region detected by the machine model, more accurate target positioning is achieved, and the missed detection rate in the complex scene is reduced; (2) by calculating different attention degrees in the target positioning region, the importance of the target in the scene is distinguished. The system preferentially processes high-attention targets according to the fixation intensity, reduces the resource allocation to secondary targets, makes the resource utilization more reasonable and efficient, and improves the response capability of the system in a multi-target environment; (3) the natural attention mode of the human eye is used to guide the target detection and processing flow in real time, so that the system can automatically adapt to the change of the user's attention point. This collaborative working mode of the human eye and the machine vision makes the human-computer interaction more intuitive and natural, and improves the user experience.

[0141] Embodiment two

[0142] The embodiment of the present application provides a target positioning device based on eye movement data, which comprises Figure 7As shown, the device specifically comprises the following components:

[0143] The acquisition module 701 is configured to acquire information of all eye movement data points in the observation interface, wherein the information of the eye movement data points comprises position information and collection time.

[0144] The classification module 702 is configured to perform classification processing on the eye movement data points according to the position information and the collection time of each eye movement data point, to obtain target eye movement data points belonging to fixation behaviors.

[0145] The determination module 703 is configured to determine a color region corresponding to the target eye movement data points in the observation interface by using a preset Gaussian function, and determine a first target region according to a distance of the color region, wherein the first target region represents a focus region of the observation interface on the user, and each first target region contains different numbers of target eye movement data points.

[0146] The detection module 704 is configured to input the observation interface into a trained target detection model, to obtain a plurality of second target regions contained in the observation interface, wherein the second target region represents a potential region of interest of the observation interface on the user.

[0147] The comparison module 705 is configured to compare the first target region and the second target region contained in the observation interface, and determine a target positioning region of the observation interface according to a comparison result.

[0148] Specifically, the classification module 702 is configured to:

[0149] calculate an Euclidean distance between the eye movement data point and a previous eye movement data point in a time sequence, and determine whether the Euclidean distance is greater than a preset distance threshold;

[0150] if yes, determine that the eye movement data point in the fixation event is a fixation end point, and determine that the eye movement data point not in the fixation event is a saccade point;

[0151] if no, determine whether there is a fixation event that has started at the collection time of the eye movement data point, and determine that the eye movement data point corresponding to the fixation event that has not started is a fixation start point;

[0152] determine that the fixation start point, the fixation end point, and the eye movement data point at the fixation start point and the fixation end point are fixation points;

[0153] determine that the fixation point and the fixation start point of the fixation event have a time interval greater than a preset time threshold.

[0154] Specifically, the determination module 703 is configured to:

[0155] create a white graph with the same size as the observation interface, and mark each target eye movement data point on the white graph according to the position information;

[0156] sequentially traverse the target eye movement data points on the white graph, and create a two-dimensional Gaussian template corresponding to each target eye movement data point through a preset Gaussian function, wherein the two-dimensional Gaussian template contains a plurality of weight values;

[0157] superimpose the two-dimensional Gaussian template on the corresponding target eye movement data point according to the position information, perform pseudo-color processing on the white graph through the weight values, and superimpose the processed image on the observation interface to obtain a color region corresponding to the target eye movement data point in the observation interface.

[0158] Further, the determination module 703 is further configured to:

[0159] respectively calculate the boundary distance between different color regions, and determine whether the boundary distance is less than or equal to a preset threshold;

[0160] If yes, it is determined that the corresponding color region belongs to the same first target region, and a rectangular frame is used to frame the first target region;

[0161] If no, it is determined that the corresponding color region belongs to different first target regions, and a rectangular frame is used to frame the first target regions respectively.

[0162] Specifically, the comparison module 705 is configured to:

[0163] determine whether the intersection over union of the first target region and the second target region is greater than a preset similarity threshold;

[0164] If yes, the first target region and the second target region in the observation interface are both regarded as the target positioning region of the observation interface;

[0165] If no, the second target region in the observation interface is regarded as the target positioning region of the observation interface.

[0166] Specifically, the apparatus further comprises:

[0167] a calculation module configured to calculate the target positioning region through a preset attention degree formula to obtain an importance index of the target eye movement data point; wherein the preset attention degree formula is: R represents all target positioning regions, represents the number of gazes of the i-th target positioning region, represents the eye movement feature index of the i-th target positioning region, represents the heat map value of the i-th target positioning region, characterize the i-th target positioning area.

[0168] Specifically, the device further comprises:

[0169] The updating module is configured to, in response to an updating operation of the eye movement data in the observation interface, acquire a new eye movement data point, determine a target eye movement data point and a color area corresponding to the target eye movement data point through the new eye movement data point, calculate a boundary distance between a color area corresponding to the current target eye movement data point and a color area of the previous target positioning area, determine a common display radius of a corresponding color area with the boundary distance less than a first preset threshold, and determine a target positioning area to which a corresponding color area with the common display radius greater than a second preset threshold belongs as a target positioning area with the maximum attention degree.

[0170] Embodiment Three

[0171] The embodiment also provides a computer device, such as a smartphone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server, or a cabinet server (including a single server or a server cluster composed of multiple servers), which can execute a program. As shown in the figure, the computer device 80 of the embodiment at least includes but is not limited to a memory 801 and a processor 802 which can be connected to each other through a system bus. It should be noted that, Figure 8 Only the computer device 80 with components 801-802 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented. Figure 8

[0172] ​In this embodiment, the memory 801 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 801 can be an internal storage unit of the computer device 80, such as a hard disk or a memory of the computer device 80. In other embodiments, the memory 801 can also be an external storage device of the computer device 80, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 80. Of course, the memory 801 can also include both the internal storage unit and the external storage device of the computer device 80. In this embodiment, the memory 801 is generally used to store an operating system and various application software installed on the computer device 80. In addition, the memory 801 can also be used to temporarily store various data that have been output or will be output.

[0173] The processor 802 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 802 is generally used to control the overall operation of the computer device 80.

[0174] Specifically, in this embodiment, the processor 802 is configured to execute a program of an eye movement data-based target positioning method stored in the memory 801, and the program of the eye movement data-based target positioning method, when executed, implements the following steps:

[0175] obtain information of all eye movement data points in an observation interface; wherein the information of the eye movement data points includes position information and collection time;

[0176] classify the eye movement data points according to the position information and the collection time of each eye movement data point to obtain target eye movement data points belonging to a fixation behavior;

[0177] determine a color region corresponding to the target eye movement data points in the observation interface by a preset Gaussian function, and determine a first target region according to a distance of the color region; wherein the first target region represents a focus region of the observation interface on the user, and each first target region contains different numbers of target eye movement data points;

[0178] input the observation interface into the trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential regions of interest of the user for the observation interface;

[0179] compare the first target regions and the second target regions contained in the observation interface to determine a target positioning region of the observation interface through a comparison result.

[0180] The specific embodiment process of the above method steps can be referred to in Embodiment One, which will not be repeated here.

[0181] Embodiment Four

[0182] The embodiment also provides a computer readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., which stores a computer program, and the computer program is executed by a processor to implement the following method steps:

[0183] obtain information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and collection time;

[0184] classify the eye movement data points through the position information and the collection time of each eye movement data point to obtain target eye movement data points belonging to a fixation behavior;

[0185] determine a color region corresponding to the target eye movement data points in the observation interface through a preset Gaussian function, and determine a first target region according to a distance of the color region; wherein the first target region represents a region of interest of the user for the observation interface, and each first target region contains different numbers of target eye movement data points;

[0186] input the observation interface into the trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential regions of interest of the user for the observation interface;

[0187] compare the first target regions and the second target regions contained in the observation interface to determine a target positioning region of the observation interface through a comparison result.

[0188] The specific embodiment process of the above method steps can be referred to in Embodiment One, which will not be repeated here.

[0189] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0190] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method.

[0192] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target positioning method based on eye movement data, characterized in that: The method comprises: Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; Determining color regions corresponding to target eye movement data points in the observation interface using a preset Gaussian function, and determining first target regions based on distances between the color regions; wherein the first target regions represent areas of user attention on the observation interface, and each first target region contains a different number of target eye movement data points; Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest to the user on the observation interface; Comparing the first target area and the second target area included in the observation interface, and determining the target positioning area of ​​the observation interface according to the comparison result; The first target area formed by eye movement data in the observation interface is compared with the second target area obtained by the YOLOv5 detection model, and the similarity is measured by the intersection-over-union (IoU) ratio. If the IoU ratio is greater than the preset similarity threshold, the second target area replaces the first target area and is recorded as the target positioning area. If the IoU ratio is less than the similarity threshold, the first target area is retained in addition to the second target area to obtain the final target positioning area.

2. The target positioning method based on eye movement data according to claim 1, characterized in that: The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain target eye movement data points belonging to the gaze behavior, including: Calculate the Euclidean distance between an eye movement data point and a previous eye movement data point in chronological order, and determine whether the Euclidean distance is greater than a preset distance threshold; If so, determine the eye movement data point in the fixation event as the fixation end point, and determine the eye movement data point not in the fixation event as the saccade point; If not, determining whether there is a gaze event that has already started at the time when the eye movement data point is collected, and determining the eye movement data point corresponding to the absence of the gaze event that has already started as the gaze start point; Determine the gaze starting point, the gaze ending point, and the eye movement data points collected at the gaze starting point and the gaze ending point as gaze points; A gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold is taken as a target eye movement data point.

3. The target positioning method based on eye movement data according to claim 1, characterized in that: The determining of the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function includes: Creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to position information; Traversing the target eye movement data points on the white image in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes multiple weight values; A two-dimensional Gaussian template is superimposed on the corresponding target eye movement data point according to the position information, the white image is pseudo-colored using the weight value, and the processed image is superimposed on the observation interface to obtain a color area corresponding to the target eye movement data point in the observation interface.

4. The target positioning method based on eye movement data according to claim 1, characterized in that: The determining the first target area according to the distance of the color area includes: Calculating the boundary distances between different color areas respectively, and determining whether the boundary distances are less than or equal to a preset threshold; If so, determining that the corresponding color regions belong to the same first target region, and framing the first target region with a rectangular frame; If not, it is determined that the corresponding color areas belong to different first target areas, and rectangular frames are used to frame the first target areas respectively.

5. The target positioning method based on eye movement data according to claim 1, characterized in that: The comparing the first target area and the second target area included in the observation interface and determining the target positioning area of ​​the observation interface according to the comparison result includes: Determining whether an intersection-over-union ratio between the first target region and the second target region is greater than a preset similarity threshold; If yes, both the first target area and the second target area in the observation interface are used as the target positioning areas of the observation interface; If not, the second target area in the observation interface is used as the target positioning area of ​​the observation interface.

6. The target positioning method based on eye movement data according to claim 1, characterized in that: The method further comprises: The target positioning area is calculated using a preset attention formula to obtain the importance index of the target eye movement data point; wherein the preset attention formula is: , R represents all target positioning areas, represents the number of fixations on the i-th target location area, The eye movement characteristic index representing the i-th target positioning area, Characterize the heat map value of the i-th target positioning area, Characterizes the active interaction value of the i-th target positioning area.

7. The target positioning method based on eye movement data according to claim 1, characterized in that: The method further comprises: In response to an update operation of the eye movement data in the observation interface, a new eye movement data point is acquired, and a target eye movement data point and a color area corresponding to the target eye movement data point are determined based on the new eye movement data point; Calculating a boundary distance between a color area corresponding to a current target eye movement data point and a color area of ​​a previous target positioning area, and determining a common display radius of corresponding color areas whose boundary distance is less than a first preset threshold; The target positioning area to which the corresponding color area whose public display radius is greater than the second preset threshold belongs is determined as the target positioning area with the maximum attention degree.

8. A target positioning device based on eye movement data, characterized in that: The device comprises: An acquisition module is used to acquire information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; A classification module is used to classify the eye movement data points according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; a determination module, configured to determine, using a preset Gaussian function, a color region corresponding to a target eye movement data point in the observation interface, and determine a first target region based on a distance between the color regions; wherein the first target region represents an area of ​​interest of the user on the observation interface, and each first target region contains a different number of target eye movement data points; a detection module, configured to input the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential regions of interest to the user on the observation interface; The comparison module is used to compare the first target area and the second target area contained in the observation interface, and determine the target positioning area of ​​the observation interface through the comparison result; compare the first target area formed by the eye movement data in the observation interface with the second target area obtained by the YOLOv5 detection model, and measure the similarity through the intersection-over-union ratio; if the intersection-over-union ratio is greater than a preset similarity threshold, replace the first target area with the second target area and record it as the target positioning area; if the intersection-over-union ratio is less than the similarity threshold, retain the first target area while retaining the second target area to obtain the final target positioning area.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images