Target positioning method and device based on eye movement data

Abstract

The invention discloses a target positioning method and device based on eye movement data. The method comprises the following steps: acquiring information of all eye movement data points in an observation interface; classifying the eye movement data points according to the position information and the acquisition time of each eye movement data point to obtain target eye movement data points belonging to the gazing behavior; determining a color region corresponding to a target eye movement data point in the observation interface through a preset Gaussian function, and determining a first target region according to the distance of the color region; inputting the observation interface into a trained target detection model to obtain a plurality of second target areas included in the observation interface; comparing a first target area and a second target area included in the observation interface, and determining a target positioning area of the observation interface according to a comparison result; according to the invention, the target area in the observation interface can be accurately and efficiently positioned based on the eye movement data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target positioning method and device based on eye movement data. Background Art

[0002] In recent years, neural networks have been widely used in the field of target detection, but they face challenges such as occlusion, incomplete targets, and insufficient clarity in complex environments. In contrast, the human visual system, with its powerful perception and cognitive abilities, can quickly adjust the focal length, adapt to complex backgrounds, and effectively compensate for missed detections and false detections of mechanical equipment. Researchers have gradually introduced the advantages of human vision into computer vision, and improved the accuracy of target positioning by combining eye movement information.

[0003] There are generally two types of target positioning methods in existing methods. The first is to identify the EEG features induced by the user's visual attention area by combining eye movement features with EEG, and obtain target location information by combining eye movement features, thereby improving the efficiency of users in collecting information under complex backgrounds. However, EEG equipment not only requires professionals to install and debug the equipment, but also has a complex data acquisition process. It is also susceptible to interference from the external environment, resulting in a decrease in signal quality and poor target detection accuracy. The second is a non-verbal communication framework based on eye gaze, in which the user's eye gaze movement is actively tracked and analyzed, and the user's intentions in daily life activities are inferred based on the relationship between visual behavior and psychological processes. However, this method is susceptible to environmental factors and the user's subjective intentions.

[0004] Therefore, how to improve the accuracy of target positioning has become a technical problem that technical personnel in this field need to solve urgently. Summary of the invention

[0005] The object of the present invention is to provide a method and device for target positioning based on eye movement data, which can accurately and efficiently locate the position of a target being gazed at in an observation interface based on eye movement data.

[0006] According to one aspect of the present invention, a method for target positioning based on eye movement data is provided, the method comprising: Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; Determine the color area corresponding to the target eye movement data point in the observation interface by a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The first target area and the second target area included in the observation interface are compared, and the target positioning area of ​​the observation interface is determined according to the comparison result.

[0007] Optionally, the eye movement data points are classified according to the position information and the acquisition time of each eye movement data point to obtain the target eye movement data point belonging to the gaze behavior, including: Calculate the Euclidean distance between the eye movement data point and the previous eye movement data point in chronological order, and determine whether the Euclidean distance is greater than a preset distance threshold; If so, determine the eye movement data point in the fixation event as the fixation end point, and determine the eye movement data point not in the fixation event as the saccade point; If not, determine whether there is a started fixation event at the time of collecting the eye movement data point, and determine the eye movement data point corresponding to the non-started fixation event as the fixation start point; Determine the fixation start point, the fixation end point, and the eye movement data points at the fixation start point and the fixation end point as fixation points; A gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold is taken as a target eye movement data point.

[0008] Optionally, determining the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function includes: Creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to the position information; Traversing the target eye movement data points on the white map in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes a plurality of weight values; The two-dimensional Gaussian template is superimposed on the corresponding target eye movement data point according to the position information, the white image is pseudo-colored using the weight value, and the processed image is superimposed on the observation interface to obtain the color area corresponding to the target eye movement data point in the observation interface.

[0009] Optionally, determining the first target area according to the distance of the color area includes: Calculating the boundary distances between different color areas respectively, and determining whether the boundary distances are less than or equal to a preset threshold; If yes, it is determined that the corresponding color regions belong to the same first target region, and the first target region is framed using a rectangular frame; If not, it is determined that the corresponding color areas belong to different first target areas, and the first target areas are framed using rectangular frames.

[0010] Optionally, comparing the first target area and the second target area included in the observation interface, and determining the target positioning area of ​​the observation interface according to the comparison result, includes: Determining whether an intersection-over-union ratio between the first target region and the second target region is greater than a preset similarity threshold; If yes, both the first target area and the second target area in the observation interface are used as the target positioning area of ​​the observation interface; If not, the second target area in the observation interface is used as the target positioning area of ​​the observation interface.

[0011] Optionally, the method further includes: The target positioning area is calculated using a preset attention formula to obtain an importance index of the target eye movement data point; wherein the preset attention formula is: , R represents all target positioning areas, represents the number of fixations on the i-th target location area, The eye movement characteristic index representing the i-th target positioning area, The heat map value representing the i-th target positioning area, Characterizes the active interaction value of the i-th target positioning area.

[0012] Optionally, the method further includes: In response to an update operation of the eye movement data in the observation interface, a new eye movement data point is acquired, and a target eye movement data point and a color area corresponding to the target eye movement data point are determined through the new eye movement data point; Calculate the boundary distance between the color area corresponding to the current target eye movement data point and the color area of ​​the previous target positioning area, and determine the common display radius of the corresponding color area whose boundary distance is less than a first preset threshold; The target positioning area to which the corresponding color area whose public display radius is greater than the second preset threshold belongs is determined as the target positioning area with the maximum attention degree.

[0013] In order to achieve the above object, the present invention also provides a target positioning device based on eye movement data, the device comprising: An acquisition module is used to acquire information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; A classification module is used to classify the eye movement data points according to the position information and collection time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; A determination module, configured to determine the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function, and determine a first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; A detection module, used for inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The comparison module is used to compare the first target area and the second target area contained in the observation interface, and determine the target positioning area of ​​the observation interface according to the comparison result.

[0014] In order to achieve the above-mentioned purpose, the present invention also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the target positioning method based on eye movement data introduced above are implemented.

[0015] In order to achieve the above-mentioned object, the present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the target positioning method based on eye movement data introduced above are implemented.

[0016] The target positioning method and device based on eye movement data provided by the present invention obtain target eye movement data points by classifying eye movement data points, use Gaussian function to determine the color area corresponding to the target eye movement data points, compare the target area formed according to the color area with the target area obtained by the target detection model, and determine the final target positioning area; the area obtained according to the detection result of the target detection algorithm combined with the eye movement data makes up for the missed detection problem of the target detection algorithm in complex scenes, improves the accuracy of target positioning, and significantly reduces the missed detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1A schematic diagram of an optional flow chart of a target positioning method based on eye movement data provided in Example 1; Figure 2 A schematic diagram of a process for classifying eye movement data points provided in Example 1; Figure 3 A schematic diagram of a process for screening target eye movement data points provided in Example 1; Figure 4 An observation interface effect diagram provided in Example 1; Figure 5 Another observation interface effect diagram provided in Example 1; Figure 6 A schematic diagram of another optional flow chart of the target positioning method based on eye movement data provided in the first embodiment; Figure 7 A schematic diagram of an optional structure of a target positioning device based on eye movement data provided in Example 2; Figure 8 This is a schematic diagram of an optional hardware structure of a computer device provided in Example 3. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] Embodiment 1 The embodiment of the present invention provides a target positioning method based on eye movement data, such as Figure 1 As shown, the method specifically comprises the following steps: S101: Acquire information of all eye movement data points in an observation interface; wherein the information of the eye movement data points includes position information and acquisition time.

[0020] In this embodiment, the observation interface is the interface that the user observes through the screen and changes constantly in any video; the user uses a high-precision eye tracking device to capture the pixel position where the line of sight falls on the observation interface in real time, and simultaneously records the changes in the pupil size of the user during observation. The eye movement data point is the pixel position seen by the user's eyes on the observation interface. The information of each eye movement data point includes position information, acquisition time, and pupil size.

[0021] Specifically, after obtaining information of all eye movement data points in the observation interface in step S101, the method further includes: Step A1: Map the pixel values ​​of all eye movement data points to angle values; Specifically, the eye movement data captured by the eye tracking device is usually quantified in pixels, but the pixel value itself does not have physical meaning, and due to the difference in the size of the viewing interface and the distance of the observer, the information of all eye movement data points needs to be converted from pixel coordinates to viewing angles. Therefore, the unit inconsistency between the pixel value and the actual physical coordinate can be eliminated through the mapping process.

[0022] Step A2: Input the information of all eye movement data points into the Kalman filter algorithm for smoothing to obtain smoothed information of all eye movement data points.

[0023] In this embodiment, the eye tracking device will be affected by the noise in the environment when capturing data in real time. By using the Kalman filter algorithm, the problem of jitter in the eye movement data is eliminated, thereby improving the processing efficiency.

[0024] S102: Classifying and processing the eye movement data points according to the position information and the acquisition time of each eye movement data point to obtain target eye movement data points belonging to the fixation behavior.

[0025] When the line of sight moves on the observation interface, there will be a period of time when the line of sight is relatively stable. The eye movement behavior in this period of time is called fixation behavior, and the eye movement data points in the fixation behavior are fixation points; there will also be rapid jumps in the line of sight, and the eye movement data points in this behavior are scan points. The preset classification algorithm can distinguish between fixation points and scan points in the eye movement data points.

[0026] It should be noted that the classification operation of eye movement data points can be implemented by a preset classification algorithm, wherein the preset classification algorithm is an I-VT algorithm (Velocity-Threshold Identification, velocity threshold identification algorithm) or an I-DT algorithm (Dispersion-Threshold Identification, dispersion threshold identification algorithm).

[0027] Specifically, the step S102 classifies the eye movement data points according to the position information and the collection time of each eye movement data point to obtain the target eye movement data points belonging to the gaze behavior, including: Step B1: calculating the Euclidean distance between an eye movement data point and a previous eye movement data point in chronological order, and determining whether the Euclidean distance is greater than a preset distance threshold; The Euclidean distance between an eye movement data point and the previous eye movement data point is the angle difference between the two eye movement data points. In particular, when the first eye movement data point is input, the Euclidean distance between the first eye movement data point and the previous eye movement data point is 0.

[0028] Step B2: If yes, determine the eye movement data point in the fixation event as the fixation end point, and determine the eye movement data point not in the fixation event as the saccade point; Among them, a gaze event represents a gaze behavior, including the gaze start point, the gaze end point, and the eye movement data points between the gaze start point and the gaze end point.

[0029] Step B3: If not, determine whether there is a gaze event that has started at the time when the eye movement data point is collected, and determine the eye movement data point that does not correspond to the gaze event that has started as the gaze start point; Step B4: determining the gaze starting point, the gaze ending point, and the eye movement data points whose acquisition time is at the gaze starting point and the gaze ending point as the gaze points; Furthermore, the step S102 of classifying the eye movement data points according to the position information and the acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the gaze behavior also includes: Step B5: determining whether there is a gaze event that has started at the time of collecting the last eye movement data point; Step B6: If yes, then determine the last eye movement data point as the fixation end point; if no, then determine the last eye movement data point as the scan point.

[0030] In this embodiment, if Figure 2 As shown, the position information and collection time of the eye movement data points are input into the system of the preset classification algorithm in the order of collection time, and each eye movement data point is judged in sequence. First, it is judged whether the eye movement data point currently analyzed is the last eye movement data point. If not, the Euclidean distance between the current eye movement data point and the adjacent eye movement data point at the previous moment is calculated, and it is judged whether the Euclidean distance is greater than the preset distance threshold. If it is greater than the preset distance threshold and in the fixation behavior, the information of the current eye movement data point is added to the end (Efix) list and the next eye movement data point is processed; if it is greater than the preset distance threshold and not in the fixation behavior, the next eye movement data point is processed. If it is less than or equal to the preset distance threshold and the current fixation behavior has not started, the information of the current eye movement data point is added to the start (Sfix) list and the next eye movement data point is processed; if it is less than or equal to the preset distance threshold and in the current fixation behavior, the next eye movement data point is processed. When the last eye movement data point is input, if the last eye movement data point is in the fixation behavior, the information of the last eye movement data point is added to the end (Efix) list. Finally, the start Sfix list and the end Efix list are output.

[0031] Among them, the Sfix list and the Efix list record multiple eye movement data points that represent fixation behaviors, and the position of the fixation start point of each fixation behavior in the Sfix list corresponds to the position of the fixation end point of each fixation behavior in the Efix list. The fixation start point, fixation end point, and eye movement data points between the fixation start point and the fixation end point of each fixation behavior are all fixation points, and the eye movement data points that are not in the fixation behavior are saccade points, and the types of all eye movement data points are marked.

[0032] In addition, the I-VT algorithm can also be used to identify eye movement data points representing fixation behavior and eye movement data points representing scanning behavior through a speed threshold, where the speed threshold is the quotient of the Euclidean distance and the time difference.

[0033] Step B7: taking a gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold as a target eye movement data point.

[0034] In this embodiment, if Figure 3 As shown, all the eye movement data points that have been classified are filtered again: input the eye movement data points one by one, and determine whether the previous eye movement data point of the currently analyzed eye movement data point is a fixation point. If not, input the next eye movement data point; if so, calculate whether the time difference between the current eye movement data point and the fixation start point of the current fixation behavior is greater than the preset time threshold. When the time difference is greater than the preset time threshold, add the current eye movement data point to the heat map drawing queue; when the time difference is less than or equal to the preset time threshold, input the next eye movement data point. The eye movement data points in the heat map drawing queue are the target eye movement data points. By classifying the eye movement data points and filtering out the scan points and the fixation points with a shorter time, the focus of the user's attention is more accurately reflected, highlighting the importance of the fixation point.

[0035] Preferably, the preset distance threshold may be 30°, and the preset time threshold may be 200 milliseconds.

[0036] S103: Determine the color area corresponding to the target eye movement data point in the observation interface through a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's focus area on the observation interface, and each first target area contains a different number of target eye movement data points.

[0037] In this embodiment, the target eye movement data points in the drawing heat map queue are calculated respectively by a two-dimensional Gaussian function, and each target eye movement data point corresponds to a color area. The color areas corresponding to different target eye movement data points may overlap. When the distance between different color areas is less than a preset value, a rectangular frame is used to select the corresponding color area to obtain a first target area. The first target area represents the user's focus area on the observation interface.

[0038] Specifically, the step of determining the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function in step S103 includes: Step C1: creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to the position information; Step C2: traversing the target eye movement data points on the white image in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes a plurality of weight values; Step C3: superimpose the two-dimensional Gaussian template on the corresponding target eye movement data point according to the position information, perform pseudo-color processing on the white image according to the weight value, and superimpose the processed image on the observation interface to obtain the color area corresponding to the target eye movement data point in the observation interface.

[0039] In this embodiment, the color area generated according to the target eye movement data point is a gaze heat map. Create a white image with the same size as the observation interface and a grayscale value of 0, and mark the target eye movement data points in the drawing heat map queue on the white image according to the position information of the target eye movement data point in the drawing heat map queue. The two-dimensional Gaussian function can calculate the two-dimensional Gaussian template corresponding to each target eye movement data point and assign a weight value to the Gaussian template. Each Gaussian template is superimposed on the corresponding target eye movement data point, and the center of each Gaussian template is at the corresponding target eye movement data point position, that is, the target eye movement data point has the highest weight, and the weight value on the Gaussian template gradually decreases as the distance from the target eye movement data point increases. The white image is pseudo-colored, and the area where the Gaussian template corresponding to each target eye movement data point is located will form a colored area, and the color changes with different weight values. The area corresponding to the target eye movement data point is weighted by the Gaussian template and a color area is formed according to the weight value, thereby improving the accuracy of target positioning.

[0040] Specifically, the determining of the first target area according to the distance of the color area in step S103 includes: Step D1: respectively calculating the boundary distances between different color regions, and determining whether the boundary distances are less than or equal to a preset threshold; Step D2: If yes, determine that the corresponding color regions belong to the same first target region, and frame the first target region with a rectangular frame; Step D3: If not, determine that the corresponding color areas belong to different first target areas, and use rectangular frames to frame the first target areas respectively.

[0041] In this embodiment, the color areas formed by adjacent target eye movement data points may intersect, and the boundary distance between different color areas is calculated. When the boundary distance is less than or equal to a preset threshold, the large area formed by multiple different intersecting color areas is divided into the same first target area and framed by a rectangular frame; when the boundary distance is greater than the preset threshold, the multiple different color areas are respectively divided into separate first target areas, and each first target area is framed by a rectangular frame.

[0042] Furthermore, after step D3, the method further includes: Step E1: framing the color area with a rectangular frame to obtain a detection frame; Step E2: Calculate the mean of all pixel values ​​in the detection frame, and determine whether the mean is greater than a preset boundary threshold; Step E3: If yes, then set the area in the detection frame as a high-attention area, and mark the high-attention area as the first target area; Step E4: If not, the area within the detection frame is set as a low attention area.

[0043] In this embodiment, the colored area is framed with a rectangular frame to obtain a detection frame. To reduce the amount of calculation, the pixel values ​​of non-zero points in the detection frame are summed and the mean is calculated, that is, the colored pixels in the detection frame are summed and the mean is calculated. In particular, the boundary threshold is twice the mean of the current detection frame. It is determined whether the mean in the detection frame is greater than the boundary threshold. If not, the detection frame is a low-attention area; if so, the detection frame is a high-attention area, and the high-attention area is recorded as the first target area and marked.

[0044] S104: Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface.

[0045] In this embodiment, the trained target detection model is used to detect the entire observation interface to obtain an observation interface including a target detection frame. The second target area is the area within the detected target detection frame, which represents the area of ​​potential interest to the user.

[0046] Preferably, the trained target detection model is a YOLOv5 detection model (You Only Look Once version 5).

[0047] S105: Compare the first target area and the second target area included in the observation interface, and determine the target positioning area of ​​the observation interface according to the comparison result.

[0048] Specifically, the step S105 of comparing the first target area and the second target area contained in the observation interface, and determining the target positioning area of ​​the observation interface according to the comparison result, includes: Step F1: determining whether the intersection-over-union ratio of the first target region and the second target region is greater than a preset similarity threshold; Step F2: If yes, both the first target area and the second target area in the observation interface are used as the target positioning area of ​​the observation interface; Step F3: If not, taking the second target area in the observation interface as the target positioning area of ​​the observation interface.

[0049] In this embodiment, the first target area formed by the eye movement data in the observation interface is compared with the second target area obtained by the YOLOv5 detection model, and the similarity is measured by the intersection and union ratio. If the intersection and union ratio is greater than the preset similarity threshold, the second target area replaces the first target area and is recorded as the target positioning area; if the intersection and union ratio is less than the similarity threshold, the second target area is retained while the first target area is also retained to obtain the final target positioning area. The high-attention area located by the eye movement data is retained by the intersection and union ratio, which makes up for the risk of missed detection.

[0050] Preferably, the method further comprises: The target positioning area is calculated using a preset attention formula to obtain an importance index of the target eye movement data point; wherein the preset attention formula is: , R represents all target positioning areas, represents the number of fixations on the i-th target location area, The eye movement characteristic index representing the i-th target positioning area, The heat map value representing the i-th target positioning area, Characterizes the active interaction value of the i-th target positioning area.

[0051] In this embodiment, the attention degree of the target positioning area detected in the observation interface is calculated to obtain the importance of different target positioning areas. Among them, statistical indicators such as gaze, saccade and pupil show obvious changes under different emotional states, and 29 eye movement features covering gaze, saccade and pupil changes are selected to calculate the target attention degree, including: number of gazes, gaze frequency, average gaze duration, standard deviation of gaze duration, first gaze time, first gaze duration, number of saccades, saccade frequency, average saccade duration, standard deviation of saccade duration, kurtosis of saccade duration, skewness of saccade duration, average saccade amplitude, standard deviation of saccade amplitude, kurtosis of saccade amplitude, skewness of saccade amplitude, average saccade speed, standard deviation of saccade speed, kurtosis of saccade speed, skewness of saccade speed, peak value of saccade amplitude, peak value of saccade speed, average pupil diameter, standard deviation of pupil diameter, skewness of pupil diameter, kurtosis of pupil diameter, and pupil activity index (IPA).

[0052] The target attention calculation formula is: , R represents all target positioning areas, and i represents the current target positioning area.

[0053] Specifically, , Indicates the current target gaze count, which increases cumulatively. When the gaze point enters the target area and leaves, the current target gaze count is accumulated plus 1. For normalization, the current target gaze count is divided by the total target gaze count as part of the attention degree score.

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] in, is a stable updated value. Only when a gaze point enters the target area and leaves, the current target It will be updated, otherwise it will remain unchanged. It includes two parts. The first part is the inner activity results calculated based on the emotion and cognitive load theory. represents the importance result of random forest prediction when the subject looks at this target. There are 3 results, namely 1, 2, 3, The score representing the cognitive load of the subject's gaze on the target is mainly represented by the pupil activity index. For normalization, the sigmoid function is used to process it. The second part is a characteristic index calculated based on the eye movement characteristics. The gaze index, the saccade index and the pupil index are calculated respectively, and the sum of the three is used as the characteristic index of the second part; represents the mean, Represents variance.

[0062] Calculated from the heat map. Since the heat map updates quickly, It changes all the time, and the closer it is to the current moment, the larger its value is, the greater the attention score of the latest attention area can be, which is convenient for updating the attention level of each target. In order to obtain a more accurate indicator of potential sensitive areas, a quantitative method is adopted. First, the heat map data within each rectangular border is summed, and then divided by the number of all pixels contained in the border to obtain the hotspot mean of the border. Secondly, the hotspot mean of each border is divided by the mean of the entire heat map at the current moment as an indicator to quantify the attention level of the area within the border. The specific calculation is shown in the following calculation formula:

[0063]

[0064]

[0065]

[0066] Among them, k represents the pixel in the current target positioning area, j represents the Gaussian function, Represents the heat map intensity of the entire interface, Represents the heatmap intensity of each boxed potential sensitive area, A heatmap score representing this potentially sensitive area.

[0067]

[0068] Only when the current target is actively selected will the maximum attention be set, and the remaining targets will all be 0.

[0069] After obtaining the final importance index, all target positioning areas are sorted. According to the sorting results, the target with the highest importance is set to 1, and the importance of the remaining targets is reduced in turn, and is set according to the relative gap with the most important target.

[0070]

[0071] Through calculation, the attention value used to measure the relative importance of each target positioning area in the entire observation interface can be obtained. And, according to the calculated attention value, all target positioning areas are sorted. Figure 4 , Figure 5 As shown, each target positioning area in the observation interface is marked with a degree of attention.

[0072] Preferably, the method further comprises: Step G1: in response to an update operation of the eye movement data in the observation interface, obtaining a new eye movement data point, and determining a target eye movement data point and a color area corresponding to the target eye movement data point through the new eye movement data point; Step G2: Calculate the boundary distance between the color area corresponding to the current target eye movement data point and the color area of ​​the previous target positioning area, and determine the common display radius of the corresponding color area whose boundary distance is less than a first preset threshold; Step G3: Determine the target positioning area to which the corresponding color area whose public display radius is greater than the second preset threshold belongs as the target positioning area with the maximum attention degree.

[0073] In this embodiment, the user observes the constantly changing interface in the video through the screen, and new eye movement data points are continuously generated. When a new eye movement data point is input, the contents in steps S101 to S105 are used to determine whether the newly acquired eye movement data point is a target eye movement data point and the color area corresponding to the target eye movement data point. The boundary distance between the color area corresponding to the current target eye movement data point and the color area of ​​the previous target positioning area is calculated. If the boundary distance is less than the first preset threshold, the radius of the color area corresponding to the current target eye movement data point is summed with the radius of the color area of ​​the previous target positioning area and the average is calculated, and the average is used as the public display radius. When the public display radius is greater than the second preset threshold, the target positioning area to which the corresponding color area belongs is determined as the target positioning area with the maximum attention. In particular, the maximum attention of the target positioning area is 1.

[0074] like Figure 6As shown, in this embodiment, the user obtains all eye movement data points of the observation interface and performs smoothing processing according to the Kalman filtering algorithm, inputs the information of the smoothed eye movement data points into the I-DT algorithm, identifies the eye movement data points with fixation behavior and the eye movement data points with scanning behavior, and performs filtering processing to obtain the target eye movement data points; obtains the first target area according to the target eye movement data points; identifies the observation interface through the traditional target detection model to obtain the second target area, and compares the first target area with the second target area to obtain the target positioning area. When the tracking algorithm is continued to be used for the target positioning area, the target positioning area is updated according to the result of the tracking algorithm, and the heat map value of the updated target positioning area is calculated; the information of the target eye movement data point in the target positioning area is obtained; whether a region is continuously fixed on is determined according to whether the color region overlaps, if a region is continuously fixed on and the cumulative fixation time exceeds the threshold, the maximum attention of the current target positioning area is set to one; pupil features are calculated according to the information of the target eye movement data point, and emotion recognition is performed through the random forest algorithm to calculate the eye movement feature index; when a new target eye movement data point enters the target positioning area, the number of fixations of the current target positioning area is increased by one, otherwise the number of fixations of the target positioning area where the new target eye movement data point is located is cleared to zero. The attention of the target positioning area is calculated according to the heat map value, maximum attention, number of fixations and eye movement feature index of the target positioning area; the above steps are repeated when new eye movement data points are continuously obtained; finally, the observation interface of the target positioning area marked with attention is output.

[0075] In this embodiment, the target positioning method based on eye movement data points achieves the following beneficial effects: (1) By introducing the first target area generated by eye movement data points, the problem of missed detection of traditional target detection algorithms in deformed targets, occluded targets or complex background scenes is compensated. Based on the intersection-over-union analysis, the first target area focused by the human eye is compared with the second target area detected by the machine model to achieve more accurate target positioning and reduce the missed detection rate in complex scenes; (2) By calculating the different attention levels in the target positioning area, the importance of the targets in the scene can be distinguished. The system prioritizes high-attention targets according to the gaze intensity, reduces the resource allocation to secondary targets, makes resource utilization more reasonable and efficient, and improves the system's responsiveness in a multi-target environment; (3) By using the natural attention pattern of the human eye, the target detection and processing process is guided in real time, so that the system can automatically adapt to changes in the user's focus. This collaborative working mode of human eye and machine vision makes human-computer interaction more intuitive and natural, and improves the user experience.

[0076] Embodiment 2 The embodiment of the present invention provides a target positioning device based on eye movement data, such as Figure 7 As shown, the device specifically includes the following components: The acquisition module 701 is used to acquire information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; A classification module 702 is used to classify the eye movement data points according to the position information and collection time of each eye movement data point to obtain target eye movement data points belonging to the fixation behavior; The determination module 703 is used to determine the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; A detection module 704 is used to input the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The comparison module 705 is used to compare the first target area and the second target area included in the observation interface, and determine the target positioning area of ​​the observation interface according to the comparison result.

[0077] Specifically, the classification module 702 is used to: Calculate the Euclidean distance between the eye movement data point and the previous eye movement data point in chronological order, and determine whether the Euclidean distance is greater than a preset distance threshold; If so, determine the eye movement data point in the fixation event as the fixation end point, and determine the eye movement data point not in the fixation event as the saccade point; If not, determine whether there is a started fixation event at the time of collecting the eye movement data point, and determine the eye movement data point corresponding to the non-started fixation event as the fixation start point; Determine the fixation start point, the fixation end point, and the eye movement data points at the fixation start point and the fixation end point as fixation points; A gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold is taken as a target eye movement data point.

[0078] Specifically, the determining module 703 is used to: Creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to the position information; Traversing the target eye movement data points on the white map in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes a plurality of weight values; The two-dimensional Gaussian template is superimposed on the corresponding target eye movement data point according to the position information, the white image is pseudo-colored using the weight value, and the processed image is superimposed on the observation interface to obtain the color area corresponding to the target eye movement data point in the observation interface.

[0079] Furthermore, the determining module 703 is further configured to: Calculating the boundary distances between different color areas respectively, and determining whether the boundary distances are less than or equal to a preset threshold; If yes, it is determined that the corresponding color regions belong to the same first target region, and the first target region is framed using a rectangular frame; If not, it is determined that the corresponding color areas belong to different first target areas, and the first target areas are framed using rectangular frames.

[0080] Specifically, the comparison module 705 is used to: Determining whether an intersection-over-union ratio between the first target region and the second target region is greater than a preset similarity threshold; If yes, both the first target area and the second target area in the observation interface are used as the target positioning area of ​​the observation interface; If not, the second target area in the observation interface is used as the target positioning area of ​​the observation interface.

[0081] Specifically, the device further includes: A calculation module is used to calculate the target positioning area by using a preset attention formula to obtain the importance index of the target eye movement data point; wherein the preset attention formula is: , R represents all target positioning areas, represents the number of fixations on the i-th target location area, The eye movement characteristic index representing the i-th target positioning area, The heat map value representing the i-th target positioning area, Characterizes the active interaction value of the i-th target positioning area.

[0082] Specifically, the device further includes: An updating module is used to respond to an update operation of eye movement data in an observation interface, obtain new eye movement data points, and determine a target eye movement data point and a color area corresponding to the target eye movement data point through the new eye movement data point; calculate a boundary distance between a color area corresponding to a current target eye movement data point and a color area of ​​a previous target positioning area, and determine a common display radius of a corresponding color area whose boundary distance is less than a first preset threshold; and determine a target positioning area to which a corresponding color area whose common display radius is greater than a second preset threshold belongs as a target positioning area of ​​maximum attention.

[0083] Embodiment 3 This embodiment also provides a computer device, such as a smart phone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs. Figure 8 As shown, the computer device 80 of this embodiment includes at least but not limited to: a memory 801 and a processor 802 that can be interconnected through a system bus. It should be noted that Figure 8 Only a computer device 80 having components 801 - 802 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0084] In this embodiment, the memory 801 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 801 can be an internal storage unit of the computer device 80, such as a hard disk or memory of the computer device 80. In other embodiments, the memory 801 can also be an external storage device of the computer device 80, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 80. Of course, the memory 801 can also include both the internal storage unit of the computer device 80 and its external storage device. In this embodiment, the memory 801 is generally used to store the operating system and various application software installed on the computer device 80. In addition, the memory 801 can also be used to temporarily store various data that have been output or are to be output.

[0085] In some embodiments, the processor 802 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 802 is generally used to control the overall operation of the computer device 80 .

[0086] Specifically, in this embodiment, the processor 802 is used to execute the program of the target positioning method based on eye movement data stored in the memory 801, and the program of the target positioning method based on eye movement data implements the following steps when being executed: Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; Determine the color area corresponding to the target eye movement data point in the observation interface by a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The first target area and the second target area included in the observation interface are compared, and the target positioning area of ​​the observation interface is determined according to the comparison result.

[0087] The specific implementation process of the above method steps can be found in Example 1, and this embodiment will not be repeated here.

[0088] Embodiment 4 This embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored. When the computer program is executed by a processor, the following method steps are implemented: Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; Determine the color area corresponding to the target eye movement data point in the observation interface by a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The first target area and the second target area included in the observation interface are compared, and the target positioning area of ​​the observation interface is determined according to the comparison result.

[0089] The specific implementation process of the above method steps can be found in the first embodiment, and this embodiment will not be repeated here.

[0090] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0091] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0093] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target positioning method based on eye movement data, characterized in that: The method comprises: Obtaining information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; The eye movement data points are classified and processed according to the position information and acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; Determine the color area corresponding to the target eye movement data point in the observation interface by a preset Gaussian function, and determine the first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; Inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The first target area and the second target area included in the observation interface are compared, and the target positioning area of ​​the observation interface is determined according to the comparison result.

2. The target positioning method based on eye movement data according to claim 1, characterized in that: The eye movement data points are classified and processed according to the position information and the acquisition time of each eye movement data point to obtain the target eye movement data points belonging to the gaze behavior, including: Calculate the Euclidean distance between the eye movement data point and the previous eye movement data point in chronological order, and determine whether the Euclidean distance is greater than a preset distance threshold; If so, determine the eye movement data point in the fixation event as the fixation end point, and determine the eye movement data point not in the fixation event as the saccade point; If not, determine whether there is a started fixation event at the time of collecting the eye movement data point, and determine the eye movement data point corresponding to the non-started fixation event as the fixation start point; Determine the fixation start point, the fixation end point, and the eye movement data points at the fixation start point and the fixation end point as fixation points; A gaze point whose time interval with the gaze start point of the corresponding gaze event is greater than a preset time threshold is taken as a target eye movement data point.

3. The target positioning method based on eye movement data according to claim 1, characterized in that: The determining the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function includes: Creating a white map of the same size as the observation interface, and marking each target eye movement data point on the white map according to the position information; Traversing the target eye movement data points on the white map in sequence, and creating a two-dimensional Gaussian template corresponding to the target eye movement data points by using a preset Gaussian function, wherein the two-dimensional Gaussian template includes a plurality of weight values; The two-dimensional Gaussian template is superimposed on the corresponding target eye movement data point according to the position information, the white image is pseudo-colored using the weight value, and the processed image is superimposed on the observation interface to obtain the color area corresponding to the target eye movement data point in the observation interface.

4. The target positioning method based on eye movement data according to claim 1, characterized in that: The determining the first target area according to the distance of the color area includes: Calculating the boundary distances between different color areas respectively, and determining whether the boundary distances are less than or equal to a preset threshold; If yes, it is determined that the corresponding color regions belong to the same first target region, and the first target region is framed using a rectangular frame; If not, it is determined that the corresponding color areas belong to different first target areas, and the first target areas are framed using rectangular frames.

5. The target positioning method based on eye movement data according to claim 1, characterized in that: The step of comparing the first target area and the second target area included in the observation interface and determining the target positioning area of ​​the observation interface according to the comparison result includes: Determining whether an intersection-over-union ratio between the first target region and the second target region is greater than a preset similarity threshold; If yes, both the first target area and the second target area in the observation interface are used as the target positioning area of ​​the observation interface; If not, the second target area in the observation interface is used as the target positioning area of ​​the observation interface.

6. The target positioning method based on eye movement data according to claim 1, characterized in that: The method further comprises: The target positioning area is calculated using a preset attention formula to obtain an importance index of the target eye movement data point; wherein the preset attention formula is: , R represents all target positioning areas, represents the number of fixations on the i-th target location area, The eye movement characteristic index representing the i-th target positioning area, The heat map value representing the i-th target positioning area, Characterizes the active interaction value of the i-th target positioning area.

7. The target positioning method based on eye movement data according to claim 1, characterized in that: The method further comprises: In response to an update operation of the eye movement data in the observation interface, a new eye movement data point is acquired, and a target eye movement data point and a color area corresponding to the target eye movement data point are determined through the new eye movement data point; Calculate the boundary distance between the color area corresponding to the current target eye movement data point and the color area of ​​the previous target positioning area, and determine the common display radius of the corresponding color area whose boundary distance is less than a first preset threshold; The target positioning area to which the corresponding color area whose public display radius is greater than the second preset threshold belongs is determined as the target positioning area with the maximum attention degree.

8. A target positioning device based on eye movement data, characterized in that: The device comprises: An acquisition module is used to acquire information of all eye movement data points in the observation interface; wherein the information of the eye movement data points includes position information and acquisition time; A classification module is used to classify the eye movement data points according to the position information and collection time of each eye movement data point to obtain the target eye movement data points belonging to the fixation behavior; A determination module, configured to determine the color area corresponding to the target eye movement data point in the observation interface by using a preset Gaussian function, and determine a first target area according to the distance of the color area; wherein the first target area represents the user's attention area on the observation interface, and each first target area contains a different number of target eye movement data points; A detection module, used for inputting the observation interface into a trained target detection model to obtain a plurality of second target regions contained in the observation interface; wherein the second target regions represent potential areas of interest of the user on the observation interface; The comparison module is used to compare the first target area and the second target area contained in the observation interface, and determine the target positioning area of ​​the observation interface according to the comparison result.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images