A graph-based visual attention encoding method

Abstract

The application discloses a kind of based on graph visual attention coding method, first using eye movement equipment to collect the eye movement data of subject when watching visual image, the object region in the image watched is labeled, according to the object region labeled, in turn, the eye movement fixation point falling into the same labeled object region is classified into a group, constitutes semantic fixation path, then the number of attention transfer between different semantic object regions in the semantic fixation path of each subject is counted, and a visual attention graph is constructed, and the visual attention graph is standardized to complete the coding based on graph visual attention, and the standardized visual attention graph can be used to evaluate the attention consistency of the subject.The method of the application can effectively solve the problem of large individual difference of original fixation point path, has the characteristics of simple calculation, good effect, etc., and is very suitable for coding group visual behavior.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cognitive computing, and particularly relates to a graph-based visual attention encoding method. BACKGROUND

[0002] Visual attention plays a crucial role in the human visual system when performing visual tasks by interacting with the physical scene. By quantitatively analyzing human eye movement behavior, visual attention behavior patterns in different states can be identified, which is of great significance to the design of human-computer interaction and other products. In the field of computer vision, a large number of studies have designed algorithms to predict visual fixation point distribution or saccadic path, thereby simulating the visual attention and behavior of human observers when observing a scene. These methods can effectively improve the performance of brain-like intelligent systems and exhibit some human-like behaviors, and have wide application prospects in the fields of assisted driving, fatigue detection, etc.

[0003] At present, most of the computing and analysis methods are typical fixation point distribution estimation methods, such as the document“A. Borji and L. Itti, State-of-the-art in visual attention modeling. IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 185-207, 2012”. Secondly, some methods are mainly used for quantitative analysis of saccadic path, such as the document“C. Xia, J. Han, and D. Zhang, Evaluation of saccadic scanpath prediction: Subjective assessment database and recurrent neural network based metric. IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 12, pp. 4378-4395, 2020”. These methods are all based on the original fixation point data or saccadic path collected by human observers to encode visual attention, and cannot well capture the common patterns of visual attention. And these attention encoding methods do not consider the object semantic information of the viewed scene, resulting in large individual differences in attention encoding, and cannot effectively encode the common visual attention behavior of humans. Therefore, it is of great significance to develop more stable visual attention encoding methods based on eye movement data to reduce individual differences in the data for improving visual attention prediction and attention behavior recognition. SUMMARY

[0004] To solve the above technical problems, the present invention provides a graph-based visual attention encoding method, which solves the problem that existing methods cannot effectively encode human common visual attention behavior.

[0005] The technical solution adopted by the present invention is: a graph-based visual attention encoding method, the specific steps are as follows:

[0006] S1. Collect the subject's eye movement data using an eye tracking device, i.e., record the subject's eye movement data while viewing visual images using an eye tracker;

[0007] Wherein, the number of subjects is greater than or equal to 3.

[0008] S2, visual image object region annotation, that is, for the visual images presented to the subjects, the main object regions contained in the images were manually annotated using open source tools;

[0009] Among them, the pixels belonging to the same object in the image are marked as the same area, and the annotations of all object areas in the image are obtained.

[0010] S3, constructing a semantic gaze path based on the object area marked in step S2 and the eye movement data recorded in step S1;

[0011] For the eye movement data collected when each subject views the visual image, the eye movement fixation points that fall into the same labeled object area are grouped together according to the time sequence of the eye movement fixation points recorded by the eye tracker. After all the fixation points are grouped in sequence based on the labeled object area, a fixation object sequence of the subject viewing the current visual image is obtained, which is the semantic fixation path.

[0012] S4, based on step S3, building a visual attention map by combining the semantic gaze path of each subject;

[0013] For each subject, the semantic gaze path of each subject is obtained based on step S3, the number of attention transfers between different semantic object areas in the semantic gaze path of each subject is counted, and a directed weighted graph is constructed with each object area as a node and the number of attention transfers between objects as the weight of the edge to form a visual attention map.

[0014] S5. Normalize the visual attention map obtained in step S4 to complete the map-based visual attention encoding;

[0015] For each node of the visual attention map obtained in step S4, the weight of each edge starting from the node is divided by the sum of the weights of all edges starting from the node to obtain the normalized weight of the edge starting from the node; all nodes of the attention map are processed in turn to obtain a normalized visual attention map.

[0016] Further, the visual attention encoding method further comprises a step S6 of calculating a score of the attention of the to-be-evaluated subject on the standardized visual attention graph, and evaluating the consistency between the attention of the to-be-evaluated subject and the attention of the subjects collected for constructing the standardized visual attention graph, and the specific process is as follows:

[0017] According to the method of steps S1 to S3, the attention of a to-be-evaluated subject is evaluated, and first, a semantic fixation path of the to-be-evaluated subject is constructed, and the edge weight of each attention shift on the semantic fixation path of the to-be-evaluated subject on the standardized visual attention graph constructed in step S5 is found.

[0018] When the attention shift on the semantic fixation path of the to-be-evaluated subject has no corresponding edge in the standardized visual attention graph, the corresponding edge weight of the attention shift is set to zero; after all the attention shifts on the semantic fixation path of the to-be-evaluated subject are traversed, the average value of all the edge weights found is taken as the score of the semantic fixation path of the to-be-evaluated subject on the standardized visual attention graph; and the score reflects the consistency degree of the to-be-evaluated subject and the attention of the subjects collected for constructing the standardized visual attention graph.

[0019] The method of the present application first collects eye movement data of subjects when watching visual images by using an eye movement device, labels object regions in the watched images, and according to the labeled object regions, sequentially classifies eye movement fixation points falling into the same labeled object region as a group to form a semantic fixation path, and then constructs a visual attention graph by counting the number of attention shifts between different semantic object regions in the semantic fixation path of each subject, and performs standardization processing on the visual attention graph to complete graph-based visual attention encoding, and the standardized visual attention graph can be used to evaluate the attention consistency of the subjects. The method of the present application can effectively solve the problem of large individual difference of original fixation point paths, has the characteristics of simple calculation and good effect, and is very suitable for encoding the visual behavior of a group of subjects. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A flowchart of a graph-based visual attention encoding method of the present application.

[0021] Figure 2 A schematic diagram of constructing a semantic fixation path in an embodiment of the present application.

[0022] Figure 3 A schematic diagram of constructing a visual attention graph in an embodiment of the present application.

[0023] Figure 4 A comparison diagram of different visual attention encoding methods in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The method of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0025] like Figure 1 As shown in FIG, a flowchart of a graph-based visual attention encoding method of the present invention is shown, and the specific steps are as follows:

[0026] S1. Use eye-tracking equipment to collect eye-movement data of the subjects, that is, use a commercial eye-tracking device - an eye tracker to record the eye-movement data of the subjects while they are viewing visual images;

[0027] The number of subjects is greater than or equal to 3, and data of 3 or more subjects are required to appropriately construct a visual attention map. In this embodiment, the number of subjects is 6.

[0028] S2, visual image object region annotation, that is, for the visual images presented to the subjects, the open source tool LabelMe was used to manually annotate the main object regions contained in the images;

[0029] The pixels belonging to the same object in the image are marked as the same area, and the annotation of all object areas in the image is obtained. Figure 2 As shown, the figure includes 10 objects, numbered A to J.

[0030] S3, constructing a semantic gaze path based on the object area marked in step S2 and the eye movement data recorded in step S1;

[0031] like Figure 2 As shown in the figure, for each subject's eye movement data (numbered 1, 2, 3, ..., 9) when viewing the visual image, the eye movement fixation points that fall into the same labeled object area are grouped together according to the time sequence of the eye movement fixation points recorded by the eye tracker. In the figure, fixation points No. 1 and No. 2 fall into the same object area F, so the 1st and 2nd fixation points are encoded as F in the semantic fixation path; after all fixation points are grouped in sequence based on the labeled object area, a fixation object sequence for the subject viewing the current stimulus is obtained, which is the semantic fixation path F→E→D→I→B→J.

[0032] S4, based on step S3, building a visual attention map by combining the semantic gaze path of each subject;

[0033] like Figure 3 As shown, for the 6 subjects in this embodiment, the semantic gaze path of each subject is obtained based on step S3, and the number of attention transfers between different semantic object areas in the semantic gaze path of each subject is counted. For example, the number of attention transfers of node E in the figure is 2, one transfer to A and one transfer to D. A directed weighted graph is constructed with each object area as a node and the number of attention transfers between objects as the weight of the edge to form a visual attention map.Figure 4 To compare the different visual attention encoding methods in this embodiment, the comparison between the method of the present application and the existing attention encoding methods is shown, Figure 4 (a) the visual image viewed by the subject, Figure 4 (b) the attention result encoded by using the fixation point distribution, Figure 4 (c) the attention result encoded by using the attention shift, Figure 4 (d) the visual attention result encoded by using the visual attention graph (the method of the present application), it can be seen that the method of the present application can encode the regularity of attention shift between semantic objects, and can better reveal the attention behavior of the subject when viewing the image.

[0034] S5, standardizing the visual attention graph obtained in step S4 to complete the graph-based visual attention encoding;

[0035] For each node of the visual attention graph obtained in step S4, the weight of each edge starting from the node is divided by the sum of the weights of all edges starting from the node to obtain the standardized weight of the edge starting from the node, as shown in Figure 3 E in the graph, the attention shift is 2 times, the shift to A is 1 time, and the shift to D is 1 time, so the standardized weights of the edges starting from the node are E→A is 1 / 2 = 0.5, E→D is 1 / 2 = 0.5; all nodes of the attention graph are processed in turn to obtain the standardized visual attention graph.

[0036] In this embodiment, the visual attention encoding method further includes step S6, calculating the score of the attention of the subject to be evaluated on the standardized visual attention graph, and evaluating the consistency between the subject to be evaluated and the subject attention collected for constructing the standardized visual attention graph, as follows:

[0037] According to the method of steps S1 to S3, the attention of a subject to be evaluated is evaluated, first, the semantic fixation path D→I→F→J of the subject to be evaluated is constructed, and the edge weight of each attention shift on the semantic fixation path of the subject to be evaluated on the standardized visual attention graph constructed in step S5 is found.

[0038] Wherein, when the attention shift on the semantic fixation path of the subject to be evaluated has no corresponding edge in the standardized visual attention graph, the corresponding edge weight of the attention shift is set to zero; after traversing all the attention shifts on the semantic fixation path of the subject to be evaluated, the average value of all the edge weights found is taken as the score of the semantic fixation path of the subject to be evaluated on the standardized visual attention graph; the score reflects the consistency degree of the subject to be evaluated and the attention of the subject collected for constructing the standardized visual attention graph.

[0039] In this embodiment, as Figure 3As shown, according to the method from step S1 to step S3, a semantic gaze path of a subject to be evaluated is constructed as D→I→F→J, and the corresponding edge weight of each attention transfer on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph is found. Figure 3 In the standardized visual attention map shown, the edge weight of the first attention transfer D→I of the subject to be evaluated is 1.0, the edge weight of I→F is 0.5, and the weight of F→J is set to 0.0 because there is no corresponding edge; after traversing all attention transfers on the subject's semantic gaze path, the average value of all corresponding edge weights found is taken, that is, (1.0+0.5+0.0) / 3=0.5, which is used as the score of the semantic gaze path of the subject to be diagnosed on the standardized visual attention map, which is 0.5.

[0040] In summary, the method of the present invention first uses an eye movement device to collect eye movement data when the subject views a visual image, annotates the object area in the viewed image, and then groups the eye movement fixations that fall into the same annotated object area according to the annotated object area to form a semantic fixation path. Then, by counting the number of attention shifts between different semantic object areas in the semantic fixation path of each subject, a visual attention map is constructed, and the obtained visual attention map is standardized to complete the map-based visual attention encoding, and the standardized visual attention map can be used to evaluate the attention consistency of the subject. The method of the present invention can effectively solve the problem of large individual differences in the original fixation point path, has the characteristics of simple calculation and good effect, and is very suitable for encoding the visual behavior of a group.

[0041] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A graph-based visual attention encoding method, the specific steps are as follows: S1. Collect the subject's eye movement data using an eye tracking device, i.e., record the subject's eye movement data while viewing visual images using an eye tracker; in, The number of subjects is greater than or equal to 3; S2, visual image object region annotation, that is, for the visual images presented to the subjects, the main object regions contained in the images were manually annotated using open source tools; Among them, the pixels belonging to the same object in the image are marked as the same area, and the annotations of all object areas in the image are obtained; S3, constructing a semantic gaze path based on the object area marked in step S2 and the eye movement data recorded in step S1; For each subject's eye movement data collected while viewing a visual image, the eye fixations that fall within the same annotated object region are grouped together according to the temporal sequence of the fixations recorded by the eye tracker. This is done until all fixations are grouped based on the annotated object regions. This results in a sequence of fixations for the subject viewing the current visual image, which is known as the semantic fixation path. S4, based on step S3, building a visual attention map by combining the semantic gaze path of each subject; For each subject, the semantic gaze path of each subject is obtained based on step S3, the number of attention shifts between different semantic object regions in the semantic gaze path of each subject is counted, and a directed weighted graph is constructed with each object region as a node and the number of attention shifts between objects as the weight of the edge to form a visual attention map; S5. Normalize the visual attention map obtained in step S4 to complete the map-based visual attention encoding; For each node of the visual attention map obtained in step S4, the weight of each edge starting from the node is divided by the sum of the weights of all edges starting from the node to obtain the normalized weight of the edge starting from the node; all nodes of the attention map are processed in turn to obtain a normalized visual attention map.

2. A graph-based visual attention encoding method according to claim 1, characterized in that The visual attention coding method further includes step S6, calculating the score of the subject's attention on the standardized visual attention map, and evaluating the consistency between the subject's attention and the subject's attention collected by constructing the standardized visual attention map, as follows: Evaluate the attention of a subject to be evaluated according to the method of steps S1 to S3, first construct a semantic gaze path of the subject to be evaluated, and find the edge weight of each attention shift on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph constructed in step S5; Among them, when there is no corresponding edge in the standardized visual attention map for the attention transfer on the semantic gaze path of the subject to be evaluated, the edge weight corresponding to this attention transfer is set to zero; after traversing all the attention transfers on the semantic gaze path of the subject to be evaluated, the average value of all the edge weights found is taken as the score of the semantic gaze path of the subject to be evaluated on the standardized visual attention map; this score reflects the degree of consistency between the attention situation of the subject to be evaluated and the subject collected to construct the standardized visual attention map.

Images