Visual attention coding method based on graph

Abstract

The invention discloses a graph-based visual attention coding method, which comprises the following steps of: firstly, acquiring eye movement data when a subject watches a visual image by using eye movement equipment, marking object areas in the watched image, and sequentially classifying eye movement fixation points falling into the same marked object area into one group according to the marked object areas; the method comprises the following steps of: constructing a semantic gazing path, constructing a visual attention graph by counting attention transfer times among different semantic object areas in the semantic gazing path of each subject, and performing standardization processing on the obtained visual attention graph to finish graph-based visual attention coding. And the attention consistency of the subject can be evaluated by using the standardized visual attention map. The method can effectively solve the problem of large individual difference of original fixation point paths, has the characteristics of simple calculation, good effect and the like, and is very suitable for visual behaviors of coded groups.

Description

Technical Field

[0001] The present invention belongs to the field of cognitive computing technology, and in particular relates to a graph-based visual attention encoding method. Background Art

[0002] Visual attention plays a vital role when the human visual system performs visual tasks by interacting with physical scenes. By quantitatively analyzing human eye movement behavior, visual attention behavior patterns in different states can be identified, which is of great significance to the design of products such as human-computer interaction. In the field of computer vision, a large number of studies have simulated the visual attention and behavior of human observers when viewing scenes by designing algorithms to predict the distribution of visual fixation points or scanning paths. These methods can effectively improve the performance of brain-like intelligent systems and exhibit some human-like behaviors, and have broad application prospects in the fields of assisted driving and fatigue detection.

[0003] At present, most of the calculation and analysis methods are typical gaze point distribution estimation methods, such as the literature "A. Borjiand L. Itti, State-of-the-art in visual attention modeling. IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 185-207, 2012". Secondly, some methods are mainly used to quantitatively analyze the scanning path, such as the literature "C. Xia, J. Han, and D. Zhang, Evaluation of saccadic scanpath prediction: Subjective assessment database and recurrent neural network based metric. IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 12, pp. 4378-4395, 2020". These methods are based on the original gaze point data or scanning paths collected by human observers to encode visual attention, and cannot capture the common pattern of visual attention well. Moreover, these attention encoding methods do not consider the semantic information of the objects in the viewed scene, which leads to large individual differences in attention encoding and cannot effectively encode the common visual attention behavior of humans. Therefore, developing more stable visual attention encoding methods based on eye movement data to reduce individual differences in the data is of great significance for improving visual attention prediction and attention behavior recognition. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a graph-based visual attention encoding method, which solves the problem that existing methods cannot effectively encode human common visual attention behavior.

[0005] The technical solution adopted by the present invention is: a graph-based visual attention encoding method, the specific steps are as follows:

[0006] S1. Collecting the eye movement data of the subject using an eye movement device, that is, recording the eye movement data of the subject while viewing the visual image based on the eye tracker;

[0007] Wherein, the number of subjects is greater than or equal to 3.

[0008] S2, visual image object region annotation, that is, for the visual images presented to the subjects, the main object regions contained in the images were manually annotated using open source tools;

[0009] Among them, the pixels belonging to the same object in the image are marked as the same area, and the annotations of all object areas in the image are obtained.

[0010] S3, constructing a semantic fixation path according to the object area marked in step S2 and the eye movement data recorded in step S1;

[0011] For the eye movement data collected when each subject views the visual image, the eye movement fixation points that fall into the same labeled object area are grouped together according to the time sequence of the eye movement fixation points recorded by the eye tracker, until all the fixation points are grouped in sequence based on the labeled object areas. In this way, a fixation object sequence for the subject when viewing the current visual image is obtained, which is the semantic fixation path.

[0012] S4, based on step S3, establishing a visual attention map in combination with the semantic fixation path of each subject;

[0013] For each subject, the semantic gaze path of each subject is obtained based on step S3, the number of attention transfers between different semantic object areas in the semantic gaze path of each subject is counted, and a directed weighted graph is constructed with each object area as a node and the number of attention transfers between objects as the weight of the edge to form a visual attention map.

[0014] S5, normalizing the visual attention map obtained in step S4 to complete the map-based visual attention encoding;

[0015] For each node of the visual attention map obtained in step S4, the weight of each edge starting from the node is divided by the sum of the weights of all edges starting from the node to obtain the standardized weight of the edge starting from the node; all nodes of the attention map are processed in turn to obtain a standardized visual attention map.

[0016] Furthermore, the visual attention encoding method further includes step S6, calculating the score of the subject's attention to be evaluated on the standardized visual attention map, and evaluating the consistency between the subject's attention to be evaluated and the subject's attention collected by constructing the standardized visual attention map, as follows:

[0017] According to the method of step S1 to step S3, the attention of a subject to be evaluated is evaluated. First, the semantic gaze path of the subject to be evaluated is constructed, and the edge weight of each attention transfer on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph constructed in step S5 is found.

[0018] Among them, when the attention transfer on the semantic gaze path of the subject to be evaluated has no corresponding edge in the standardized visual attention map, the edge weight corresponding to this attention transfer is set to zero; after traversing all the attention transfers on the semantic gaze path of the subject to be evaluated, the average of all the edge weights found is taken as the score of the semantic gaze path of the subject to be evaluated on the standardized visual attention map; this score reflects the degree of consistency between the attention of the subject to be evaluated and the subject collected to construct the standardized visual attention map.

[0019] Beneficial effects of the present invention: The method of the present invention first uses an eye movement device to collect eye movement data when a subject views a visual image, annotates the object area in the viewed image, and group eye movement fixation points that fall into the same annotated object area according to the annotated object area to form a semantic fixation path, and then constructs a visual attention map by counting the number of attention transfers between different semantic object areas in the semantic fixation path of each subject, and standardizes the obtained visual attention map to complete the map-based visual attention encoding, and the standardized visual attention map can be used to evaluate the attention consistency of the subject. The method of the present invention can effectively solve the problem of large individual differences in the original fixation point path, has the characteristics of simple calculation and good effect, and is very suitable for encoding the visual behavior of a group. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of a graph-based visual attention encoding method of the present invention.

[0021] Figure 2 A schematic diagram of constructing a semantic gaze path in an embodiment of the present invention.

[0022] Figure 3 A schematic diagram of constructing a visual attention map in an embodiment of the present invention.

[0023] Figure 4 2 is a comparison chart of different visual attention encoding methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0025] like Figure 1 As shown, a flowchart of a graph-based visual attention encoding method of the present invention is shown, and the specific steps are as follows:

[0026] S1. Use eye movement equipment to collect eye movement data of subjects, that is, based on commercial eye movement collection equipment - eye tracker, record the eye movement data of subjects when they are viewing visual images;

[0027] Among them, the number of subjects is greater than or equal to 3, and data of 3 or more subjects are needed to appropriately construct a visual attention map. In this embodiment, the number of subjects is 6.

[0028] S2, visual image object region annotation, that is, for the visual images presented to the subjects, the open source tool LabelMe was used to manually annotate the main object regions contained in the images;

[0029] The pixels belonging to the same object in the image are marked as the same area, and the annotations of all object areas in the image are obtained. Figure 2 As shown, the figure includes 10 objects, numbered A to J.

[0030] S3, constructing a semantic fixation path according to the object area marked in step S2 and the eye movement data recorded in step S1;

[0031] like Figure 2 As shown in the figure, for each subject's eye movement data (numbered 1, 2, 3, ..., 9) collected when viewing the visual image, the eye movement fixation points that fall into the same labeled object area are grouped together according to the time sequence of the eye movement fixation points recorded by the eye tracker. In the figure, fixation points No. 1 and No. 2 fall into the same object area F, so the 1st and 2nd fixation points are encoded as F in the semantic fixation path; after all fixation points are grouped in sequence based on the labeled object areas, a fixation object sequence for the subject viewing the current stimulus is obtained, which is the semantic fixation path F→E→D→I→B→J.

[0032] S4, based on step S3, establishing a visual attention map in combination with the semantic fixation path of each subject;

[0033] like Figure 3 As shown, for the 6 subjects in this embodiment, the semantic gaze path of each subject is obtained based on step S3, and the number of attention transfers between different semantic object areas in the semantic gaze path of each subject is counted. For example, the number of attention transfers of node E in the figure is 2, one transfer to A and one transfer to D. A directed weighted graph is constructed with each object area as a node and the number of attention transfers between objects as the weight of the edge to form a visual attention map. Figure 4 This is a comparison of different visual attention encoding methods in this embodiment, showing the comparison between the method of the present invention and the existing attention encoding method. Figure 4 (a) is the visual image viewed by the subject, Figure 4 (b) To encode the attention results using the gaze point distribution, Figure 4 (c) To encode the attention results using attention transfer, Figure 4 (d) is the method of encoding visual attention results using a visual attention map (the method of the present invention). It can be seen that the method of the present invention can encode the law of attention transfer between semantic objects and can better reveal the attention behavior of the subjects when viewing images.

[0034] S5, normalizing the visual attention map obtained in step S4 to complete the map-based visual attention encoding;

[0035] For each node of the visual attention graph obtained in step S4, the weight of each edge from the node is divided by the sum of the weights of all edges from the node to obtain the standardized weight of the edge from the node, such as Figure 3 As shown, in the figure, the attention of node E is transferred out 2 times, transferred to A once, and transferred to D once, so the standardized weights of the edges starting from this node are E→A is 1 / 2=0.5, and E→D is 1 / 2=0.5; all nodes of the attention graph are processed in turn to obtain a standardized visual attention graph.

[0036] In this embodiment, the visual attention encoding method further includes step S6, calculating the score of the subject's attention to be evaluated on the standardized visual attention map, and evaluating the consistency between the subject's attention to be evaluated and the subject's attention collected by constructing the standardized visual attention map, as follows:

[0037] According to the method of step S1 to step S3, the attention of a subject to be evaluated is evaluated. First, the semantic gaze path D→I→F→J of the subject to be evaluated is constructed, and the edge weight of each attention transfer on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph constructed in step S5 is found.

[0038] Among them, when the attention transfer on the semantic gaze path of the subject to be evaluated has no corresponding edge in the standardized visual attention map, the edge weight corresponding to this attention transfer is set to zero; after traversing all the attention transfers on the semantic gaze path of the subject to be evaluated, the average of all the edge weights found is taken as the score of the semantic gaze path of the subject to be evaluated on the standardized visual attention map; this score reflects the degree of consistency between the attention of the subject to be evaluated and the subject collected to construct the standardized visual attention map.

[0039] In this embodiment, Figure 3As shown, according to the method from step S1 to step S3, a semantic gaze path of a subject to be evaluated is constructed as D→I→F→J, and the corresponding edge weight of each attention transfer on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph is found. Figure 3 In the standardized visual attention map shown, the edge weight of the first attention transfer D→I of the subject to be evaluated is 1.0, the edge weight of I→F is 0.5, and the weight of F→J is set to 0.0 because there is no corresponding edge; after traversing all attention transfers on the subject's semantic gaze path, the average value of all corresponding edge weights found is taken, that is, (1.0+0.5+0.0) / 3=0.5, which is taken as the score of the semantic gaze path of the subject to be diagnosed on the standardized visual attention map, which is 0.5.

[0040] In summary, the method of the present invention first uses an eye movement device to collect eye movement data when the subject views a visual image, annotates the object area in the viewed image, and group the eye movement fixation points that fall into the same annotated object area according to the annotated object area to form a semantic fixation path, and then constructs a visual attention map by counting the number of attention transfers between different semantic object areas in the semantic fixation path of each subject, and standardizes the obtained visual attention map to complete the map-based visual attention encoding, and the standardized visual attention map can be used to evaluate the attention consistency of the subject. The method of the present invention can effectively solve the problem of large individual differences in the original fixation point path, has the characteristics of simple calculation and good effect, and is very suitable for encoding the visual behavior of the group.

[0041] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A graph-based visual attention encoding method, the specific steps are as follows: S1. Collecting the eye movement data of the subject using an eye movement device, that is, recording the eye movement data of the subject while viewing the visual image based on the eye tracker; in, The number of subjects is greater than or equal to 3; S2, visual image object region annotation, that is, for the visual images presented to the subjects, the main object regions contained in the images were manually annotated using open source tools; Among them, the pixels belonging to the same object in the image are marked as the same area, and the annotations of all object areas in the image are obtained; S3, constructing a semantic fixation path according to the object area marked in step S2 and the eye movement data recorded in step S1; For each subject's eye movement data collected when viewing the visual image, the eye movement fixation points falling into the same annotated object area are grouped together in sequence according to the time sequence of the eye movement fixation points recorded by the eye tracker, until all fixation points are grouped in sequence based on the annotated object areas, and a fixation object sequence of the subject viewing the current visual image is obtained, which is the semantic fixation path; S4, based on step S3, establishing a visual attention map in combination with the semantic fixation path of each subject; For each subject, based on step S3, the semantic gaze path of each subject is obtained respectively, the number of attention transfers between different semantic object regions in the semantic gaze path of each subject is counted, and a directed weighted graph is constructed with each object region as a node and the number of attention transfers between objects as the weight of the edge to form a visual attention map; S5, normalizing the visual attention map obtained in step S4 to complete the map-based visual attention encoding; For each node of the visual attention map obtained in step S4, the weight of each edge starting from the node is divided by the sum of the weights of all edges starting from the node to obtain the standardized weight of the edge starting from the node; all nodes of the attention map are processed in turn to obtain a standardized visual attention map.

2. A graph-based visual attention encoding method according to claim 1, characterized in that: The visual attention coding method further includes step S6, calculating the score of the subject's attention on the standardized visual attention map, and evaluating the consistency between the subject's attention and the subject's attention collected by constructing the standardized visual attention map, as follows: According to the method of step S1 to step S3, the attention of a subject to be evaluated is evaluated, firstly, a semantic gaze path of the subject to be evaluated is constructed, and the edge weight of each attention transfer on the semantic gaze path of the subject to be evaluated on the standardized visual attention graph constructed in step S5 is found; Among them, when the attention transfer on the semantic gaze path of the subject to be evaluated has no corresponding edge in the standardized visual attention map, the edge weight corresponding to this attention transfer is set to zero; after traversing all the attention transfers on the semantic gaze path of the subject to be evaluated, the average of all the edge weights found is taken as the score of the semantic gaze path of the subject to be evaluated on the standardized visual attention map; this score reflects the degree of consistency between the attention of the subject to be evaluated and the subject collected to construct the standardized visual attention map.

Images