A method and system for supporting dynamic prediction of visual attention with variable duration

Abstract

The application discloses a kind of variable length supported visual attention dynamic prediction method and system, comprising: (1) the fixation point state diagram sequence of collection multi-age user fixation static image is fixed;(2) feature splitting and merging are carried out to static image to obtain low-resolution map, and then obtain its corresponding high-resolution map;(3) based on the fixation point state diagram, determine the fixation point mask, replace the fixation point mask position of fixation point state diagram with high-resolution map, replace the rest position with low-resolution map, to obtain dynamic visual fixation point sequence enhanced by fixation point;(4) using dynamic visual fixation point sequence and static image, GAIL model containing generator and discriminator is trained to optimize model parameters;(5) the visual fixation point image sequence of predicted output static image to be measured is output using generator;(6) visual fixation point image sequence is fused into video labeled with visual attention, and the method effectively represents the eye movement of human continuous dynamic occurrence.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method and system for dynamic prediction of visual attention with variable duration. Background Technology

[0002] Regarding the visualization of visual attention prediction results, existing representations of visual attention prediction results for static images are all static. For example, Chinese invention patent CN101493890B proposes a feature-based dynamic visual attention region extraction method. This method uses the principle of efficient coding to measure the incremental coding length index for each feature in the image, continuously sampling, so that data from different frames can guide the saliency processing together. However, the final display result is still represented as a static prediction map, without considering that the visual attention of the real human eye is continuously and dynamically changing over time and is coherent, and without making dynamic predictions of variable duration.

[0003] Regarding the assessment of visual attention among different user groups, existing visual attention prediction technologies often use datasets representing a single population, failing to segment users and resulting in unique predictions. For example, Chinese invention patent CN114170537A constructs a multimodal model to derive fixed visual interest regions. However, this single-result visual attention prediction does not consider individual differences; the heatmap region represents a single user's result, without segmenting the research population or showcasing differences between users, thus failing to simulate individual variations. Similarly, Chinese invention patent CN114092900A discloses a driver's visual attention prediction method, targeting a single user group—drivers—in driving scenarios. Another example is Chinese invention patent CN111951637A, which discloses a task-scenario-related method for extracting visual attention allocation patterns for drone pilots. This method predicts visual attention for drone pilots, covering a single user group without distinguishing individuals within the same user group, resulting in a fixed and unique prediction. Summary of the Invention

[0004] In view of the above, the purpose of this invention is to provide a method and system for dynamic prediction of visual attention with variable duration, which predicts the dynamic changes of the user's gaze point and fuses the dynamic changes to form a video for visualization.

[0005] To achieve the above-mentioned objectives, an embodiment provides a method for dynamic prediction of visual attention with variable duration, comprising the following steps:

[0006] Step 1: Collect gaze state map sequences of static images viewed by users of multiple age groups;

[0007] Step 2: Perform feature splitting and merging on the static image to obtain a low-resolution image, and then obtain the corresponding high-resolution image based on the low-resolution image;

[0008] Step 3: Determine the gaze point mask based on the gaze point state map, replace the gaze point mask position in the gaze point state map with a high-resolution map, and replace the remaining positions with a low-resolution map to obtain a dynamic visual gaze point sequence with gaze point enhancement.

[0009] Step 4: Train the GAIL model containing the generator and discriminator using dynamic visual gaze sequence and static images to optimize the model parameters;

[0010] Step 5: Input the static image to be predicted into the parameter-optimized generator, and use the generator to predict the output image sequence of visual gaze points representing visual attention.

[0011] Step 6: Based on the user-specified variable duration, fuse the visual fixation point image sequence into a video labeled with visual attention.

[0012] Preferably, in step 1, the user group is first segmented according to age group, and the dynamic eye movement data of the user under different task-driven conditions is recorded using an eye tracker, which is the obtained fixation state map sequence.

[0013] Preferably, when acquiring the gaze state map sequence, a gap of at least 5 seconds is left between images to avoid delays in responding to previously noticed objects or locations.

[0014] Preferably, in step 2, after decomposing the static image into three dimensions of color, intensity, and orientation, the three dimensions of features are superimposed to obtain a low-resolution image.

[0015] Preferably, in step 2, the super-resolution reconstruction algorithm SRCNN is used to convert the low-resolution image into the corresponding high-resolution image.

[0016] Preferably, in step 3, determining the gaze point mask based on the gaze point state map includes: discretizing the gaze point state map into a grid in space, and using the grid where the gaze point is located as the gaze point mask;

[0017] In step 3, the dynamic visual fixation sequence for fixation enhancement is obtained using the following formula:

[0018] B t+1 =M t ⊙H+(1-M t )⊙B t

[0019] B0 = L

[0020] Among them, B tand B t+1 resolution representation of the t-th and t+1-th gaze point state map, L represents a low resolution map, B0 represents an initial gaze point state map, H represents a high resolution map, M t gaze point mask determined according to the t-th gaze point state map, and the symbol represents a Hadamard product.

[0021] Preferably, in the GAIL model, the generator receives a static image and generates a predicted visual gaze point image sequence, the input static image as a state, the predicted visual gaze point image sequence as a predicted action, a gaze point state map sequence collected for the static image as a real action, the state and the predicted action form false data, and the state and the real action form true data, the discriminator receives the false data and the true data and outputs a discrimination result of the false data and the true data, and each time the discriminator makes a wrong discrimination, the false data of the generator will be rewarded, so that the false data is more likely to be generated in the future.

[0022] Preferably, in step 6, when the visual gaze point image sequence contains T visual gaze point images and the user-specified variable length is S, the number of frames F required for fusion is calculated according to the variable length and the preset frame rate, the 0th visual gaze point image to the T-Fth visual gaze point image are fused into one frame, the 1st visual gaze point image to the T-F+1th visual gaze point image are fused into one frame, and so on, until the Fth visual gaze point image to the Tth visual gaze point image are fused into the last frame, and the gaze points are visualized as circular scatter points, the synthesized frame number is F, and the video picture with a length of S is synthesized.

[0023] To achieve the above-mentioned purposes, the embodiment further provides a visual attention dynamic prediction system supporting variable length, comprising a front-end display and a back-end server, wherein the front-end display completes desktop application configuration by using electron and react technologies, and the back-end server stores a parameter-optimized GAIL model constructed by the above-mentioned visual attention dynamic prediction method supporting variable length;

[0024] The back-end server receives a static image input by a user, calls the parameter-optimized GAIL model, and predicts and outputs a visual gaze point image sequence representing visual attention by using a parameter-optimized generator; then, according to a variable length specified by the user, the visual gaze point image sequence is fused into a video labeled with visual attention;

[0025] The front-end display visually presents the video labeled with visual attention.

[0026] Compared with the prior art, the present application has at least the following beneficial effects:

[0027] In constructing sample data, sample data is constructed based on different user groups and different user differentiation, and on this basis, a usable visual attention model is obtained by training, realizing the subdivision of user age layers and the prediction of individual difference disturbance, and more accurately predicting the visual attention changes of individuals and groups and their differential expression.

[0028] In the gaze point prediction of the attention representation, the time dimension is considered, and the predicted visual gaze point image sequence can more effectively represent the continuous dynamic occurrence of human gaze point motion. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0030] Figure 1 is a flowchart of the visual attention dynamic prediction method with variable length provided by the embodiment;

[0031] Figure 2 is a construction schematic diagram of GAIL sample data provided by the embodiment;

[0032] Figure 3 is a training schematic diagram of the GAIL model provided by the embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0034] Figure 1 is a flowchart of the visual attention dynamic prediction method with variable length provided by the embodiment. As shown in Figure 1 The visual attention dynamic prediction method with variable length provided by the embodiment comprises the following steps:

[0035] Step 1, collecting the gaze point state diagram sequence of the gaze static image of the multi-age group users.

[0036] In the embodiment, the dynamic eye movement data set of different users when gazing at the static image, that is, the gaze point state diagram sequence, is collected by the eye tracker, including: dynamic eye movement data record of human eyes on the picture for 1-3 seconds in the natural state without task driving and dynamic eye movement data record of human eyes on the picture for 1-3 seconds in the specific task driving. The experiment subdivides the user group according to the age stage, collects the dynamic eye movement data set of users in different age stages under different task driving, and there is a 5s blank time between pictures in the process of data collection, so as to avoid the delay in reaction to the object or position that has been noticed before. The collected dynamic eye movement data set is stored and dynamically updated in the user data in the database, so as to improve the accuracy of the model.

[0037] Step 2, feature splitting and merging of the static image to obtain a low-resolution image, and obtaining a corresponding high-resolution image based on the low-resolution image.

[0038] In the embodiment, as shown in Figure 2 , noise data cleaning is performed on the static image, and the static image is decomposed in three dimensions of color, intensity and direction, then the three dimension features are superimposed to obtain a low-resolution image L, and then the low-resolution image is converted into a corresponding high-resolution image H through the super-resolution reconstruction algorithm SRCNN. The specific principle is as follows: for the input low-resolution image, SRCNN first uses bicubic interpolation to enlarge it to the target size, then uses a three-layer convolutional neural network to fit the nonlinear mapping between the low-resolution image and the high-resolution image, and finally outputs the result of the convolutional neural network as the reconstructed high-resolution image.

[0039] Step 3, determining the gaze point mask based on the gaze point state diagram, replacing the gaze point mask position of the gaze point state diagram with the high-resolution image, and replacing the remaining positions with the low-resolution image, to obtain a dynamic visual gaze point sequence enhanced by the gaze point.

[0040] In the embodiment, for each gaze point state diagram in the gaze point state diagram sequence, each gaze point state diagram is discretized into a 20*32 grid in space, which is defined as an action space to reduce the complexity of prediction. Each gaze point state diagram contains a single gaze point, and for each gaze point, as shown in Figure 2 , the state is updated by replacing the low-resolution part with the corresponding high-resolution part at the new gaze point, which is expressed as follows:

[0041] B t+1 =M t ⊙H+(1-M t )⊙B t

[0042] B0=L

[0043] Wherein, B t and Bt+1 L represents a low-resolution map, B0 represents an initial gaze map, H represents a high-resolution map, M t L represents a low-resolution map, B0 represents an initial gaze map, H represents a high-resolution map, M

[0044] Step 4, the GAIL model containing the generator and the discriminator is trained using the dynamic visual gaze point sequence and the static image to optimize the model parameters.

[0045] In the embodiment, the GAIL model containing the generator and the discriminator is trained using the dynamic visual gaze point sequence and the static image to optimize the model parameters. The generator after parameter optimization is used as a visual attention prediction model. The visual attention prediction model is used to dynamically predict the user's interest area.

[0046] As shown in Figure 3 In the GAIL model, the generator receives a static image and generates a predicted visual gaze point image sequence (pseudo eye movement), the input static image as a state, the predicted visual gaze point image sequence as a predicted action, the gaze point state map sequence collected for the static image as a real action, the state and the predicted action form a false state-action pair as false data, the state and the real action form a true state-action pair as true data, the discriminator receives the false data and the true data and outputs the discrimination results of the false data and the true data, and each time the discriminator makes a wrong discrimination, the false data of the generator will be rewarded to make the false data more likely to be generated in the future. During the training, the discriminator becomes more and more difficult to distinguish the data of the generator and the real data, and the data generated by the generator becomes more and more like human.

[0047] Step 5, input the static image to be predicted into the generator after parameter optimization, and use the generator to predict and output a visual gaze point image sequence representing visual attention.

[0048] In the embodiment, the static image to be predicted is input into the visual attention prediction model, and a visual gaze point image sequence representing visual attention is predicted and output after calculation.

[0049] Step 6, according to the variable length specified by the user, the visual gaze point image sequence is fused into a video labeled with visual attention.

[0050] In the embodiment, a video processing algorithm with variable length is supported, the visual fixation point image sequence is visualized as a video with variable length, and is superimposed on the input image to realize dynamic visual attention display.

[0051] In the embodiment, the visual fixation point image sequence corresponding to the output of step 5 contains T visual fixation point images, when the variable length specified by the user is S, the fusion required frame number F is calculated according to the variable length and the preset frame rate, the 0th visual fixation point image to the T-Fth visual fixation point image is fused into one frame, the 1st visual fixation point image to the T-F+1th visual fixation point image is fused into one frame, and so on, until the Fth visual fixation point image to the Tth visual fixation point image is fused into the last frame, and the fixation point is visualized in the form of a circular scatter point, the synthesized frame number is F, and the video picture with a length of S is obtained.

[0052] Based on the same inventive concept, the embodiment also provides a visual attention dynamic prediction system supporting variable length, comprising a front-end display and a back-end server, wherein the front-end display completes the desktop application configuration by using electron and react technologies, and the back-end server stores a parameter-optimized GAIL model constructed by the above-mentioned visual attention dynamic prediction method supporting variable length.

[0053] The user inputs a static image in the front-end display, completes the setting of the prediction scenario and the prediction parameter, the back-end server receives the static image input by the user, starts the visual attention prediction model, and predicts the visual fixation point image sequence representing the visual attention by using the parameter-optimized generator; then, according to the variable length specified by the user, the visual fixation point image sequence is fused into a video labeled with visual attention; the front-end display visualizes and presents the video labeled with visual attention, and the specific video is presented in the form of a dynamic heat map in the application.

[0054] The visual attention prediction method and system supporting variable length provided in the above-mentioned embodiment support user group subdivision and visual attention prediction of individual differences in multiple simulations, the usable visual attention model is obtained by training the dynamic visual attention dataset of the subdivided user group, the output of the dynamic visual attention prediction and evaluation of the static picture input with variable length is realized, and the accuracy of the prediction result is improved.

[0055] The visual attention prediction method supporting variable length provided in the above-mentioned embodiment is trained on the user-subdivided visual attention dataset with time sequence to obtain the usable visual attention model, and the video is processed by the algorithm, the dynamic visual interest region is generated after the input of any static image, and the visual user visual attention focus is realized.

[0056] The visual attention prediction system with variable time length provided by the above embodiment constructs a system application differentiated by different groups of users. For the picture data input by the user, parameters such as selected time length, simulation times, and user group are supported, the background can call the model to process the input data, and finally feedback the generated visual attention prediction dynamic diagram.

[0057] The specific embodiments described above have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modification, supplement, and equivalent replacement within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. A method for dynamic prediction of visual attention with variable duration, characterized in that, Includes the following steps: Step 1: Collect gaze state map sequences of static images viewed by users of multiple age groups; Step 2: Perform feature splitting and merging on the static image to obtain a low-resolution image, and then obtain the corresponding high-resolution image based on the low-resolution image; Step 3: Determine the gaze point mask based on the gaze point state map, replace the gaze point mask position in the gaze point state map with a high-resolution map, and replace the remaining positions with a low-resolution map to obtain a dynamic visual gaze point sequence with gaze point enhancement. Step 4: The GAIL model, which includes a generator and a discriminator, is trained using dynamic visual gaze point sequences and static images to optimize model parameters. In the GAIL model, the generator receives static images and generates a predicted sequence of visual gaze point images. The input static images serve as the state, the predicted sequence of visual gaze point images serves as the predicted action, and the gaze point state map sequence collected from the static images serves as the real action. The state and the predicted action form fake data, and the state and the real action form real data. The discriminator receives the fake data and the real data and outputs the discrimination result between the fake data and the real data. Each time the discriminator makes an incorrect judgment, the generator's fake data will be rewarded, making it more likely that fake data will be generated in the future. Step 5: Input the static image to be predicted into the parameter-optimized generator, and use the generator to predict the output image sequence of visual gaze points representing visual attention. Step 6: Based on the user-specified variable duration, fuse the visual fixation point image sequence into a video labeled with visual attention. Specifically, when the visual fixation point image sequence contains T visual fixation point images and the user-specified variable duration is S, calculate the number of frames F required for fusion based on the variable duration and the preset frame rate. Fuse the 0th visual fixation point image to TF visual fixation point images into one frame, fuse the 1st visual fixation point image to the T-F+1th visual fixation point image into one frame, and so on, until the Fth visual fixation point image to the Tth visual fixation point image are fused into the last frame. The fixation points are visualized using circular scattered points, and a video frame with F frames and a duration of S is synthesized.

2. The method for dynamic prediction of visual attention with variable duration according to claim 1, characterized in that, In step 1, the user group is first segmented according to age group, and the dynamic eye movement data of users under different task-driven conditions is recorded using an eye tracker, which is the resulting fixation state map sequence.

3. The method for dynamic prediction of visual attention with variable duration according to claim 2, characterized in that, When acquiring gaze state map sequences, a gap of at least 5 seconds is left between images to avoid delays in responding to previously noticed objects or locations.

4. The method for dynamic prediction of visual attention with variable duration according to claim 1, characterized in that, In step 2, the static image is decomposed into three dimensions of features: color, intensity, and orientation. The three dimensions of features are then superimposed to obtain a low-resolution image.

5. The method for dynamic prediction of visual attention with variable duration according to claim 1, characterized in that, In step 2, the super-resolution reconstruction algorithm SRCNN is used to convert the low-resolution image into the corresponding high-resolution image.

6. The method for dynamic prediction of visual attention with variable duration according to claim 1, characterized in that, In step 3, determining the gaze point mask based on the gaze point state map includes: discretizing the gaze point state map into a grid in space, and using the grid where the gaze point is located as the gaze point mask. In step 3, the dynamic visual fixation sequence for fixation enhancement is obtained using the following formula: B t+1 =M t ⊙H+(1-M t )⊙B t B0 = L Among them, B t and B t+1 Let L represent the low-resolution map, B0 represent the initial fixation state map, H represent the high-resolution map, and M represent the fixation state map. t This represents the gaze mask determined based on the t-th gaze state diagram, and the ⊙ symbol represents the Hadamard product.

7. A dynamic prediction system for visual attention with variable duration, characterized in that, It includes a front-end display and a back-end server, wherein the front-end display uses Electron and React technologies to complete the desktop application configuration, and the back-end server stores the parameter-optimized GAIL model constructed by the dynamic prediction method for visual attention with variable duration as described in any one of claims 1-6. The backend server receives static images input by the user, calls the parameter-optimized GAIL model, and uses the parameter-optimized generator to predict and output a sequence of visual fixation points representing visual attention; then, according to the variable duration specified by the user, the sequence of visual fixation points is fused into a video labeled with visual attention. The front-end display visualizes the video labeled with visual attention.

Images