Audiovisual integration method based on portable eye tracker

Abstract

The invention discloses an audiovisual integration method based on a portable eye tracker. The method comprises: a plurality of short videos with audiovisual data are recorded, wherein each short video has two display screens side by side, one of which is synchronized with the sound and the other one of which has a time lapse with the sound; a subject watches the short videos to search out and payattention to the synchronized videos; after watching the plurality of recorded short videos, the subject describes the watched video content verbally while facing the computer screen; the eye trackerrecords the eye movement trajectories of the subject in real time, and a computer saves the video data verbally reported by the subject; according to the eye movement trajectories of the subject andthe verbally reported video data, an audiovisual integration evaluation result is generated. Through a few minutes of short video experiments, the method can comprehensively demonstrate the ability ofthe subject to integrate the audiovisual information according to a time clue, and the experimental operation is simple and feasible. The method can be applied to healthy people of all ages, such asschool-age children and the elderly, to eescribe the development and change trajectory of the audiovisual integration capabilities.

Description

technical field [0001] The present invention relates to the technical field of audio-visual integration, in particular to an audio-visual integration method based on a portable eye tracker. Background technique [0002] Audio-visual integration is very important to our daily life. For example, observing lip changes in a noisy environment can enhance people's understanding of language, and the information redundancy of audio-visual channels can also help us detect and locate targets faster and more accurately, and can promote emotion recognition and social interaction. When integrating audiovisual information, temporal proximity is one of the important cues for the brain to decide whether to bundle or separate two sensory stimuli. In other words, only when the time difference between a pair of sensory stimuli (such as speech sound and lip movement image) is less than a certain range, can our brain perceive them as synchronous and integrate them together; The time range is c...

AI Technical Summary

Problems solved by technology

[0010] (1) The first three paradigms above (audio-visual synchrony judgment task, sound-induced flash illusion task, and McGurk variant task) are suitable for children, adolescents and adults, but the stimulus materials used are meaningless flash and pure tone stimuli, or Simple syllable videos have low ecological validity, while verbal stimuli in daily life communication and interaction are often much more complicated

Therefore, the conclusions drawn from these experiments cannot be well extended to real situations outside the laboratory, and cannot well reflect the connection between the audio-visual time integration window and more advanced verbal communication and social functions.

[0011] (2) The habituation paradigm is mainly suitable for infants and young children, and it overcomes the defect that younger groups cannot report verbally, but the experimental time is very long (it takes a long time for the subjects to synchronize the repeated occurrence video habituation)

However, the task's stimulus material is relatively simple, unable to attract the interest of older subjects, and is not suitable for older groups

More importantly, the "specific dislocation time of audio-visual information of out-of-sync videos" in this experiment was determined in advance and could not be adjusted according to the characteristics of the subjects (such as age, health status, etc.)