An audio-visual integration method based on a portable eye tracker

Abstract

The invention discloses an audio-visual integration method based on a portable eye tracker. Several short videos with audio-visual materials are recorded, and two display screens are set side by side in each short video, one of which is synchronized with the sound, and the other is synchronized with the sound. There is a time dislocation of the sound; the subject watches the short video, looks for the synchronous video and pays attention to it; after watching several short videos recorded, the subject verbally describes the content of the watched video facing the computer screen; The instrument records the subject's eye movement trajectory in real time, and saves the video data of the subject's oral report through the computer; according to the subject's eye movement trajectory and oral report video data, an audio-visual integrated evaluation result is generated. The present invention can comprehensively demonstrate the subject's ability to integrate audio-visual information according to time clues through a short video experiment of a few minutes, and the experimental operation is simple and feasible, and can be applied to healthy people of various age groups such as school-age children and the elderly , depicting the development and change trajectory of audio-visual integration ability.

Description

technical field [0001] The present invention relates to the technical field of audio-visual integration, in particular to an audio-visual integration method based on a portable eye tracker. Background technique [0002] Audio-visual integration is very important to our daily life. For example, observing lip changes in a noisy environment can enhance people's understanding of language, and the information redundancy of audio-visual channels can also help us detect and locate targets faster and more accurately, and can promote emotion recognition and social interaction. When integrating audiovisual information, temporal proximity is one of the important cues for the brain to decide whether to bundle or separate two sensory stimuli. In other words, only when the time difference between a pair of sensory stimuli (such as speech sound and lip movement image) is less than a certain range, can our brain perceive them as synchronous and integrate them together; The time range is c...

AI Technical Summary

Problems solved by technology

[0010] (1) The first three paradigms above (audio-visual synchrony judgment task, sound-induced flash illusion task, and McGurk variant task) are suitable for children, adolescents and adults, but the stimulus materials used are meaningless flash and pure tone stimuli, or Simple syllable videos have low ecological validity, while verbal stimuli in daily life communication and interaction are often much more complicated

Therefore, the conclusions drawn from these experiments cannot be well extended to real situations outside the laboratory, and cannot well reflect the connection between the audio-visual time integration window and more advanced verbal communication and social functions.

[0011] (2) The habituation paradigm is mainly suitable for infants and young children, and it overcomes the defect that younger groups cannot report verbally, but the experimental time is very long (it takes a long time for the subjects to synchronize the repeated occurrence video habituation)

However, the task's stimulus material is relatively simple, unable to attract the interest of older subjects, and is not suitable for older groups

More importantly, the "specific dislocation time of audio-visual information of out-of-sync videos" in this experiment was determined in advance and could not be adjusted according to the characteristics of the subjects (such as age, health status, etc.)