Device and method for measuring visual and auditory mutual influence

By using a multi-channel spherical speaker system and LED light in the silencer room, combined with a head rotation tracking device and an eye tracker, the problem of difficulty in controlling mutual influence in the prior art audio vision experiments is solved, and the mutual influence of auditory and visual information is achieved The precise detection and analysis of the role deepens the understanding of the impact of hearing on visual perception in complex environments.

CN119073900APending Publication Date: 2024-12-06WUHAN POLYTECHNIC UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411194195.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing audio-visual experimental methods are difficult to accurately control and detect the mutual influence of auditory and visual stimuli at the same time, and there is a lack of effective experimental methods and environment to study the impact of multiple continuous auditory events on visual event location.

Method used

Design an audio-visual mutual influence experimental method, and use a multi-channel spherical speaker system, LED lights and head rotation tracking device to conduct experiments in the silence room. By setting visual and auditory stimuli, the head rotation and gaze direction of the experimental subject is recorded. Combining eye tracker data ensures accurate control of experimental conditions and data collection.

Benefits of technology

In-depth research on the interaction between auditory and visual information in complex environments has been achieved, the accuracy of experimental results has been improved, and a more comprehensive understanding of the impact of auditory on visual perception has been provided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119073900A_ABST
    Figure CN119073900A_ABST
Patent Text Reader

Abstract

The invention provides an audio and visual interaction experiment method. In an anechoic chamber equipped with a multi-channel spherical loudspeaker system, an LED lamp, a head rotation tracking device and an eye tracker, a plurality of experiment objects are invited to perform an experiment so as to determine the interaction relationship between hearing and vision. The multi-channel spherical loudspeaker system is composed of X loudspeakers which are distributed in all directions of the laboratory. N (n is an odd number) LED yellow lamps are installed on a 0-degree horizontal plane, one LED yellow lamp is located at a central azimuth angle of 0 degree, two remaining LED lamps are taken as a group, and each group of LED lamps are respectively arranged at a horizontal azimuth angle + / -theta i degree ((n-1) / 2 different theta is selected according to the number of n, and i is equal to (n-1) / 2); a series of auditory and visual stimulation is generated through a multi-channel spherical loudspeaker system and an LED lamp, and an experiment object is required to sit in the center of a room and answer whether the last light flickering position is located on the left side or the right side after the experiment is finished; the head rotation tracking device tracks the head movement by arranging a high-definition camera in front of the human face, and the experiment is disabled if the head obviously rotates; in addition, an experiment object needs to use an eye tracker to record the gazing direction and time, and if the gazing direction is inconsistent with the light flickering position or the time is insufficient, the experiment is abandoned. According to the invention, the mutual influence between auditory and visual stimulation can be accurately detected and analyzed, and the accuracy of an experimental result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of perceptual psychology and relates to a device and method for measuring visual and auditory interaction. Background Art

[0002] Human vision and hearing are two important ways of perceiving the world. They collect information through different sensory organs and transmit the information to the brain through the nervous system for integrated processing. There is a close connection and interaction between hearing and vision. For example, when we hear an attention-grabbing sound, such as an explosion, our visual attention will be drawn to the source of the sound so that we can better observe the surrounding environment and respond to potential threats or emergencies. Or when we communicate with others, observing the speaker's facial expressions while hearing the language helps us understand their language meaning and emotional state more accurately. Hearing and vision are intertwined in our perceptual experience. They not only play unique roles in their respective fields, but also influence and collaborate with each other in many ways, enriching our overall perception and understanding of the world. This cross-sensory integration helps to improve the accuracy of perception and plays a vital role in people's daily lives.

[0003] The impact of hearing on vision is a complex and thought-provoking research field, involving multiple disciplines such as perceptual psychology, neuroscience, and cognitive science. Existing audio-visual experiments usually conduct auditory and visual stimulation experiments separately, and rarely involve research on the mutual influence between the two. Traditional experimental methods and experimental environments often cannot simultaneously meet the precise control and detection of auditory and visual stimuli, so a new experimental method and laboratory layout are needed to overcome these problems.

[0004] The main purpose of this invention is to study the impact of multiple continuous auditory events on the positioning of visual events, to gain an in-depth understanding of how auditory events change people's perception and positioning of visual events in complex environments, to determine in which situations hearing has a more significant impact on visual perception, to deepen our understanding of the interaction between auditory and visual information in complex environments, and to provide a more comprehensive understanding of people's perception in multi-stimulus environments. Summary of the invention

[0005] In view of the shortcomings of the prior art methods, the present invention provides an audio-visual interaction experimental method.

[0006] The present invention provides an audio-visual interaction experimental method, in which multiple experimental subjects are experimented in an anechoic room equipped with a multi-channel spherical speaker system, an LED lamp, a head rotation tracking device, and an eye tracker to determine the interaction relationship between hearing and vision, n (n is an odd number) LED yellow lights are installed on a 0-degree plane, one of which is located at a central azimuth angle of 0 degrees, and the remaining LED lamps are grouped in two, and each group of LED lamps is respectively arranged at a horizontal azimuth angle of + / -θ i The experimental subject's seat was arranged in the center of the room, 3 meters away from the LED light and the speaker. The speaker and the LED light were covered with a black light-transmitting curtain to ensure that the light intensity was strong enough to penetrate the curtain. At the experimental subject's observation point, the size of the light was 0.42 degrees of the visual field and the brightness was 41 cd / m 2 ,The room was kept dark during the experiment.

[0007] Visual and auditory stimulations need to be set before the experiment. The visual stimulation consists of a single 60-millisecond sequence of light flashes. The flashing order of the LED light is set by the program, and the time interval between each light flash is 150 milliseconds. The auditory stimulation consists of a single 60-millisecond sequence of white noise bursts. The sound pressure of the noise burst is 70-75dB at the hearing point of the experimental subject. The light flashes and noise bursts are combined. Each experiment presents a series of light flashes and burst sequences. The number of light flashes varies randomly, with a minimum of X1 times and a maximum of X2 times. The position, number and time of the noise bursts are set by yourself.

[0008] During the experiment, the head rotation of the subject was recorded by the head rotation tracking device in the laboratory. If the subject made obvious head rotation (such as horizontal rotation exceeding θ1 degrees, vertical tilt exceeding θ2 degrees), the experiment was invalidated. In addition, it was necessary to use a glasses-type eye tracker to record the gaze direction and time of the subject. If the gaze direction was inconsistent with the light flash position or the gaze time was less than m milliseconds, the experiment was invalidated.

[0009] During the experiment, the subjects were required to pay attention to the light flashes and report the location of the last flash at the end of the test. The answer time was limited to 1 second. Experiments that exceeded 1 second or had missing results were repeated until 20 repeated results were obtained for each subject in each experimental condition.

[0010] After the test, the subjects gave their results through the keyboard. Pressing the left arrow means that the last light flash perceived was on the left, the right arrow means it was on the right, and the top arrow starts a new experiment. During the experiment, it is recommended that the subjects use the rest time to rest their eyes and start the next experiment when they are ready.

[0011] Through the above laboratory layout and experimental method, the present invention can accurately detect and analyze the mutual influence between sound and visual stimulation under different experimental conditions, improve the accuracy of experimental results, and provide a powerful experimental method for in-depth study of the interaction between hearing and vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A diagram of a multi-channel spherical speaker system implemented in the present invention.

[0013] Figure 2 A simplified laboratory layout diagram for the implementation of the present invention. DETAILED DESCRIPTION

[0014] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0015] The present invention provides an audio-visual interaction experimental method, the experimental method comprising:

[0016] See also Figure 2 , the laboratory layout of the present invention: the laboratory is an anechoic room, equipped with a multi-channel spherical speaker system, a head rotation tracking device, and LED lights, see Figure 1 The multi-channel spherical speaker system consists of 25 speakers spread across the laboratory. Three yellow LED lights are set on the plane, one at the center azimuth of 0 degrees, and the other two at the horizontal azimuth of + / -45 degrees. The experimental subject's seat is arranged in the center of the room, 3 meters away from the LED light and the speaker. The speaker and LED light are covered with black translucent curtains. The LED light is strong enough to penetrate the curtains. The high-definition camera is installed in front of the seat. The experiment needs to ensure that at the observation point of the experimental subject, the size of the light is 0.42 degrees of the field of view, the brightness is 41cd / m2, and the room is kept dark.

[0017] The specific steps of the experimental method are as follows:

[0018] Step 1, set the visual stimulus, which consists of a single 60-millisecond light flash sequence, that is, the flashing order of the three LED lights is set by the program, and the time interval between each light flash is 150 milliseconds. With the end of the last light flash at the center position, that is, the second to last light flash must be set at the center position, and the test always ends with a light flash on the left or right side, that is, the last light flash must be set to the left or right of the center point.

[0019] Step 2: Set the auditory stimulus, which consists of a 60-millisecond white noise burst sequence. The sound pressure of the noise burst is set to 70 dB at the hearing point of the experimental subject.

[0020] Step 3, light flashing setting, set the flashing times of the flashing sequence to change randomly, with a minimum of 3 times and a maximum of 10 times.

[0021] Step 4: Noise burst setting: the location, number and order of noise bursts can be set by yourself.

[0022] Step 5: Light flashes are combined with noise bursts, and each trial presents a series of light flashes and noise bursts.

[0023] Step 6: Set up a head rotation tracking device to evaluate the head posture. Use a high-definition camera to record the head rotation of the subject during the experiment through OpenCV-based head posture estimation. If the subject is found to have obvious head rotation during the experiment, the experiment will be invalidated, such as horizontal head rotation exceeding 30 degrees or head tilting up and down exceeding 10 degrees.

[0024] Step 7, set up eye tracking, by requiring the experimental subjects to wear glasses-type eye trackers, and record the experimental subjects' gaze direction and gaze time during the experiment. If the experimental subjects' gaze direction is inconsistent with the light flashing position, the experiment is invalid. If the experimental subjects' gaze time is less than 30 milliseconds, the experiment is invalid.

[0025] Step 8, set the experimental rules and inform the subjects of the following rules: Usually, there will be sound when the light flashes, but there may be no sound. The subjects need to try to ignore the sound and only report which side the last flash appeared at the end of the test. The answer time is limited to 1 second. Trials that take more than 1 second to answer and trials with missing answers will be repeated later until 20 repeated results are obtained for each subject in each experimental condition.

[0026] Step 9: After the test, the subject reports the response using the keyboard. Pressing the left arrow indicates that the last light event was perceived on the left side, and pressing the right arrow indicates that it was perceived on the right side. The subject presses the top arrow to start the experiment and initiate a new trial.

[0027] Step 10. During the experiment, it is recommended that the subjects use the rest time to blink and rest their eyes, and start the next experiment when they are ready.

Claims

1. A device and method for measuring visual and auditory interaction, characterized in that: The anechoic room is equipped with a multi-channel spherical speaker system, LED lights, head rotation tracking devices, and eye trackers. The audio-visual interaction measurement method is used to measure the audio-visual interaction in a complex audio-visual environment.

2. The device and method for measuring audiovisual interaction according to claim 1, characterized in that: The multi-channel spherical speaker system consists of X speakers spread across the laboratory, including X1 speakers on the top floor, X2 speakers on the middle floor, and X3 speakers on the bottom floor.

3. The device and method for measuring audiovisual interaction according to claim 1, characterized in that: n (n is an odd number) yellow LED lights are installed on the 0-degree plane in the laboratory, one of which is located at the central azimuth angle of 0 degrees, and the remaining LED lights are grouped in two, and each group of LED lights is set at the horizontal azimuth angle of + / -θ i degrees (select (n-1) / 2 different θ angles according to the number n, i=(n-1) / 2).

4. The apparatus and method for measuring audiovisual interaction according to claim 1, characterized in that: The laboratory is equipped with a head rotation tracking device and a glasses-type eye tracker. The high-definition camera used by the head rotation tracking device is installed slightly in front of the experimental subject. The subjects need to wear the eye tracker during the experiment.

5. The device for measuring visual and auditory interaction according to claims 2 and 3, characterized in that: The speakers and LED lights are controlled by computer programs, which can produce precisely time-controlled and spatially distributed sound and light stimulation to simulate complex audio-visual environments.

6. The device for measuring visual and auditory interaction according to claim 4, characterized in that: The high-definition camera uses OpenCV-based head posture estimation technology to analyze the subjects' head movements during the experiment, including horizontal rotation and tilting up and down. The eye tracker records and analyzes the subjects' eye gaze time and direction during the experiment.