Multi-mode visual and auditory Oddball stimulation method and system

Abstract

The invention discloses a multi-mode visual and auditory Oddball stimulation method and system, and the method comprises the steps: outputting a visual stimulation signal, an auditory stimulation signal and an audio-visual combined synchronous stimulation signal to a testee, and enabling the playing speed of each stimulation signal to be divided into a first stimulation speed and a second stimulation speed, each stimulation signal comprises target stimulation and non-target stimulation; obtaining the occurrence frequency of the target stimulation from the testee, and collecting fMRI data of the testee in the whole process; and analyzing a brain response mechanism of the brain to visual and auditory stimulation at different speeds by using the preprocessed fMRI data.

Description

Technical Field

[0001] The present invention relates to the field of brain research, and in particular to a multimodal audiovisual odddball stimulation method and system. Background Art

[0002] The brain needs to integrate multi-sensory input from noisy backgrounds in a timely manner to form a coherent perceptual representation. The audiovisual integration mechanism is an important process for the brain to process multimodal information. When visual and auditory information arrive at the brain at the same time or with a certain degree of misalignment, the brain needs to integrate this information to form a corresponding perceptual experience.

[0003] In psychological research, audio-visual combined stimulation is generally believed to improve the accuracy or stimulation effect of subjects. However, many audio-visual combined stimulation experiments based on electroencephalogram (EEG) have shown that audio-visual combined stimulation does not necessarily cause more significant brain activation. This phenomenon suggests that there are complex neural mechanisms in the audio-visual integration process, and it is difficult to reveal its full mechanism by relying solely on behavioral data or EEG data.

[0004] The effect of stimulation speed on brain state is also one of the most attractive research directions in neuroscience and psychology. In brain-computer interface (BCI) research, stimulation speed is crucial to achieve highly accurate brain signal decoding. Current research focuses on improving the communication speed and accuracy of BCI systems to achieve rapid signal interpretation and control of external devices. However, in many studies, there is less understanding of the underlying neural mechanisms and high-level cognitive processes of the brain when facing different stimulation speeds; in contrast, psychological research pays more attention to the psychological impact of stimulation speed on cognitive and perceptual processes. In psychological experiments, stimulation speed is considered to be a key factor affecting high-level cognitive functions such as attention, memory, and decision-making. Although some psychological experiments have studied the effect of stimulation speed on brain activity, the understanding of its mechanism at the neural level is still limited. Summary of the invention

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention discloses a multimodal audiovisual odddball stimulation method, comprising: Outputting three types of stimulation signals to the subjects: a visual stimulation signal, an auditory stimulation signal, and a visual-audio combined synchronous stimulation signal, and the playback speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed, and each stimulation signal includes a target stimulation and a non-target stimulation; Obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process; Perform data preprocessing on fMRI data; The preprocessed fMRI data were used to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

[0006] Furthermore, the visual stimulation content is Arabic numerals 1, 2, 3, 4 presented in the center of the screen, and the fonts of the four numerals are all Times New Roman, but the colors are different.

[0007] Furthermore, the auditory stimulation is the standard Mandarin pronunciation of the numbers 1, 2, 3, and 4, and the prompts of 1 and 3 appear in the left channel, and the prompts of 2 and 4 appear in the right channel.

[0008] Furthermore, the audio-visual combined synchronous stimulation includes visual information and auditory information. The visual information content is Arabic numerals 1, 2, 3, 4 presented in the center of the screen. The fonts of the four numbers are all Times New Roman, but the colors are different. The auditory information is the standard Mandarin pronunciation of the numbers 1, 2, 3, 4, and the prompts of 1 and 3 appear in the left channel, and the prompts of 2 and 4 appear in the right channel. The visual information and auditory information are released and retracted synchronously.

[0009] Furthermore, the ratio of the target stimulus to the non-target stimulus is 1:3.

[0010] Furthermore, at the first stimulation speed, in each stimulation signal, the presentation time of each stimulation is 200 ms, and the interval between two stimulations is 50 ms.

[0011] Furthermore, at the second stimulation speed, in each stimulation signal, the presentation time of each stimulation is 200 ms, and the interval between two stimulations is 1800 ms.

[0012] Furthermore, the parameters for collecting the fMRI data of the subject are set as follows: FOV = 240 × 240 mm 2 The MRI intensity was 3 Tesla, the scanning sequence was a gradient echo sequence, and the voxel size was 3 × 3 × 3 mm 3 , repetition time TR=2000ms, echo time TE=30ms, FA=85 degrees.

[0013] On the other hand, the present invention also provides a multimodal audiovisual oddball stimulation system, comprising: A signal generation module, used for outputting three kinds of stimulation signals, namely, a visual stimulation signal, an auditory stimulation signal and a visual and audio combined synchronous stimulation signal, to the subject, wherein the playing speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed, and each stimulation signal includes a target stimulation and a non-target stimulation; The data acquisition module is used to obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process; The data processing module is used to preprocess the fMRI data; the preprocessed fMRI data is used to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

[0014] By adopting the above-mentioned technical scheme and introducing visual, auditory and combined audio-visual stimulation methods, it is possible to more comprehensively study the brain's processing mechanism for different types of stimuli; and by setting fast and slow stimulation conditions, it is possible to explore the differences in the brain's response to stimuli at different time intervals, and analyze the differences in the brain areas designed when the brain faces different stimulus speed information; in addition, collecting fMRI data during the experiment can provide high spatial resolution brain functional imaging data, which makes up for the shortcoming of insufficient spatial resolution of EEG data, helps to reveal the neural mechanism of the brain when processing target stimuli, and can be applied to multiple research fields such as cognitive neuroscience, psychology, and brain science, providing new experimental methods and data support for research in related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which: Figure 1 Flowchart of a multimodal audiovisual oddball stimulation method. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0017] Reference Figure 1 The embodiment of the present invention provides a multimodal audiovisual odddball stimulation method, comprising the following steps: S1: Output three types of stimulation signals, namely, visual stimulation signal, auditory stimulation signal and audio-visual combined synchronous stimulation signal, to the subjects. The playback speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed. Each stimulation signal includes a target stimulation and a non-target stimulation.

[0018] Specifically, the visual stimulation content is Arabic numerals 1, 2, 3, and 4 presented in the center of the screen. The fonts of the four numerals are all Times New Roman, but the colors are different.

[0019] The auditory stimulation is the standard Mandarin pronunciation of the numbers 1, 2, 3, and 4, and the prompts of 1 and 3 appear in the left channel, and the prompts of 2 and 4 appear in the right channel.

[0020] In audio-visual combined stimulation, visual information and auditory information are released and retracted synchronously. The stimulation parameters are as follows: the stimulus presentation time is 200ms, the interval between two stimuli is 50ms at the first stimulation speed, and the interval between two stimuli is 1800ms at the second stimulation speed; the stimulation is in a block of 1, 2, 3, and 4, and the 4 numbers appear randomly in this block, and the ratio of target to non-target numbers is 1:3. A complete experiment lasts for 3 minutes in both experimental conditions.

[0021] S2: Obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process.

[0022] As mentioned above, before the experiment begins, the subjects are informed in advance of the target number specified in the next experiment. During the experiment, the subjects are required to remember the number of times the target stimulus appears, and the fMRI data of the subjects are collected during this process.

[0023] The nuclear magnetic resonance device is a PHILIPS Ingenia 3.0T CX acquisition device. Before the formal experiment begins, a set of structural data will be collected from the subject, FOV = 256 × 256mm 2 The intensity of magnetic resonance imaging was 3 Tesla, the scanning sequence was Gradient Recalled Echo (GRE), and the voxel size was 1×1×1 mm. 3 , repetition time TR = 6.4ms, echo time TE = 2.9ms, FA = 8 degrees; functional data of the subjects were collected during the experiment, FOV = 240 × 240mm 2 The intensity of magnetic resonance imaging was 3 Tesla, the scanning sequence was Gradient Recalled Echo (GRE), and the voxel size was 3×3×3 mm. 3 , repetition time TR=2000ms, echo time TE=30ms, FA=85 degrees.

[0024] S3: Perform data preprocessing on fMRI data.

[0025] fMRI data preprocessing included slice time correction, head motion correction, spatial normalization and spatial smoothing. The preprocessing process was completed using the Matlab toolbox DPABI.

[0026] S4: Use preprocessed fMRI data to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

[0027] After the fMRI data were preprocessed, activation state analysis was performed using the generalized linear model method to determine the corresponding activation area of ​​the brain under each stimulation method, analyze the functional connections of the brain under different stimulation methods, construct a brain network, and analyze the characteristics of the brain network using graph theory. The specific network characteristics are as follows: Small-worldness: Comparing the clustering and short path length characteristics of brain networks under different stimulation conditions, reflecting the brain's information processing ability.

[0028] Node efficiency: Evaluate the information transmission efficiency of each node in the brain network under different stimulation conditions to understand whether audio-visual integration improves the efficiency of certain key nodes.

[0029] Node centrality: Analyze the importance and influence of each node in the network. Compare the differences in node centrality between audiovisual integration and unimodal stimulation or between fast and slow stimulation.

[0030] Associativity: Assess the connection tendency of nodes in the network and observe whether audiovisual integration or stimulation speed affects the degree of connection between nodes with high similarity in the network.

[0031] The embodiment of the present invention provides a multimodal audiovisual oddball stimulation method, which can more comprehensively study the brain's processing mechanism for different types of stimuli by introducing visual, auditory and audiovisual combined stimulation modes; and by setting two stimulation conditions, fast and slow, it is possible to explore the differences in the brain's responses to stimuli at different time intervals, and analyze the differences in the brain regions designed when the brain faces different stimulation speed information; in addition, fMRI data is collected during the experiment, which can provide brain functional imaging data with high spatial resolution, making up for the shortcoming of insufficient spatial resolution of EEG data, and helping to reveal the neural mechanism of the brain when processing target stimuli. It can be applied to multiple research fields such as cognitive neuroscience, psychology, and brain science, and provide new experimental methods and data support for research in related fields.

[0032] An embodiment of the present invention further provides a multimodal audiovisual oddball stimulation system, comprising: A signal generation module, used for outputting three kinds of stimulation signals, namely, a visual stimulation signal, an auditory stimulation signal and a visual and audio combined synchronous stimulation signal, to the subject, wherein the playing speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed, and each stimulation signal includes a target stimulation and a non-target stimulation; The data acquisition module is used to obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process; The data processing module is used to preprocess the fMRI data; the preprocessed fMRI data is used to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

[0033] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0034] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0035] It can be known from the description of the above implementation modes that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute the methods described in the various implementation modes of the present application or certain parts of the implementation modes.

[0036] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal audiovisual oddball stimulation method, characterized in that: include: Outputting three types of stimulation signals, namely, visual stimulation signals, auditory stimulation signals, and visual and audio combined synchronous stimulation signals, to the subjects respectively, and the playback speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed, and each stimulation signal includes a target stimulation and a non-target stimulation; Obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process; Perform data preprocessing on fMRI data; The preprocessed fMRI data were used to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

2. The method according to claim 1, characterized in that The visual stimulation content is Arabic numerals 1, 2, 3, 4 presented in the center of the screen. The fonts of the four numerals are all Times New Roman, but the colors are different.

3. The method according to claim 1, characterized in that The auditory stimulation is the standard Mandarin pronunciation of the numbers 1, 2, 3, and 4, and the prompts of 1 and 3 appear in the left channel, and the prompts of 2 and 4 appear in the right channel.

4. The method according to claim 1, characterized in that: The audiovisual combined synchronous stimulation includes visual information and auditory information. The visual information content is Arabic numerals 1, 2, 3, and 4 presented in the center of the screen. The four numerals are all in Times New Roman fonts, but in different colors. The auditory information is the standard Mandarin pronunciation of the numerals 1, 2, 3, and 4, and the prompts of 1 and 3 appear in the left channel, and the prompts of 2 and 4 appear in the right channel. The visual information and the auditory information are released and retracted synchronously.

5. The method according to claim 1, characterized in that The ratio of the target stimulus to the non-target stimulus is 1:

3.

6. The method according to claim 1, characterized in that At the first stimulation speed, in each stimulation signal, the presentation time of each stimulation is 200 ms, and the interval between two stimulations is 50 ms.

7. The method according to claim 1, characterized in that At the second stimulation speed, in each stimulation signal, the presentation time of each stimulation is 200 ms, and the interval between two stimulations is 1800 ms.

8. The method according to claim 1, characterized in that The parameters for collecting the fMRI data of the subjects are set as follows: FOV = 240 × 240 mm 2 The MRI intensity was 3 Tesla, the scanning sequence was a gradient echo sequence, and the voxel size was 3 × 3 × 3 mm 3 , repetition time TR=2000ms, echo time TE=30ms, FA=85 degrees.

9. A multimodal audiovisual oddball stimulation system, characterized in that: include: A signal generation module, used for outputting three kinds of stimulation signals, namely, a visual stimulation signal, an auditory stimulation signal and a visual and audio combined synchronous stimulation signal, to the subject, wherein the playing speed of each stimulation signal is divided into a first stimulation speed and a second stimulation speed, and each stimulation signal includes a target stimulation and a non-target stimulation; The data acquisition module is used to obtain the number of times the target stimulus appears from the subject and collect the fMRI data of the subject during the whole process; The data processing module is used to preprocess the fMRI data; the preprocessed fMRI data is used to analyze the brain's response mechanism to visual and auditory stimuli of different speeds.

Images