A method for detecting interpersonal neural synchrony under audiovisual stimulation and a system thereof

Abstract

The application discloses a kind of methods for detecting interpersonal neural synchronization under audiovisual stimulation, comprising the following steps: designing double-person ultra-scan experimental paradigm, and collecting double-person electroencephalogram signal, construct a database of visual stimulation and auditory stimulation;The data in the database are preprocessed to obtain four different frequency bands δ, θ, α, β of electroencephalogram signal, the electroencephalogram signal of alpha band is extracted for segmentation, and the inter-subject correlation ISCV of each segment is calculated;The intracerebral network and interbrain network are constructed by constructing functional connection matrix, and the similarity of intracerebral network and the global efficiency of interbrain network are calculated, so as to effectively evaluate the cooperation and synchronization degree between neural activities between subjects under visual stimulation and auditory stimulation.The application also discloses a system for detecting interpersonal neural synchronization under audiovisual stimulation.The application studies the influence of single sensory stimulation on intracerebral network activation mode by standardizing experimental conditions to reduce subject differences.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-person brain-computer interface and electroencephalogram hyperscanning, and particularly relates to a method for detecting interpersonal neural synchronization under audio-visual stimulation and a system thereof. BACKGROUND

[0002] In the fast-paced modern life, the quality of interpersonal interaction deeply affects mental health and life experience. Understanding the neural mechanisms behind it, especially interpersonal neural synchronization (the synchronization of brain activities of different individuals in social interaction), is crucial for analyzing social behavior. Traditional neuroscience research mainly focuses on single brain activity, while hyperscanning technology (such as EEG, fMRI, fNIRS, MEG) reveals the neural synchronization mechanism in interaction by synchronously recording the brain activities of multiple people. Among them, EEG hyperscanning has become the mainstream method due to its high temporal resolution and low cost.

[0003] Research has found that interpersonal neural synchronization widely exists in various social scenarios, such as action coordination, music listening, decision-making cooperation, and face-to-face communication. This synchronization relies on the integration of multi-sensory (visual, auditory) information and involves high-order brain regions (such as the cortex of the cingulate gyrus), not just the sensory cortex. In addition, alpha band activity is related to the resting state of the brain and attention regulation, and the synchronization of alpha waves in EEG may produce significant oscillation changes and synchronization in cooperative tasks. Alpha waves correspond to the resting state of the brain, and sensory stimuli usually suppress these oscillations in the corresponding sensory areas of the brain. However, the alpha band is related to various cognitive processes, especially attention processes and social interaction mechanisms.

[0004] Existing research on interpersonal neural synchronization has made many results, but still has limitations:

[0005] (1) Existing research is mostly based on natural interaction with multi-sensory integration (such as watching movies, playing musical instruments), making it difficult to separate the effects of single sensory channels;

[0006] (2) Although existing research only examines the effects of specific sensory channels on interpersonal neural synchronization from visual or auditory tasks, there are often differences in subject populations (such as age, cognitive ability, experimental conditions, etc.) between different studies, making it difficult to directly compare research results.

[0007] Previous research has mostly compared inter-brain networks and cross-subject synchronization between brain regions, ignoring the synchronization of sensory stimulation on the activation pattern of the subject's brain network. Therefore, there is an urgent need to provide a new method for detecting interpersonal neural synchronization under audio-visual stimulation and a system thereof to solve the above problems. SUMMARY

[0008] The technical problem solved by the present application is to provide a method and system for detecting interpersonal neural synchronization under audio-visual stimulation, which reduces differences between subjects by standardizing experimental conditions and studies the influence of sensory stimulation on network activation patterns in the brain.

[0009] To solve the above technical problems, the present application adopts one technical solution: providing a method for detecting interpersonal neural synchronization under audio-visual stimulation, comprising the following steps:

[0010] S1: designing a two-person hyperscanning experimental paradigm and collecting two-person electroencephalogram signals to construct a database of visual and auditory stimuli;

[0011] S2: preprocessing the data in the database to obtain four different frequency bands of electroencephalogram signals, namely delta, theta, alpha, and beta, extracting the alpha band electroencephalogram signals for segmentation to extract the electroencephalogram signals under visual and auditory tasks, dividing the electroencephalogram signals into several segments according to the time of the stimulus material, calculating the inter-subject correlation ISC value of each segment, analyzing the trend and stability of the correlation between subjects over time; calculating the ISC value of the entire stimulation period for each task by the electroencephalogram signals of the two subjects to evaluate the similarity of brain activity between subjects under the same task or stimulation;

[0012] S3: constructing intracerebral and intercerebral networks by constructing a functional connectivity matrix, and calculating the similarity of intracerebral networks and the global efficiency of intercerebral networks, thereby effectively evaluating the cooperation and synchronization degree between neural activities of subjects under visual and auditory stimulation.

[0013] In a preferred embodiment of the present application, in step S1, the design method of the two-person hyperscanning experimental paradigm comprises:

[0014] The experiment includes two experimental tasks: visual task and auditory task, in which two types of stimulus materials, pictures and videos, are used in the visual task, and two types of stimulus materials, sentences and music, are used in the auditory task, and each type of stimulus material is further divided into different types of emotional valence, i.e. positive, neutral and negative, a total of 12 different conditions are generated;

[0015] The experiment is conducted in three rounds, and the stimulus materials used in each round cover pictures, videos, sentences and music, and the emotional valence of the stimulus materials is different in each round, in each experimental round, the subjects receive electroencephalogram recording in four natural stimulation sessions, i.e. watching pictures, listening to sentences with eyes closed, watching videos and listening to music, the video is a silent movie clip, and the sentence is a non-native language sentence.

[0016] In a preferred embodiment of the present application, in step S2, the step of preprocessing the data in the database comprises:

[0017] S201: Previewing the electroencephalogram signal, and removing data with obvious drift;

[0018] S202: Removing 50Hz line noise by using notch filter technology, and then performing band-pass filtering between 0.1Hz and 45Hz to remove noise with too high or too low frequency;

[0019] S203: Changing the original reference electrode FCz to average reference to optimize the signal reference mode;

[0020] S204: Decomposing the electroencephalogram signal into several independent components by independent component analysis (ICA), and combining visual inspection to accurately identify and remove artifacts, so as to effectively eliminate noise components.

[0021] In a preferred embodiment of the present application, in step S2, the specific steps of calculating the inter-subject correlation (ISC) value of each segment include:

[0022] First, for each pair of subjects k and l, the cross-electrode covariance matrix C kl is calculated as follows:

[0023]

[0024] Wherein, X k (t) and X l (t) represent the electroencephalogram signals of individuals k and l at time point t, respectively, μ k and μ l are the average values of individuals k and l over the entire time series, respectively, and T represents the total number of time points of the signal.

[0025] Then, the inter-subject correlation (ISC) value between each pair of subjects is calculated by using the following formula:

[0026]

[0027] Wherein, V i represents the i-th feature vector extracted by PCA or other methods, which is used to capture the key features of the signal; T is the total number of time points of the signal.

[0028] In a preferred embodiment of the present application, in step S3, the specific steps of constructing the intracerebral network and the intercerebral network by constructing the functional connectivity matrix are as follows:

[0029] First, the circular correlation coefficient CCorr is selected as the synchronization index:

[0030]

[0031] Wherein, and respectively represent the phase values of two channels being compared and the average phase of two signals, k represents the kth data point, and N represents the total number of data points, and represent the phase values of the kth data points of two channels;

[0032] Then, the two subjects use the cap equipped with 32 electrodes, combine the electroencephalogram signals of the two subjects into a 64-electrode two-dimensional array, and construct a functional connection matrix using correlation, wherein the nodes in the matrix represent network nodes, and the values represent the weights of the connections between the nodes; the first 32 electrodes are the electrodes in the brain of the first subject, the last 32 electrodes are the electrodes in the brain of the second subject, the CCorr between the electrodes in the brain of the first subject is calculated to obtain the intracerebral network of the first subject, the CCorr between the electrodes in the brain of the second subject is calculated to obtain the intracerebral network of the second subject, and the CCorr between the electrodes in the brain of the first subject and the electrodes in the brain of the second subject is calculated to obtain the interbrain network.

[0033] In a preferred embodiment of the present application, in step S3, the similarity of the intracerebral network is calculated using the following formula:

[0034]

[0035] wherein C 1,k,t represents the edge weight of the first node pair (k, t) in the functional connection matrix, C 2,k,t represents the edge weight of the second node pair (k, t) in the functional connection matrix, and Ne is the number of nodes; SimiNet ∈ [0, 1], 0 and 1 respectively represent that the two connection matrices are not similar and completely identical.

[0036] In a preferred embodiment of the present application, in step S3, the global efficiency of the interbrain network is calculated using the following formula:

[0037]

[0038] wherein n is the total number of nodes in the network, N is the set of nodes, d ij is the shortest path length between node i and node j.

[0039] In a preferred embodiment of the present application, in step S3, under the same stimulation, the higher the similarity of the intracerebral network structure of the two subjects, the stronger the synchronization between the multiple brains; the global efficiency of the cross-brain network is used to measure the efficiency and stability of information transmission between the brain networks of the subjects, and the higher the global efficiency, the stronger the integration ability of the interbrain network, the higher the coordination ability of different electrodes or brain areas, and thus the higher the synchronization.

[0040] To solve the above technical problems, the application adopts another technical solution: to provide a system for detecting interpersonal neural synchronization under audio-visual stimulation, comprising:

[0041] An audio-visual stimulation database construction module is used to design a two-person hyperscanning experiment paradigm and collect two-person electroencephalogram signals to construct an audio-visual stimulation database;

[0042] An electroencephalogram data preprocessing module is used to preprocess the data in the database to obtain electroencephalogram signals in four different frequency bands, i.e., delta, theta, alpha and beta, and extract the electroencephalogram signals in the alpha band for segmentation to extract the electroencephalogram signals under visual and auditory tasks;

[0043] A subject intercorrelation calculation module is used to divide the electroencephalogram signals preprocessed by the electroencephalogram data preprocessing module into several segments according to the time of the stimulation material, calculate the subject intercorrelation ISC value of each segment, analyze the trend and stability of the subject correlation with time, and calculate the ISC value of the entire stimulation through the electroencephalogram signals of the two subjects during each task to evaluate the similarity of the brain activities of the subjects under the same task or stimulation;

[0044] A brain network construction and evaluation module is used to construct intracerebral networks and intercerebral networks through the construction of a functional connection matrix and calculate the similarity of the intracerebral networks and the global efficiency of the intercerebral networks to effectively evaluate the cooperation and synchronization degree between the neural activities of the subjects under visual and auditory stimulation.

[0045] In a preferred embodiment of the application, the step of preprocessing the data in the database by the electroencephalogram data preprocessing module comprises:

[0046] First, the electroencephalogram signals are previewed to remove data with obvious drift;

[0047] Second, notch filter technology is used to remove 50Hz line noise, and then band-pass filtering is performed between 0.1Hz and 45Hz to remove noise with excessively high or low frequency;

[0048] Then, the original reference electrode FCz is changed to an average reference to optimize the signal reference mode;

[0049] Finally, the electroencephalogram signals are decomposed into several independent components through independent component analysis (ICA), and the artifacts are accurately identified and removed through visual inspection, so that the noise components are effectively removed.

[0050] The beneficial effects of this invention are as follows: This invention focuses on the differences in the roles of visual and auditory perception in interpersonal neural synchronization under the alpha band, standardizing experimental conditions to reduce subject variability, and studying the impact of sensory stimulation on brain network activation patterns. Understanding the effects of different sensory stimuli on interpersonal neural synchronization can help design more natural and intuitive interactive interfaces, improve interaction efficiency and experience, contribute to a deeper understanding of the impact of interpersonal neural synchronization on social interaction and mental health, and provide a scientific basis for improving interpersonal interaction. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the method for detecting interpersonal neural synchronization under audiovisual stimuli according to the present invention.

[0052] Figure 2 This is a schematic diagram of the design of the dual-person hyperscanning experimental paradigm;

[0053] Figure 3 This is a schematic diagram of the functional connectivity matrix within and between the brains of the test subject.

[0054] Figure 4 This is a diagram illustrating the correlations among subjects under different conditions;

[0055] Figure 5 This is a schematic diagram showing the similarity of brain networks in subjects under different conditions;

[0056] Figure 6 This is a schematic diagram of the global efficiency of the interbrain network of subjects under different conditions;

[0057] Figure 7 This is a structural block diagram of the system for detecting interpersonal neural synchronization under visual and auditory stimuli. Detailed Implementation

[0058] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0059] Please see Figure 1 The embodiments of the present invention include:

[0060] A method for detecting interpersonal neural synchronization under visual and auditory stimuli includes the following steps:

[0061] S1: Design a dual-person hyperscanning experimental paradigm and collect EEG signals from both individuals to construct a database of visual and auditory stimuli.

[0062] The main purpose of the present experiment is to investigate the influence of different sensory stimuli on interpersonal neural synchrony. Two experimental tasks are designed in the present experiment: visual task and auditory task. In each task, two types of stimulus materials are used: pictures and videos in the visual task, and sentences and music in the auditory task. In addition, each type of stimulus material is further divided into different types of emotional valence, i.e. positive, neutral and negative, resulting in a total of 12 (2x2x3) different conditions. The experiment is conducted in three rounds, and the stimulus materials used in each round cover pictures, videos, sentences and music. However, the emotional valence of these materials is different in each round. Figure 2 The specific procedure of the experiment is shown. In order to balance the potential influence of audio-visual materials and emotional valence on the experimental results, the order of the materials is changed after five groups of experiments are conducted.

[0063] After entering the recording room, the subjects are required to close their eyes, relax and sit quietly for 3 minutes to collect the resting-state electroencephalogram as the baseline activity. Then, the subjects enter the task-induced phase. In each round of experiment, the subjects receive induced electroencephalogram recording in four natural stimulus sessions, i.e. watching pictures, listening to sentences, watching videos and listening to music. In the picture watching module, the subjects need to watch 20 pictures in 1 minute, each picture is played for 3 seconds, and then automatically jumps to the next picture. After a 30-second rest, the picture watching module is switched to the music listening module, at this time the subjects close their eyes and listen to 20 German sentences in 1 minute; each sentence is played for 3 seconds, and then the next sentence is automatically played. After a 30-second rest, the movie clip watching module is switched, at this time they are required to watch a 1-minute long silent movie clip. After a 30-second rest, they listen to 1 minute of pure German music. It is worth noting that the reason for choosing German sentences and music as experimental materials is to control the interference of semantic content and individual cognition on the measurement of emotional dimensions. All subjects are Chinese native speakers and completely ignorant of German, which can ensure that any emotional response is mainly caused by sound and valence rather than language content. During the visual stimulation, the subjects are required to focus on the computer screen. During the auditory stimulation, the subjects are required to close their eyes and quietly listen to the speech or music. During the whole experiment, the subjects are required to keep quiet and try not to have body movements, and there is no eye contact, body contact or any other interaction between the subjects.

[0064] In the above experimental paradigm, the rest time can be set at will.

[0065] S2: preprocessing the data in the database to obtain the electroencephalogram signals of four different frequency bands δ, θ, α, β, segmenting the electroencephalogram signals of the alpha band to extract the electroencephalogram signals under visual and auditory tasks, dividing the electroencephalogram signals into several segments according to the time of the stimulation material, calculating the intersubject correlation (ISC) value of each segment, analyzing the trend and stability of the correlation between subjects over time; calculating the ISC value of the entire stimulation by the electroencephalogram signals of two subjects during each task, to evaluate the similarity of brain activity between subjects under the same task or stimulation;

[0066] Figure 1 Part A of the middle part represents the data acquisition and preprocessing. In the experiment, the electroencephalogram (EEG) signal is susceptible to various interferences (such as electromyogram, electrooculogram, electrode failure, 50Hz power frequency, etc.), resulting in the appearance of artifacts and affecting the accuracy of data analysis. Therefore, it is crucial to preprocess the EEG signal to remove artifacts, and the specific steps are as follows: In the process of EEG signal preprocessing, first, preview the signal to remove data with obvious drift; then, use notch filter technology to remove 50Hz line noise, and then perform band-pass filtering between 0.1Hz and 45Hz to remove noise at too high or too low frequencies. Then, change the original reference electrode FCz to average reference to optimize the signal reference method. Through independent component analysis (ICA), the electroencephalogram signal is decomposed into multiple independent components, and combined with visual inspection to accurately identify and remove artifacts, thereby effectively eliminating noise components.

[0067] This example uses the ISC method to evaluate the synchronization of brain activity of subjects under the same sensory stimulation, which is particularly suitable for naturalistic stimulation paradigm, and can measure the consistency of neural responses between different individuals. First, for each pair of subjects k and l, calculate the cross-electrode covariance matrix C kl , which is:

[0068]

[0069] Here, X k (t) and X l (t) represent the electroencephalogram signals of individuals k and l at time point t, respectively. μ k and μ l are the average values of individuals k and l over the entire time series, which are usually obtained by calculating the average value of each individual signal at all time points. T represents the total number of time points of the signal. For each time point t, calculate the signals X k (t) and X l (t) of individuals k and l and their respective average values μ k and μ lthe two deviations are summed to measure the degree of synchronization of the two individual signals at each time point. Then, the products of all time points are averaged to obtain the covariance C kl Finally, the following formula is applied to calculate the inter-subject correlation (ISC) value between each pair of subjects:

[0070]

[0071] Here, v i represents the i-th eigenvector extracted by PCA or other methods, which is used to capture the key features of the signal. T is the total number of time points of the signal. For each pair of time points t, the dot product X k (t)*v i and X l (t)*v i of the individual signals and the eigenvector are calculated, and the square of their sum (numerator and denominator) is summed to obtain the weighted signal intensity. The numerator is the sum of the product of the two individual weighted signals, reflecting the synchronization of the signals; the denominator is the product of the weighted sum of squares, which is used for normalization. The ISC value is the result of the calculation of these two ratios, indicating the degree of time synchronization of the two individuals under the specific eigenvector v i The present invention deeply studies the ISC between subjects in the alpha band, divides the EEG signal into ten segments at the scalp level, each segment being 6s, calculates the ISC value of each segment, and analyzes the trend and stability of the correlation between subjects over time.

[0072] By calculating the ISC of the two subjects during each task (visual task and auditory task), the similarity of brain activity between the subjects under the same task or stimulus is evaluated.

[0073] S3: Constructing the intracerebral network and intercerebral network by constructing the functional connection matrix, and calculating the similarity of the intracerebral network and the global efficiency of the intercerebral network, thereby effectively evaluating the cooperation and synchronization degree of multiple brains under visual stimulation and auditory stimulation.

[0074] In this example, CCorr (circular correlation coefficient) is selected as the synchronization indicator:

[0075]

[0076] CCorr measures the circular covariance of the difference between the observed phase and the expected phase or the average phase. Among them, and respectively represent the phase values of the two channels being compared and the average phase of the two signals, k represents the kth data point, and N represents the total number of data points, and respectively represent the phase values of the kth data points of the two channels. Essentially, CCorr is the covariance of the phase variance, i.e., it indicates whether the phases of the two oscillators change in a similar manner. Positive values (0 < CCorr ≤ 1): indicate that there is positive synchronization between the signals, and the closer the value is to 1, the stronger the positive linear correlation. Negative values (-1≤ CCorr < 0): indicate that there is negative synchronization between the signals, and the closer the value is to -1, the stronger the negative linear correlation. Zero values (CCorr≈ 0): indicate that there is no linear synchronization between the signals.

[0077] Specifically, the collected electroencephalogram is filtered to obtain four different frequency bands (δ, θ, α, β), and CCorr is calculated according to the alpha band and the formula introduced above. Since a cap equipped with 32 electrodes is used, the electroencephalogram signals of the two subjects are merged into a 64-electrode two-dimensional array (the vertical axis is the electrode, and the horizontal axis is the sampling point). A functional connection matrix is constructed using CCorr, where the nodes in the matrix represent network nodes, and the values represent the weights of the connections between the nodes. By calculating the correlation between the electrodes, a 64*64 symmetric matrix is generated, with the first 32 electrodes being the electrodes in the brain of the first subject and the last 32 electrodes being the electrodes in the brain of the second subject. The CCorr between the electrodes in the brain of the first subject is calculated to obtain the intracerebral network of the first subject, the CCorr between the electrodes in the brain of the second subject is calculated to obtain the intracerebral network of the second subject, and the CCorr between the electrodes in the brain of the first subject and the electrodes in the brain of the second subject is calculated to obtain the interbrain network. For example, Figure 3 The functional connections between the electrodes in the brain of the subject and across subjects are shown. Figure 3 a in Figure 3 d in Figure 3 b in Figure 3 c in Figure 3 b in Figure 3 c in

[0078] SimiNet is a method for quantifying the similarity between brain networks, which not only considers the edge weights of the network but also takes into account the differences in the spatial positions of the network nodes and weighs the corresponding costs of their differences.

[0079]

[0080] wherein C 1,k,t denotes the edge weight of the first node pair (k, t) in the functional connectivity matrix, C 2,k,t denotes the edge weight of the second node pair (k, t) in the functional connectivity matrix, Ne is the number of nodes; SimiNet is in [0, 1], 0 and 1 respectively represent that the two connection matrices are dissimilar and completely identical.

[0081] The cross-brain functional connectivity can effectively evaluate the coordination and synchronization degree between multiple brains. The global efficiency efficiency quantifies the overall efficiency of information transmission in the entire network, and the average inverse shortest path length is called global efficiency. The higher the global efficiency of the cross-brain network means that information can be transmitted and integrated in parallel between the brain networks more quickly and effectively. In the context of social interaction, this means that the brains of two people can better coordinate and synchronize when processing information, thereby promoting more efficient communication and emotional exchange. By calculating the global efficiency of the cross-brain network, the efficiency and stability of information transmission between the brain networks of the subjects are measured.

[0082]

[0083] wherein n is the total number of nodes in the network, and N is the set of nodes. d ij is the shortest path length between node i and node j.

[0084] Figure 1 B in the above formula represents calculating the correlation between subjects, including the ISC change and the ISC average by dividing the data into ten segments. Figure 1 C in the above formula represents constructing the intracerebral network and the interbrain network, and calculating the similarity of the intracerebral network and the global efficiency of the interbrain network. Under the same stimulus, the higher the similarity of the intracerebral network structure of the two subjects, the stronger the synchronization; the global efficiency can reflect the transmission efficiency of information in the network, and the higher the global efficiency, the stronger the integration ability of the interbrain network, the higher the coordination ability of different electrodes or brain regions, and thus the higher the synchronization; the results show that both visual and auditory stimuli can significantly enhance the synchronization between subjects, and the effect is higher under visual stimulation.

[0085] The effects of the method described in the present application will be illustrated below in combination with experimental results:

[0086] Figure 4are the experimental results of inter-subject correlation (ISC) under different conditions, which shows that both visual and auditory stimuli significantly increase ISC, and the ISC caused by visual stimuli is significantly higher than that caused by auditory stimuli (p<0.001). Further analysis of the mean square error of these time periods ISC finds that the mean square error of visual ISC is significantly lower than that of auditory ISC. This indicates that in the alpha band, visual stimuli not only induce stronger neural synchronization, but also show greater temporal stability and continuity, with relatively small fluctuations. In terms of average ISC values, the ISCs produced by visual and auditory stimuli are significantly higher than those in the resting state, and there is also a significant difference between the ISC values caused by visual and auditory stimuli (p<0.001).

[0087] Figure 5 The results of inter-subject correlation (ISC) under different task conditions based on valence are shown. Under visual stimulation, the SimiNet similarity of brain networks between the two subjects is significantly higher than that under auditory stimulation (p<0.01), and both are significantly higher than that in the resting state (p<0.01 between the resting state and the visual stimulation condition, and p<0.05 between the resting state and the auditory stimulation condition). These results suggest that visual stimulation significantly enhances the structural similarity of brain networks between subjects, which may be closely related to the enhancement of functional synchronization between brain regions involved in visual processing. Therefore, it can be inferred that under visual stimulation, the brain network structure of the subjects shows higher similarity. In addition, visual and auditory stimulation significantly enhances the synchronization between brain regions of the subjects, providing important insights into the influence of different sensory modalities on the function of brain networks.

[0088] Figure 6 The results of network properties of inter-brain networks of subjects under different conditions are shown. It is found that the global efficiency of inter-brain networks under visual stimulation is significantly greater than that under auditory stimulation, and both are greater than that in the resting state. This finding is verified by independent sample t-test (p<0.05), indicating that there is a significant difference in the global efficiency of inter-brain networks between subjects under different sensory stimuli.

[0089] Referring to Figure 7 In the examples of the present application, a system for detecting interpersonal neural synchronization under audio-visual stimulation is also provided, which includes an audio-visual stimulation database construction module, an electroencephalogram data preprocessing module, an inter-subject correlation calculation module, and a brain network construction and evaluation module.

[0090] The audio-visual stimulation database construction module is used to design a two-person hyperscan experiment paradigm and collect two-person electroencephalogram signals to construct an audio-visual stimulation database;

[0091] The electroencephalogram data preprocessing module is configured to preprocess data in the database to obtain electroencephalogram signals in four different frequency bands, namely, delta, theta, alpha and beta, and segment the electroencephalogram signals in the alpha band to extract electroencephalogram signals under visual and auditory tasks.

[0092] The inter-subject correlation calculation module is configured to divide the electroencephalogram signals preprocessed by the electroencephalogram data preprocessing module into several segments according to the time of the stimulation material, calculate an inter-subject correlation (ISC) value of each segment, analyze the trend and stability of the inter-subject correlation over time, and calculate the ISC value of the entire stimulation process by calculating the electroencephalogram signals of two subjects during each task, to evaluate the similarity of brain activities of the subjects under the same task or stimulation.

[0093] The brain network construction and evaluation module is configured to construct intra-brain networks and inter-brain networks by constructing a functional connection matrix, and calculate the similarity of the intra-brain networks and the global efficiency of the inter-brain networks, to effectively evaluate the degree of cooperation and synchronization between neural activities of the subjects under visual stimulation and auditory stimulation.

[0094] The present example is a system for detecting interpersonal neural synchronization under audio-visual stimulation, which can perform the method for detecting interpersonal neural synchronization under audio-visual stimulation provided by the present application, and can perform any combination of the steps of the method example, and has the corresponding functions and beneficial effects of the method.

[0095] The above description is only an example of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation based on the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A method of detecting interpersonal neural synchrony under audiovisual stimulation, characterized in that, The method comprises the following steps: S1: design a two-person hyper-scanning experiment paradigm, collect two-person electroencephalogram signals, and construct a database of visual and auditory stimuli; S2: pre-process the data in the database to obtain electroencephalogram signals in four different frequency bands δ, θ, α, and β, segment the electroencephalogram signals in the alpha band to extract electroencephalogram signals under visual and auditory tasks, divide the electroencephalogram signals into several segments according to the time of the stimulus material, calculate the inter-subject correlation ISC value of each segment, analyze the trend and stability of the inter-subject correlation with time, and calculate the ISC value of the entire stimulus to evaluate the similarity of brain activities between the two subjects under the same task or stimulus; S3: construct intracerebral and intercerebral networks by constructing a functional connection matrix, and calculate the similarity of the intracerebral network and the global efficiency of the intercerebral network, thereby effectively evaluating the cooperation and synchronization between neural activities between subjects under visual and auditory stimuli; the specific steps of constructing the intracerebral and intercerebral networks by constructing a functional connection matrix are as follows: First, the circular correlation coefficient CCorr is selected as the synchronization indicator: , wherein, and denotes the phase value of the electroencephalogram signal on the two channels being compared, and denotes the average value of the phase of the signal itself over time on the two channels, k denotes the kth time point, and N denotes the total number of time points, and denotes the phase value of the two channels at the kth time point; Then, the two subjects use a cap equipped with 32 electrodes, the electroencephalogram signals of the two subjects are combined into a 64-electrode two-dimensional array, and a functional connection matrix is constructed using correlation. The nodes in the matrix represent network nodes, and the numerical values represent the weights of the connections between the nodes; the first 32 electrodes are the intracerebral electrodes of the first subject, and the last 32 electrodes are the intracerebral electrodes of the second subject. The CCorr between the first subject's intracerebral electrodes is calculated to obtain the first subject's intracerebral network, the CCorr between the second subject's intracerebral electrodes is calculated to obtain the second subject's intracerebral network, and the CCorr between the first subject's intracerebral electrodes and the second subject's intracerebral electrodes is calculated to obtain the intercerebral network.

2. The method of detecting interpersonal neural synchronization to audiovisual stimuli of claim 1, wherein, In step S1, the design method of the two-person hyper-scanning experiment paradigm comprises: The experiment includes two experimental tasks: visual task and auditory task. In the visual task, two types of stimulus materials, pictures and videos, are used. In the auditory task, two types of stimulus materials, sentences and music, are used. Each type of stimulus material is further divided into different types of emotional valence, i.e. positive, neutral and negative, resulting in a total of 12 different conditions. The experiment is conducted in three rounds, and the stimulus materials used in each round include pictures, videos, sentences and music. The emotional valence of the stimulus materials is different in each round. In each round of experiment, the subjects receive electroencephalogram recording in four natural stimulus sessions, i.e. watching pictures, listening to sentences with eyes closed, watching videos and listening to music. The video is a silent movie clip, and the sentence is a non-native language sentence.

3. The method of claim 1, wherein the method further comprises: In step S2, the pre-processing steps of the data in the database include: S201: preview the electroencephalogram signals and remove data with obvious drift; S202: use notch filter technology to remove 50Hz line noise, and then perform band-pass filtering between 0.1Hz and 45Hz to remove noise at excessively high or low frequencies; S203: change the original reference electrode FCz to an average reference to optimize the signal reference method; S204: The electroencephalogram signal is decomposed into several independent components by independent component analysis (ICA), and the artifacts are accurately identified and removed by visual inspection, so as to effectively eliminate the noise components.

4. The method of claim 1, wherein the method further comprises: In step S2, the specific steps for calculating the inter-subject correlation (ISC) value of each segment include: First, for each pair of subjects k and l, compute the cross-electrode covariance matrix C kl which is: , where X k (t) and X l (t) represent the electroencephalogram signals of individuals k and l at time point t, respectively, μ k and μ l are the average values of individuals k and l over the entire time series, respectively, and T represents the total number of time points of the signal. Then, the inter-subject correlation (ISC) value between each pair of subjects is calculated using the following formula: , where V i represents the i-th eigenvector extracted by PCA or other methods to capture the key features of the signal; T is the total number of time points of the signal.

5. The method of claim 1, wherein the method further comprises: In step S3, the similarity of the intracerebral network is calculated using the following formula: , wherein C 1,k,t represents the edge weight of the node pair (k, t) in the functional connectivity matrix of the first subject, C 2,k,t represents the edge weight of the node pair (k, t) in the functional connectivity matrix of the second subject, Ne is the number of nodes; SimiNet ∈ [0, 1], 0 and 1 respectively represent that the brain functional connectivity matrices of the two subjects are not similar and completely identical.

6. The method of claim 1, wherein the method further comprises: In step S3, the global efficiency of the intercerebral network is calculated using the following formula: , where n is the total number of nodes in the network, N is the set of nodes, d ij is the shortest path length between node i and node j.

7. The method of claim 1, wherein the method further comprises: In step S3, under the same stimulation, the higher the similarity of the intracerebral network structure of the two subjects, the stronger the synchronization between the multiple brains. The global efficiency of the cross-brain network is used to measure the efficiency and stability of information transmission between the brain networks of the subjects. The higher the global efficiency, the stronger the integration ability of the intercerebral network, the higher the coordination ability of different electrodes or brain regions, and thus the higher the synchronization.

8. A system for detecting interpersonal neural synchrony under audiovisual stimulation, characterized in that, It includes: An audiovisual stimulus database construction module for designing a two-person hyperscanning experiment paradigm and collecting two-person electroencephalogram signals to construct an audiovisual stimulus database; An electroencephalogram data preprocessing module for preprocessing the data in the database to obtain electroencephalogram signals in four different frequency bands: delta, theta, alpha, and beta, and extracting electroencephalogram signals in the alpha band for segmentation to extract electroencephalogram signals under visual and auditory tasks; An inter-subject correlation calculation module for dividing the electroencephalogram signals preprocessed by the electroencephalogram data preprocessing module into several segments according to the time of the stimulus material, calculating the inter-subject correlation (ISC) value of each segment, and analyzing the trend and stability of the inter-subject correlation over time. The ISC value calculated from the electroencephalogram signals of the two subjects during each task is used to evaluate the similarity of brain activity between the subjects under the same task or stimulation. A brain network construction and evaluation module for constructing intracerebral networks and intercerebral networks by constructing a functional connectivity matrix, calculating the similarity of the intracerebral network, and calculating the global efficiency of the intercerebral network, thereby effectively evaluating the cooperation and synchronization between the neural activities of the subjects under visual and auditory stimulation. The specific steps for constructing the intracerebral network and the intercerebral network by constructing a functional connectivity matrix are as follows: First, the circular correlation coefficient (CCorr) is selected as the synchronization indicator: ， wherein, and denotes the phase value of the electroencephalogram signal on the two channels being compared, and denotes the average value of the phase of the signal itself over time on the two channels mentioned above, k denotes the kth time point, and N denotes the total number of time points, and denotes the phase value of the two channels at the kth time point; Then, the two subjects use a cap equipped with 32 electrodes, and the electroencephalogram signals of the two subjects are combined into a 64-electrode two-dimensional array. A functional connectivity matrix is constructed using correlation. The nodes in the matrix represent network nodes, and the values represent the weights of the connections between the nodes. The first 32 electrodes are intracerebral electrodes of the first subject, and the last 32 electrodes are intracerebral electrodes of the second subject. The CCorr is calculated between the intracerebral electrodes of the first subject to obtain the intracerebral network of the first subject, the CCorr is calculated between the intracerebral electrodes of the second subject to obtain the intracerebral network of the second subject, and the CCorr is calculated between the intracerebral electrodes of the first subject and the intracerebral electrodes of the second subject to obtain the intercerebral network.

9. The system for detecting interpersonal neuro- synchronization under audiovisual stimuli according to claim 8, characterized in that, The steps for preprocessing the data in the database by the electroencephalogram data preprocessing module include: First, the EEG signal is previewed to remove data with obvious drift; Second, notch filter technology is used to remove 50Hz line noise, and then band-pass filtering is performed between 0.1Hz and 45Hz to remove noise at too high or too low frequencies; Next, the original reference electrode FCz is changed to an average reference to optimize the signal reference method; Finally, the EEG signal is decomposed into several independent components by independent component analysis (ICA), and combined with visual inspection to accurately identify and remove artifacts, thereby effectively eliminating noise components.

Images