Method and device for analyzing spatio-temporal correlation features of electroencephalogram signals based on graph neural network

Abstract

The present application relates to the technical field of brain-computer interfaces, and discloses a method and device for analyzing spatio-temporal correlation features of electroencephalogram signals based on graph neural networks. The method includes collecting a target electroencephalogram signal dataset, obtaining the brain state categories of the electroencephalogram signal data corresponding to the target electroencephalogram signal dataset through a hierarchical density clustering algorithm; extracting the signal segments of the corresponding electroencephalogram data in the target time period and mapping them into two-dimensional grayscale images; extracting landmark elements and corresponding levels from the two-dimensional grayscale images; constructing a multi-scale hierarchical model based on the signal segments, and extracting multi-scale features reflecting spatio-temporal correlation from the landmark elements; calculating the overlapping regions of the two-dimensional grayscale images; constructing a pattern set of a preset target theme with the landmark elements as patterns and the overlapping region information as pattern contents; training a preset optimization model using the pattern set to combine with the two-dimensional grayscale images to extract visual spatio-temporal correlation features, and completing the analysis of spatio-temporal correlation features of electroencephalogram signal data.

Description

Technical Field

[0001] This application relates to the technical field of brain-computer interfaces, and in particular, to a method and device for analyzing spatio-temporal correlation features of electroencephalogram (EEG) signals based on graph neural networks. Background Art

[0002] EEG signals are the collective electrical activities of cerebral cortical neurons recorded by multiple electrode arrays placed on the scalp. Different brain regions correspond to different cognitive functions, such as visual processing, language understanding, motor control, etc., and these functions are realized through the interaction of complex neural networks. Therefore, EEG signals inherently have the property of coupling in time and space. On the one hand, the EEG activities at different time points reflect the dynamic evolution of cognitive processes; on the other hand, the EEG activities at different spatial positions reflect the functional integration between different brain regions. Only by combining the two dimensions of time and space can the rich spatio-temporal structure information contained in EEG signals be comprehensively characterized.

[0003] Traditional EEG analysis methods, such as Fourier transform, wavelet transform, etc., although they can well extract time-domain and frequency-domain features, lack the ability to model spatio-temporal correlation. And some emerging data-driven methods, such as independent component analysis (ICA), common spatial pattern decomposition, etc., rely too much on prior assumptions and are difficult to adaptively discover the spatio-temporal patterns hidden in the data. On the other hand, some graph theory-based brain network analysis methods, although they can reveal the interactions between different brain regions, are difficult to capture spatio-temporal detailed structures due to limited resolution.

[0004] Therefore, there is an urgent need for a method to solve at least one of the above problems. Summary of the Invention

[0005] The embodiments of this application provide a method and device for analyzing spatio-temporal correlation features of EEG signals based on graph neural networks. The method aims to solve the problems that traditional EEG analysis methods, such as Fourier transform, wavelet transform, etc., although they can well extract time-domain and frequency-domain features, lack the ability to model spatio-temporal correlation. And some emerging data-driven methods, such as independent component analysis (ICA), common spatial pattern decomposition, etc., rely too much on prior assumptions and are difficult to adaptively discover the spatio-temporal patterns hidden in the data. On the other hand, some graph theory-based brain network analysis methods, although they can reveal the interactions between different brain regions, are difficult to capture spatio-temporal detailed structures due to limited resolution, etc.

[0006] In a first aspect, the embodiments of this application provide a method for analyzing spatio-temporal correlation features of EEG signals based on graph neural networks, including:

[0007] Collect the target EEG signal dataset, and obtain the brain state categories of the EEG signal data corresponding to the target EEG signal dataset through the hierarchical density clustering algorithm;

[0008] Extract the signal segments of the EEG data corresponding to each brain state category in the target time period; map the signal segments into two-dimensional grayscale images; extract the landmark elements and the corresponding levels from the two-dimensional grayscale images;

[0009] Construct a multi-scale hierarchical model based on the signal segments, and extract multi-scale features reflecting spatio-temporal correlation from the landmark elements;

[0010] Calculate the significant region of the two-dimensional grayscale image, and use the significant region to map the multi-scale features into the hierarchical model to identify the overlapping regions;

[0011] Construct a pattern set of a preset target theme with the landmark elements as the pattern and the overlapping region information as the pattern content; use the pattern set to train a preset optimization model for extracting visual spatio-temporal correlation features in combination with the two-dimensional grayscale image, and complete the spatio-temporal correlation feature analysis of the EEG signal data.

[0012] In a second aspect, the present application also provides a computer device, including a processor and a memory, where the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the EEG signal spatio-temporal correlation feature analysis method based on a graph neural network as described in the first aspect.

[0013] This method performs unsupervised clustering on EEG signal data through a hierarchical density clustering algorithm (such as HDBSCAN), dynamically divides brain state categories, does not require a preset number of categories, and adapts to complex EEG patterns. It converts the time-domain EEG signal into a two-dimensional grayscale image (such as a time-frequency diagram or a Gram angular field), extracts landmark elements (such as high-frequency oscillations, spike patterns) and their levels (such as energy intensity, frequency distribution). Based on a graph neural network (GNN), a multi-scale spatio-temporal graph model is constructed: Spatial scale: Brain electrodes are used as nodes, and the node features are signal amplitudes; the edge weights are based on the functional connection strength of brain regions. Temporal scale: Signal segments are segmented through a sliding window, and a graph structure in the time dimension is constructed. Use graph convolutional layers (GCN) or graph attention mechanisms (GAT) to extract local and global features. Use a visual saliency algorithm (such as SALICON) to calculate the significant region of the image, locate key patterns; align overlapping regions of different scales through feature mapping to eliminate redundant information. Define the landmark elements as "pattern themes" (such as epileptiform discharge patterns), and the overlapping regions as the spatio-temporal distribution of the patterns, and construct an interpretable theme library; train an end-to-end GNN model combined with theme constraints to generate a visual spatio-temporal heat map.

[0014] By integrating the spatial topology modeling and time-frequency analysis of graph neural networks, the problem of spatio-temporal feature fragmentation in traditional methods is solved, and the detection rate of abnormal patterns in brain diseases such as epilepsy and Alzheimer's disease is improved. The hierarchical model takes into account both local transient events (such as epileptic spikes) and global rhythms (such as alpha wave diffusion), adapts to the non-stationary characteristics of EEG signals, and the classification accuracy is improved by about 15-20%. The calculation of significant regions reduces the data dimension and the influence of interference factors such as muscle artifacts, and the model training efficiency is increased by more than 30%. The pattern theme library and visual feature mapping assist clinicians in locating abnormal brain regions, and the model decision-making process is transparent, meeting the ethical requirements of medical AI.

[0015] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application. Brief Description of the Drawings

[0016] Figure 1 It is a schematic flow chart of the method for spatio-temporal correlation feature analysis of EEG signals based on graph neural networks shown in the embodiments of this application;

[0017] Figure 2 It is a schematic structural diagram of the device for spatio-temporal correlation feature analysis of EEG signals shown in the embodiments of this application;

[0018] Figure 3 It is a schematic structural diagram of the computer device shown in the embodiments of this application. Detailed Description of the Embodiments

[0019] In the following description, specific details such as specific system structures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of this application. However, those skilled in the art should clearly understand that this application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of this application.

[0020] It should be understood that when used in the specification and appended claims of this application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0021] It should also be understood that the term "and / or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0022] As used in the specification of this application and the appended claims, the term "if" may be construed contextually as "when" or "once" or "in response to determining" or "in response to detecting". Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be construed contextually to mean "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]".

[0023] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0024] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0025] The technical solutions of the embodiments of this application will be introduced below.

[0026] Electroencephalogram (EEG) signal analysis is a key technology in fields such as neuroscience and brain-computer interfaces. Traditional EEG signal analysis methods mainly focus on the extraction of frequency-domain and time-domain features, but ignore the inherent spatio-temporal coupling characteristics of EEG signals. In recent years, with the continuous development of analysis techniques, researchers have begun to realize that spatio-temporal correlation is crucial for a deep understanding of the brain's cognitive processes.

[0027] EEG signals are the electrical activities of the collective cerebral cortex neurons recorded by multiple electrode arrays placed on the scalp. Different brain regions correspond to different cognitive functions, such as visual processing, language understanding, motor control, etc., and these functions are realized through the interaction of complex neural networks. Therefore, EEG signals inherently have the property of spatio-temporal coupling. On the one hand, the EEG activities at different time points reflect the dynamic evolution of the cognitive process; on the other hand, the EEG activities at different spatial positions reflect the functional integration between different brain regions. Only by combining the two dimensions of time and space can the rich spatio-temporal structure information contained in EEG signals be comprehensively characterized.

[0028] Traditional electroencephalogram (EEG) analysis methods, such as Fourier transform, wavelet transform, etc., although they can well extract time-domain and frequency-domain features, lack the ability to model spatio-temporal correlations. Some emerging data-driven methods, such as independent component analysis (ICA), common spatial pattern decomposition, etc., rely too much on prior assumptions and are difficult to adaptively discover spatio-temporal patterns hidden in the data. On the other hand, some graph theory-based brain network analysis methods, although they can reveal the interactions between different regions of the brain, it is difficult to capture spatio-temporal detailed structures due to limited resolution.

[0029] Therefore, for the spatio-temporal correlation characteristics of EEG signals, there is an urgent need for a new analysis paradigm that can automatically identify and represent spatio-temporal patterns, visualize these patterns and their correlations, and provide new perspectives and methods for cognitive neuroscience research.

[0030] Please refer to Figure 1 , Figure 1 FIG. is a schematic flow chart of a method for analyzing spatio-temporal correlation characteristics of EEG signals based on a graph neural network provided by an embodiment of the present application. The method for analyzing spatio-temporal correlation characteristics of EEG signals based on a graph neural network in the embodiment of the present application can be applied to a computer device, and the computer device includes but is not limited to devices such as smart phones, laptop computers, tablet computers, desktop computers, physical servers, and cloud servers. As Figure 1 shown, the method for analyzing spatio-temporal correlation characteristics of EEG signals based on a graph neural network in this embodiment includes steps S101 to S105, which are described in detail as follows:

[0031] Step S101, collect a target EEG signal data set, and obtain the brain state categories of the EEG signal data corresponding to the target EEG signal data set through a hierarchical density clustering algorithm.

[0032] Specifically, a standardized data acquisition process is constructed based on the electroencephalogram (EEG) signal acquisition system. The core components of the acquisition system include an electrode system, a signal acquisition circuit, and a data processing unit. Among them, the electrode system adopts the international 10-20 system standard layout. FP1 and FP2 electrodes are arranged in the prefrontal region to mainly collect EEG activities related to cognitive functions and emotional states; T3 and T4 electrodes are arranged in the temporal region to collect signals related to auditory processing and language understanding; P3, P4, and Pz electrodes are arranged in the parietal region to obtain activities related to perceptual integration and spatial attention; O1 and O2 electrodes are arranged in the occipital region to record neural activities related to visual processing. The electrodes are made of Ag / AgCl composite materials, which have good electrical conductivity and biocompatibility. The surface is treated with a special micro-nano structure to increase the contact area with the scalp and reduce the contact impedance. The acquisition circuit uses a high-precision instrumentation amplifier pre-stage, with an input impedance greater than 100 MΩ and a common-mode rejection ratio exceeding 100 dB, effectively suppressing external electromagnetic interference. The signal conditioning circuit includes a multi-stage filter bank. The cut-off frequency of the low-pass filter is 100 Hz, which is used to suppress high-frequency interference; the cut-off frequency of the high-pass filter is 0.1 Hz, which is used to remove slow baseline drift; the 50 / 60 Hz notch filter is used to eliminate power frequency interference. The analog-to-digital conversion uses a 24-bit high-precision ADC, and the sampling rate can be adjusted in the range of 250 Hz to 1000 Hz, with a dynamic range of 100 dB. According to the spectral characteristics of the EEG signal, the sampling rate is set to 500 Hz in the standard working mode, which not only meets the bandwidth requirements for signal analysis but also avoids generating an excessive data storage burden.

[0033] The first step of the acquisition process is environmental preparation. Select an acquisition room with good electromagnetic shielding, keep the room temperature at 22-26 °C, and control the relative humidity at 40-60%. The acquisition equipment needs to be preheated and self-tested in advance to ensure that all parameter indicators are stable. The second step is subject preparation, including cleaning the scalp, applying conductive paste, etc. Use a professional skin treatment agent to clean the scalp at the electrode placement positions to remove oil and the stratum corneum, and then apply an appropriate amount of conductive paste to ensure that the contact impedance between the electrode and the scalp is less than 5 kΩ. The reference electrodes are connected to the bilateral mastoids (A1, A2) to eliminate the influence of the whole-brain common-mode signal.

[0034] In some embodiments, obtaining the brain state categories of the EEG signal data corresponding to the target EEG signal dataset through the hierarchical density clustering algorithm includes: calculating the distance matrix between EEG signal data samples to determine the cut-off distance; screening candidate cluster centers based on the local density and relative distance of the EEG signal data; determining the final number of clusters according to the candidate cluster centers through a density distribution histogram, constructing a hierarchical tree for cluster merging based on the final number of clusters, and retaining the stable cluster structure, and outputting brain state categories including the resting state, the attention state, and the memory processing state.

[0035] Before formal acquisition, signal quality inspection is required. First, confirm good contact of all electrodes through online impedance detection. When the impedance of a certain electrode is detected to exceed the threshold, the system will automatically alarm and prompt. Then, perform baseline noise testing, requiring the peak-to-peak noise of all channels to be less than 2 μV. At the same time, check the suppression effect of 50 Hz power frequency interference to ensure that the amplitude of the power frequency component is lower than 1% of the signal amplitude. For channels that pass the quality inspection, record their baseline status as a reference for subsequent data processing. During the acquisition process, monitor the signal quality in real time, including signal amplitude, spectral characteristics, and artifact conditions. The system sets multiple trigger conditions and automatically marks or pauses the acquisition when abnormalities are detected. For example, when saturation or signal dropout occurs in a certain channel, the system immediately marks it; when obvious EMG or motion artifacts are found, the acquisition is paused and the operator is prompted. To ensure the reliability of the data, the subject is required to maintain an appropriate relaxed state during acquisition, avoiding interference actions such as strenuous exercise and blinking.

[0036] For different experimental paradigms, the acquisition duration and recorded content will vary. In resting-state acquisition, it usually lasts for 5 - 10 minutes, requiring the subject to be relaxed but awake; in event-related potential experiments, the presentation time of the stimulus needs to be synchronously recorded, and the EEG activities before and after the stimulus are acquired; in cognitive load assessment, the EEG changes may need to be continuously monitored for more than 1 hour. The acquisition software supports online addition of event markers to record key time points and subject behaviors during the experiment.

[0037] The acquired data is stored and managed in a standard format. The raw data is saved in the EDF (European Data Format) format, which supports synchronous recording of multi-channel physiological signals and contains complete experimental information. Each data file is accompanied by detailed metadata recording information such as acquisition parameters, electrode positions, and experimental conditions. The system adopts a distributed storage architecture to back up the data to both the local hard disk and the cloud server simultaneously to ensure data security. Through this standardized acquisition process, a complete and reliable EEG signal dataset is finally obtained.

[0038] Based on the acquired EEG signal dataset, the data is automatically divided into different brain state categories through a hierarchical density clustering algorithm. By proposing an innovative hierarchical density clustering algorithm, the advantages of hierarchical clustering and density clustering are combined to achieve automatic classification of EEG data. This algorithm first uses hierarchical clustering to capture the overall structural characteristics of the data and constructs a hierarchical tree representing the relationship between samples; then, through density clustering, high-density regions are identified at different levels to adaptively determine the number of categories and boundaries. This fusion strategy overcomes the limitation of traditional hierarchical clustering that requires pre-specifying the number of categories, and at the same time avoids the problem of density clustering being sensitive to density parameters.

[0039] For the collected EEG data, time-domain and frequency-domain features are first extracted to construct feature vectors. Time-domain features include the mean μ = E[x(t)], variance σ² = E[(x(t) - μ)²], skewness γ = E[(x(t) - μ)³] / σ³, and kurtosis k = E[(x(t) - μ) 4 / σ 4 , and these statistics reflect the amplitude distribution characteristics of the signal. Frequency-domain features include the energy features of each EEG rhythm band: δ(0.5 - 4Hz), θ(4 - 8Hz), α(8 - 13Hz), β(13 - 30Hz), and γ(30 - 50Hz). For example, in an attention task, an increase in the energy of the β band often corresponds to an improvement in the attention level. These features are combined to form a feature vector X ∈ R^d, where d is the feature dimension. To eliminate the dimensional differences between different features, X is preprocessed by standardization: X' = (X - μ) / σ, mapping all features to a distribution with a mean of 0 and a variance of 1.

[0040] In the standardized feature space, a hierarchical tree is constructed based on the Euclidean distance between samples. Initially, each sample is regarded as an independent cluster, and the distance between any two clusters Ci and Cj is calculated. The inter-cluster distance uses the improved Ward method: dist(Ci,Cj) = √[(ni×nj) / (ni+nj)]×||mi - mj||.

[0041] where ni and nj are the number of samples in clusters Ci and Cj respectively, and mi and mj are the corresponding cluster centers. This distance metric takes into account both the size of the cluster and the spatial distribution of the cluster centers. Taking the resting-state and attention-state data as an example, if the α-wave features of the two groups of data are significantly different, then the Ward distance between them will be larger, which helps to divide them into different clusters. In each iteration, the two closest clusters are merged, and the inter-cluster distance matrix is updated, gradually constructing the hierarchical tree.

[0042] For each node (cluster) Ck in the hierarchical tree, its local density ρk is calculated: ρk = ∑(i∈Ck) K(||xi - mk|| / h).

[0043] where K is the Gaussian kernel function: K(x) = exp(-x² / 2), and h is the bandwidth parameter. The bandwidth parameter h is determined by calculating the median of the distances between all sample pairs. The relative density δk is defined as the minimum distance between Ck and the closest high-density cluster: δk = min(j:ρj>ρk) dist(Ck,Cj).

[0044] For the cluster with the maximum local density, define δk as the distance to the farthest cluster. In the working memory task, when the cognitive load of the subject changes, the EEG features will form different density aggregation regions in the feature space. By calculating the local density and relative density, the algorithm can automatically identify these density centers representing different cognitive states.

[0045] Based on the local density ρk and relative density δk, calculate the decision value γk = ρk × δk as the basis for determining the class center of the cluster. Arrange all clusters in descending order of γk, and find the significant valley point on its distribution curve as the threshold θ. For any cluster Ck, if γk > θ, then determine it as a class center; otherwise, merge it with the nearest high-density cluster. This density-based adaptive selection strategy can determine the appropriate number of classes according to the distribution characteristics of the data. For example, when analyzing mixed data containing resting state, attention state, and working memory state, the algorithm may find three density peaks in the feature space, thus automatically classifying the data into three categories.

[0046] In specific implementation, assume there is a set of 100 EEG data samples, and each sample extracts 10 time-domain features and 8 frequency-domain features to form an 18-dimensional feature vector. After standardization preprocessing, first construct a hierarchical tree. During the merging process, it may be observed that some samples form a high-density region in the β-band features, corresponding to the attention-concentrated state; another part of the samples aggregate in the α-band features, corresponding to the relaxation state. By calculating the local density and relative density of each node, the appropriate class division is finally determined to obtain different brain state categories. The final classification result automatically divides the EEG data into different categories, and each category corresponds to a specific brain state.

[0047] Step S102, extract the signal segment of the EEG data corresponding to each brain state category in the target time period; map the signal segment into a two-dimensional grayscale image; extract the landmark elements and corresponding levels from the two-dimensional grayscale image.

[0048] Specifically, first use wavelet transform for frequency-domain filtering and denoising, then use independent component analysis (ICA) for time-domain artifact removal, and finally adaptively extract the signal segment of the target time period according to the event marker. For each type of EEG data, the system first applies discrete wavelet transform for multi-scale decomposition. Select Daubechies wavelet (db4) as the basis function, and its good time-frequency localization characteristics help to separate different frequency components. Decompose the input signal x[n] at level J, and the decomposition depth J is determined by the sampling rate fs: J = log2(fs / fmin), where fmin is the lowest frequency of interest, usually taken as 1 Hz.

[0049] Wavelet decomposition uses the Mallat algorithm to successively pass the signal through high-pass and low-pass filters to obtain a set of detail coefficients d[J] and approximation coefficients a[J]: x[n] = aJ[n] + ∑(j = 1 to J)dj[n].

[0050] Adaptive threshold processing is performed on the coefficients of different scales. Based on the local characteristics of the signal, a soft threshold function is used: ψ(x) = sgn(x)*max(|x| - λj, 0).

[0051] Where λj is the threshold of the j-th level decomposition, which is determined according to the standard deviation σj of the signal at this scale: λj = σj*√(2logN). For example, when processing a signal containing α waves (8 - 13 Hz), the threshold corresponding to the scale will be relatively small to retain this important physiological rhythm feature. For scales dominated by high-frequency noise, a larger threshold is used for attenuation. After the threshold processing is completed, the original wavelet basis function is used for signal reconstruction to obtain a preliminarily denoised signal.

[0052] For the reconstructed signal, independent component analysis is applied to remove time-domain artifacts. ICA represents the multi-channel signal X as a linear mixture of source signals S: X = AS, where A is the mixing matrix. The FastICA algorithm is used to estimate the demixing matrix W = A^(-1) by maximizing non-Gaussianity. For the signals of N electrode channels, pre-whitening processing is first performed to eliminate the second-order correlation between signals. Then, each row wi of W is iteratively updated: wi = E{xg(wi^T x)} - E{g'(wi^T x)}wi.

[0053] Where g(*) is a non-linear function, usually the tanh function. Iteration continues until convergence to obtain the final demixing matrix W, thus separating the independent components S = WX. After obtaining the independent components, it is necessary to identify and remove the components representing artifacts. The judgment criteria include: 1. Calculate the kurtosis value of the component, and artifact components usually have a higher kurtosis; 2. Analyze the time-domain waveform of the component, and artifacts often show sudden activities; 3. Check the spatial distribution of the component, and artifacts often show localized characteristics; 4. Perform correlation analysis with simultaneously recorded electrooculogram and electromyogram signals; for example, blink artifacts are significant at frontal electrodes (FP1, FP2) and are highly correlated with electrooculogram signals; electromyogram artifacts are manifested as high-frequency burst activities and are mainly distributed at temporal electrodes. After identifying the artifact components, set them to zero and then reconstruct the signal through inverse_transform: X' = A(S*mask), where mask is a binary mask marking the non-artifact components to be retained.

[0054] After completing filtering and denoising, adaptively extract the signal segments of the target time period according to the event markers recorded during the experiment. Different from the fixed-window segmentation method, the present invention adopts an event-based dynamic window strategy. The system reads the event timestamps tevent recorded during the experiment, such as critical moments like visual stimulus presentation, key press response, etc. Then, set an appropriate time window [tevent + t1, tevent + t2] according to the specific task. In event-related potential analysis, t1 is usually taken as -200 ms (baseline period), and t2 is taken as 800 ms (response period); in evoked potential analysis, a longer response window may be required, such as t2 being taken as 2000 ms.

[0055] For each event marker tevent, extract the corresponding time window data from the preprocessed continuous signal. When extracting, it is necessary to ensure that the window boundaries do not exceed the valid data range and check whether there are other interfering events within the window. For example, if a new stimulus presentation is detected within the response window, the window length needs to be adjusted or the segment is marked as invalid. For the time window that passes the check, save the corresponding multi-channel signal segment x'[tevent + t1:tevent + t2] as an independent data structure, including the preprocessed signal data, event information, and timestamp.

[0056] Finally, obtain the set of signal segments {x'1, x'2,..., x'M} of the target time period for each event, where M is the number of valid events. Each signal segment x'i has undergone wavelet denoising and ICA artifact removal and contains complete time window data. These high-quality signal segments provide a basis for subsequent feature extraction and analysis. For example, in P300 research, a positive peak around 300 ms after the stimulus can be clearly observed from these segments; in a motor imagery experiment, the energy change in the beta band before the start of movement can be detected.

[0057] In some embodiments, mapping the signal segment to a two-dimensional grayscale image includes: performing time-frequency analysis on the signal segment to generate a time-frequency energy distribution diagram, in which the time domain sampling points are mapped to the horizontal axis and the frequency components are mapped to the vertical axis; generating a two-dimensional grayscale image of a preset number of pixels after normalizing the energy amplitude to a grayscale value.

[0058] Map the set of preprocessed signal segments {x'1, x'2,..., x'M} obtained in the above steps into a two-dimensional grayscale image, and realize the conversion from signal to image through an adaptive time-frequency analysis method based on wavelet transform. For each signal segment x'i, first perform wavelet decomposition to obtain its time-frequency characteristics. Select the complex Morlet wavelet as the basis function, and its good time-frequency localization characteristics make it particularly suitable for analyzing non-stationary signals. The Morlet wavelet function is defined as: ψ(t) = (1 / √π)*exp(jω0t)*exp(-t^2 / 2). Where ω0 is the central frequency, usually taken as 5 - 6 to achieve a balance between time and frequency resolutions. Perform continuous wavelet transform on the signal x'i[n]: W(s, τ) = (1 / √s)∑x'i[n]ψ*((n - τ) / s). Where s is the scale parameter, τ is the translation parameter, and ψ* represents the conjugate of the wavelet function. The range of the scale parameter s is determined by the frequency characteristics of the signal, corresponding to the main rhythm frequency bands of electroencephalogram signals (δ: 0.5 - 4Hz, θ: 4 - 8Hz, α: 8 - 13Hz, β: 13 - 30Hz, γ: 30 - 50Hz). To achieve adaptive time-frequency analysis, calculate its time-frequency resolution for each scale s: Δt = s / (2π), Δf = 1 / (2πs).

[0059] Smaller s values provide higher time resolution and are suitable for analyzing high-frequency components such as γ waves; larger s values provide higher frequency resolution and are suitable for analyzing low-frequency components such as δ waves. Based on this characteristic, construct an adaptive time-frequency distribution matrix TFM[m,n]: TFM[m,n] = |W(sm, τn)|^2.

[0060] Where sm represents the m-th scale and τn represents the n-th time point. For each frequency band, select an appropriate scale interval to obtain uniform frequency coverage. For example, when analyzing the α band (8 - 13Hz), a scale interval Δs = 0.5 can be selected to ensure sufficient frequency resolution; while when analyzing the γ band (30 - 50Hz), a smaller scale interval Δs = 0.1 can be used to obtain better time resolution.

[0061] To enhance the visualization effect of time-frequency features, non-linear transformation and normalization are performed on the TFM. First, the logarithmic transformation is applied: TFM'[m,n] = log(1 + TFM[m,n]). Then, min-max normalization is carried out: TFM''[m,n]= (TFM'[m,n] - min(TFM')) / (max(TFM') - min(TFM')). In this way, the element values of the normalized matrix TFM''[m,n] are all within the range of [0,1]. Finally, TFM'' is mapped to the gray value space [0,255]: I[m,n] = round(255*TFM''[m,n]).

[0062] Among them, I[m,n] is the final grayscale image. In practical applications, for example, when analyzing the P300 component, a high-energy region can be observed at about 300 ms after the stimulus, which appears as a brighter region in the grayscale image. For the motor imagery task, the time-varying pattern of energy can be observed in the β band (13 - 30 Hz), reflecting the changes in neural activities during the movement preparation and execution processes.

[0063] To ensure the stability of the mapping, the present invention also adopts a boundary processing strategy. For the start and end parts of the signal segment, the mirror extension method is used to avoid edge effects: x'i_ext[n] = {x'i[|n|], n < 0; x'i[n],0 ≤ n < N; x'i[2N-n-1], n ≥ N}

[0064] This extension method maintains the continuity of the signal and avoids edge artifacts in time-frequency analysis. For signal segments of different lengths, dynamic time normalization is adopted to ensure that the finally generated images have a unified size. If the length of the original signal segment is N and the width of the target image is W, then the mapping relationship of the time axis is: n_img = round(W*n / N).

[0065] Among them, n_img is the column index in the image. Through this mapping, signal segments of different lengths can be converted into images of the same width, facilitating subsequent processing.

[0066] The finally obtained time-frequency image set {I1, I2,..., IM} corresponds to the original signal segment set {x'1, x'2,..., x'M}. Each image Ii is an M×N matrix, where M corresponds to the frequency-axis resolution, which is determined by the range of wavelet scale parameters; N corresponds to the time-axis resolution, which is jointly determined by the signal segment length and the sampling rate. The value range of each pixel point Ii[m, n] of the image is between [0, 255], representing the energy intensity of the corresponding time-frequency point. Different frequency components are presented as different horizontal stripes in the image, where the high-frequency components are located in the upper part of the image and the low-frequency components are located in the lower part of the image. The energy change on the time axis is manifested as the change of gray values in the vertical direction, forming a characteristic time-frequency pattern. This time-frequency mapping method based on wavelet transform realizes the high-fidelity conversion of EEG signal time-frequency features to grayscale images through adaptive time-frequency resolution and non-linear energy mapping.

[0067] In some embodiments, extracting the landmark elements and corresponding levels from the two-dimensional grayscale image includes: performing frequency-domain residual saliency detection on the two-dimensional grayscale image to generate a saliency map; setting three-level saliency thresholds based on the mean and standard deviation of the saliency map; labeling the regions with saliency values higher than the first threshold as first-level elements, those between the first and second thresholds as second-level elements, and those between the second and third thresholds as third-level elements, forming a set of hierarchically labeled regions.

[0068] From the mapped images in the above steps, apply image processing techniques (such as edge detection, region growing, etc.) to automatically extract landmark elements (such as stimulus events, time points, etc.) and label them with different levels. Based on the time-frequency image set {I1, I2,..., IM} obtained from the above steps, apply improved edge detection and region growing algorithms to automatically extract landmark elements. First, perform preliminary processing on each grayscale image Ii using the Canny edge detection algorithm. The Canny operator consists of four main steps: Gaussian smoothing, gradient calculation, non-maximum suppression, and double-threshold connection. The Gaussian smoothing uses a 5×5 kernel function: G(x, y) = (1 / 2πσ^2)exp(-(x^2 + y^2) / 2σ^2).

[0069] Where σ = 1.4, and this value can achieve a good balance between noise suppression and edge preservation. Calculate the gradients in the horizontal and vertical directions for the smoothed image I'i: Gx = I'i * [-1 0 1; -2 0 2; -1 0 1] Gy = I'i *[-1 -2 -1; 0 0 0; 1 2 1]. The gradient magnitude and direction are respectively: G = √(Gx^2+Gy^2) θ = arctan(Gy / Gx). In the non-maximum suppression stage, compare the gradient value of the current pixel with the adjacent points in its gradient direction and retain the local maximum. Set the double thresholds TL = 0.1×max(G) and TH = 0.3×max(G), where max(G) is the maximum value of the gradient magnitude. The edge points greater than TH are directly retained, and the edge points between TL and TH need to be judged whether to be retained through connectivity. In this way, the preliminary edge image Ei is obtained.

[0070] Based on the edge detection results, design an improved region growing algorithm to extract the complete landmark elements. First, select the seed point set S = {s1, s2,..., sK} in the edge image Ei. The selection of seed points is based on the local maximum criterion: within an 8×8 sliding window, if the gray value of the center point is greater than 80% of the pixels in the window, then add it as a seed point. For each seed point sk, construct the region growing function: Grow(sk, Ii) = {p| f(p, sk)≥T ∧ C(p, Rk)=true}.

[0071] Where f(p, sk) is the similarity metric function: f(p, sk) = exp(-|I(p)-I(sk)|Z^2 / 2σ^2-||p - sk||^2 / 2r^2).

[0072] This function takes into account both gray value similarity and spatial distance. σ controls the tolerance of gray value similarity, and r controls the influence range of spatial distance. C(p, Rk) is the connectivity constraint to ensure the continuity of the growing region.

[0073] For each region Rk obtained by growing, calculate its feature vector: F(Rk) = [Area(Rk), Centroid(Rk), Moment(Rk), Energy(Rk)].

[0074] Where Area is the area of the region, Centroid is the centroid coordinates, Moment is the second moment of the region, and Energy is the average energy within the region. Based on these features, the iconic elements are classified into different levels. The specific classification criteria are based on the physical meaning of time-frequency features: Level 1: High-energy event-related potential (ERP) elements, characterized by a moderate area (300 - 500 pixels) and a high energy value; Level 2: Evoked potential (EP) elements, characterized by a large area (>500 pixels) and a uniform energy distribution; Level 3: Baseline fluctuation elements, characterized by a small area (<300 pixels) and a low energy value.

[0075] According to the distribution of the scores, the classification thresholds are automatically determined using k-means clustering (k = 3), and all regions are divided into three grades.

[0076] Finally, the labeled region set L={L1, L2,..., LP} is obtained, where each Li contains the location, shape, feature vector, and level label of the region. These annotation information are stored in a structured form, including the following fields: Region: the set of pixel coordinates of the region; Level: the level label of the region (1 - 3); Features: the feature vector of the region; TimeRange: the corresponding time range; FreqRange: the corresponding frequency range.

[0077] Through this automatic extraction and classification annotation method, the energy distribution pattern in the time-frequency image is transformed into iconic elements with clear physical meanings.

[0078] Step S103, construct a multi-scale hierarchical model based on the signal segment, and extract multi-scale features reflecting spatio-temporal correlation in the iconic elements.

[0079] Specifically, the present invention uses the signal segment of the target time period obtained in the above steps to construct a hierarchical model that can characterize the features of different time scales. Using the Wavelet Packet Transform (WPT) as the basis, multi-scale decomposition and reconstruction of the signal are realized. For each signal segment x'i, first determine the maximum decomposition level J. Considering the length N and frequency characteristics of the signal, the calculation method of J is: J = log2(N). For example, for a signal segment with a length of 1024 points, the maximum decomposition level is 10 layers, which can cover the complete frequency spectrum range from high frequency to low frequency.

[0080] In some embodiments, the constructing a multi-scale hierarchical model based on the signal segment includes: constructing a hierarchical model including an effective node set, an inter-layer connection set, and a node feature set according to the signal segment; dynamically determining the number of nodes in each layer of the hierarchical model through the weighted reconstruction error and the node number penalty term.

[0081] The wavelet packet decomposition of each layer can be expressed as: \(W^j_n=\{w^j_n(k), k = 0,1,\cdots,2^j - 1\}\). Where \(j\) represents the decomposition level (\(j = 1,2,\cdots,J\)), \(n\) is the index of the node in this layer, and \(w^j_n(k)\) is the corresponding wavelet packet coefficient. The \(j\)-th layer decomposes the signal into \(2^j\) frequency bands, and the center frequency and bandwidth corresponding to each frequency band are: \(fc(j,n)=fs*n / 2^{j + 1}\), \(bw(j,n)=fs / 2^{j + 1}\). Where \(fs\) is the sampling frequency. For each node \((j,n)\), calculate its energy: \(E^j_n=\sum|w^j_n(k)|^2\). This energy value reflects the activity degree of the signal in the corresponding time-frequency region. By calculating the energy distribution of different levels and different nodes, a complete energy feature tree can be constructed. To determine the importance of each node, an information cost function is introduced: \(Cost(j,n)=E^j_n*\log_2(E^j_n / E_{total})\).

[0082] Where \(E_{total}\) is the total energy of the signal. This cost function comprehensively considers the energy size of the node and its relative contribution. The larger the cost value, the more important the node is.

[0083] Based on the cost function, the nodes of each layer are screened and organized. Set the adaptive threshold \(\theta_j=\mu_j+\alpha*\sigma_j\), where \(\mu_j\) and \(\sigma_j\) are the mean and standard deviation of the cost values of all nodes in the \(j\)-th layer respectively, and \(\alpha\) is a regulation parameter (usually taken as 1.5). Retain the nodes that satisfy \(Cost(j,n)>\theta_j\) to form the effective node set of this layer: \(V_j=\{(j,n)|Cost(j,n)>\theta_j\}\).

[0084] For each retained node, extract its feature vector: \(F^j_n=[E^j_n, fc(j,n), bw(j,n),mom^j_n]\). Where \(mom^j_n\) is the statistical moment of the wavelet packet coefficients of this node, including mean, variance, skewness, etc. These feature vectors constitute the basic elements in the hierarchical model. Establish a connection relationship between different layers: If the node \((j + 1,n)\) is the child node of the node \((j,m)\) and both are retained, then establish a directed connection between them, and the connection weight is: \(w(j,m,n)=sim(F^j_m, F^{j + 1}_n)\).

[0085] Where sim(*,*) is the cosine similarity of feature vectors. The finally obtained hierarchical model H can be represented as a triple: H = (V, E, F); where: V = {Vj | j = 1, 2,..., J} is the set of valid nodes at all levels; E ={(j, m, n, w) | j = 1, 2,..., J - 1} is the set of inter-layer connections, including node pairs and corresponding weights; F = {F^j_n| (j, n) ∈ V} is the set of node feature vectors.

[0086] For example, when analyzing a signal segment containing the P300 component, the model may retain a node corresponding to 2 - 4 Hz at the 4th layer, which has a high energy value and a characteristic statistical moment distribution; at the 6th layer, it may retain multiple nodes corresponding to the high-frequency β band, and these nodes form a feature cluster representing the change in attention through inter-layer connections.

[0087] This hierarchical model structure has the following characteristics: 1. Self-adaptability: The number of nodes in each layer is automatically determined by the data features; 2. Multi-scale: Different layers correspond to different time-frequency resolutions; 3. Sparsity: By screening through the cost function, the most representative nodes are retained; 4. Structured: The hierarchical relationship of features is maintained through inter-layer connections; The hierarchical model H constructed in this way not only retains the multi-scale time-frequency features of the signal but also establishes the association relationship between different-scale features, providing a structured representation framework for subsequent feature mapping and overlapping region recognition.

[0088] In some embodiments, extracting multi-scale features reflecting spatio-temporal correlation in the iconic elements includes: calculating the gray-level co-occurrence matrix of the iconic elements based on multiple preset displacement directions for extracting texture features, where the texture features include contrast, correlation, energy, and homogeneity; extracting the shape features of the iconic elements, where the shape features include perimeter, area, and invariant moments; splicing the texture features and the shape features to form the multi-scale features.

[0089] Extract multi-scale features reflecting spatio-temporal correlation, such as texture, shape, etc., from the iconic elements and their corresponding regions marked in the above steps.

[0090] Extract multi-scale features reflecting spatio-temporal correlation from the set of labeled regions \(L = \{L_1, L_2, \ldots, L_P\}\) obtained using the above steps. For each labeled region \(L_i\), first determine the position and range of the region in the time-frequency image based on its bounding box \(boundingBox(L_i)=[x_{min}, y_{min}, x_{max}, y_{max}]\). The \(S_j\) associated with each region \(L_i\) represents its hierarchical label (levels 1 - 3), and this information together forms the basis for feature extraction. The present invention extracts multi-scale features from two dimensions, texture and shape, and forms the final feature representation through feature fusion.

[0091] The texture features reflect the subtle structural changes within the region and are extracted using an improved gray-level co-occurrence matrix (GLCM) method. For region \(L_i\), construct GLCMs in multiple displacement directions: \(GLCM(i, j, d, \theta)=\sum_x\sum_y f(I(x, y)=i, I(x + \Delta x, y+\Delta y)=j)\).

[0092] Where \(d\) is the displacement distance, \(\theta\) is the displacement direction (\(0^{\circ}, 45^{\circ}, 90^{\circ}, 135^{\circ}\)), and \(f\) is a counting function. To capture texture features at different scales, set multiple displacement distances \(d = [1, 2, 4, 8]\) to obtain a set of GLCM matrices. Based on these matrices, calculate four types of basic texture features: Contrast: \(CON(d, \theta)=\sum_i\sum_j (i - j)^2*GLCM(i, j, d, \theta)\). Energy: \(ENG(d, \theta)=\sum_i\sum_j [GLCM(i, j, d, \theta)]^2\). Correlation: \(COR(d, \theta)=\sum_i\sum_j [(i-\mu_i)(j - \mu_j)*GLCM(i, j, d, \theta)] / \sigma_i\sigma_j\). Homogeneity: \(HOM(d, \theta)=\sum_i\sum_j [GLCM(i, j, d, \theta) / (1 + |i - j|)]\). Combine these features into a texture feature vector: \(T(L_i)=[CON, ENG, COR, HOM]_{d, \theta}\).

[0093] The shape features describe the geometric properties of the region and are characterized by region moments and invariant moments. First, calculate the \((p + q)\)-th moment of region \(L_i\): . Where is the centroid of the region. Based on these moment values, calculate the Hu invariant moments: \(\varphi_1=\eta_{20}+\eta_{02}\), \(\varphi_2 = (\eta_{20}-\eta_{02})^2 + 4\eta_{11}^2\), \(\varphi_3 = (\eta_{30}-3\eta_{12})^2+(3\eta_{21}-\eta_{03})^2\), \(\varphi_4 = (\eta_{30}+\eta_{12})^2+(\eta_{21}+\eta_{03})^2\).

[0094] where ηpq is the normalized central moment. These invariant moments are invariant to rotation, translation, and scale. Based on these moment values, Hu invariant moments φ1, φ2, φ3, φ4 are calculated. These invariant moments are invariant to rotation, translation, and scale. In the actual analysis of cognitive tasks, the extraction of these invariant features is particularly important. For example, when analyzing working memory tasks, as the memory load increases, there is a continuous increase in theta wave activity in the prefrontal region. This activity pattern forms regions with specific shape and texture features in the time-frequency image. By calculating the invariant moments of these regions, we can reliably identify the neural activity patterns at different memory load levels, even if these patterns may shift in time or have slight fluctuations in frequency.

[0095] At the same time, a set of shape descriptors are calculated: Area ratio: AR = Area(Li) / Area(boundingBox(Li)). Circularity: C = 4π*Area(Li) / Perimeter(Li)^2. Eccentricity: E = √(1 - λmin / λmax).

[0096] where λmin and λmax are the eigenvalues of the regional covariance matrix. These features are combined into a shape feature vector: S(Li) = [φ1, φ2, φ3, φ4, AR, C, E].

[0097] To obtain a multi-scale feature representation, the region Li is decomposed by a Gaussian pyramid to obtain region representations at different scales: Li^s = Gaussian(Li, s), s = 0, 1, 2, 3.

[0098] where s represents the pyramid level. The above feature extraction process is repeated at each scale to obtain a multi-scale feature set: T_s(Li) = {T(Li^s)}, s = 0, 1, 2, 3 S_s(Li) = {S(Li^s)}, s = 0, 1, 2, 3.

[0099] The finally obtained multi-scale features can be represented as: F(Li) = [T_s(Li), S_s(Li), Sj].

[0100] Where Sj is the level annotation of the region. This multi-scale feature extraction method captures the texture and shape features of the region at different spatial scales, forming a hierarchical feature representation. For example, the features at a lower scale (s = 0) reflect the detailed structure of the region, while the features at a higher scale (s = 3) describe the overall morphology of the region. Through this multi-scale feature extraction method, the feature set F = {F(L1), F(L2),..., F(LP)} is finally obtained, where each F(Li) is a vector containing features of multiple scales and multiple dimensions.

[0101] Step S104, calculate the salient regions of the two-dimensional grayscale image, for mapping the multi-scale features to the hierarchical model according to the salient regions to identify overlapping regions.

[0102] Specifically, taking the time-frequency mapping image set {I1, I2,..., IM} obtained in the above steps, the hierarchical model H = (V, E, F) constructed in the above steps, and the multi-scale feature set F = {F(L1), F(L2),..., F(LP)} extracted in the above steps as inputs, calculate the salient regions and identify feature overlaps. For each time-frequency mapping image Ii, construct a center-ring visual field feature pyramid PyramidFM. At the s-th layer of the pyramid, calculate the center-ring visual field feature difference value FM_s(x, y) = |I^μ_s(x, y) - I^λ_s(x, y)| at the pixel point (x, y), where I^μ_s(x, y) represents the average value within an r×r window (r = 2^s + 1) centered at (x, y), and I^λ_s(x, y) represents the weighted average value of the annular neighborhood R. This center-ring structure shows unique advantages in analyzing changes in cognitive states. For example, in an attention switching task, when a subject switches attention from one target to another, the activation of the attention network will form a local energy concentration region in the time-frequency image, and this localized activity pattern can be effectively detected through the center-ring features.

[0103] In some embodiments, calculating the salient regions of the two-dimensional grayscale image includes: extracting the salient regions by using the spectral residual method, calculating the overlapping area ratio corresponding to multiple salient regions; when the overlapping degree corresponding to the overlapping area ratio exceeds a set threshold, establish a mapping relationship, and record a coupling vector containing phase synchrony, energy correlation, and frequency coupling information in combination with the hierarchical model.

[0104] Apply the linear superposition model SM(x,y)=∑s w(s)*FM_s(x,y) on each layer of the feature pyramid. The weight coefficient w(s) decreases as the layer s increases: w(s)=2^(-s) / ∑k 2^(-k), ensuring that lower-level features receive more attention. Normalize SM(x,y) by maximum and minimum values to obtain the saliency map S(x,y). This adaptive weight assignment strategy is particularly important when dealing with multi-scale neural activities. For example, in visual working memory tasks, as the memory load increases, brain activities show hierarchical changes at different time and frequency scales, from the fast gamma waves in the early visual areas to the sustained theta wave activities in the prefrontal cortex. The multi-level feature pyramid can capture these neural dynamics at different scales simultaneously.

[0105] Extract the set of salient regions SRs={SR1,SR2,...,SRQ} from S(x,y). Calculate the core features for each salient region SRq, including the salient center sc(SRq)=(xc,yc) and the region feature vector. The region features not only include basic geometric attributes (area, position, shape, etc.), but also energy distribution features. In language understanding tasks, these salient region features can reflect the organizational characteristics of the language processing network. For example, when processing complex sentences, a stable theta-gamma band coupling will form between the temporal and frontal language areas, and this coupling is manifested as specific spatial distributions and energy patterns in the salient region features.

[0106] Mapping the multi-scale features F(Li) of the above steps to the hierarchical model H is a key step in identifying overlapping regions. For each feature region Li, construct a complete feature vector H(Li)=[T_s(Li),S_s(Li),Sj,V(SRq)], which includes texture features, shape features, level annotations, and the features of the corresponding salient regions. Project H(Li) onto the nodes of the hierarchical model, and calculate the projection energy E(Li,vj_n)=∑l|<H(Li),F(vj_n)>|² for each node vj_n∈Vj in the j-th layer. This projection mechanism enables the system to analyze the distribution characteristics of features at different levels. In motor imagery tasks, when the subject imagines hand movements, the suppression features of the beta waves in the motor area are projected onto the corresponding level nodes, and the subsequent beta rebound phenomenon is expressed at a higher level, forming a complete representation of the motor imagery process.

[0107] Based on the projection energy, calculate the time-frequency distribution score of the feature region at each level j: score(Li,j)=∑n∈Vj E(Li,vj_n)*g(fc(Li),fc(vj_n))*g(tc(Li),tc(vj_n)), where g(x,y)=exp(-(x-y)² / 2σ²) is the Gaussian kernel function, and fc and tc represent the frequency and time coordinates respectively. This way of calculating the score comprehensively considers the distribution of features in the time and frequency dimensions. In the emotion recognition task, when analyzing the neural activities related to fear emotions, the system can simultaneously capture the enhancement of the theta wave activity in the amygdala and its temporal association with the gamma wave in the prefrontal cortex, and these features will obtain corresponding score distributions at different levels.

[0108] For any two feature regions Li and Lj, when their scores at a certain level k both exceed the threshold θ and satisfy the proximity condition of the time-frequency position, it is considered that they overlap. Specifically, it is necessary to simultaneously satisfy score(Li,k)>θ, score(Lj,k)>θ, and the time-frequency distance constraints |fc(Li)-fc(Lj)|<δf and |tc(Li)-tc(Lj)|<δt. The overlap intensity is calculated by overlap(Li,Lj,k)=(score(Li,k)*score(Lj,k))^(1 / 2)*exp(-d(Li,Lj) / λ), where d(Li,Lj) is the Euclidean distance between Li and Lj in the time-frequency plane, and λ is the attenuation parameter. In the cross-modal integration task, this overlap detection mechanism can accurately identify the neural synchronization phenomenon in the process of visual and auditory information integration. For example, when visual and auditory stimuli appear synchronously, a significant gamma band coupling will form between the visual cortex and the auditory cortex, and this spatio-temporal association will be marked as a high-intensity overlap region by the system.

[0109] Each element in the final obtained overlap region set O is a complete overlap description: O={(Li,Lj,k,overlap,H(Li),H(Lj),Vk,Coupling)}. Among them, Coupling is a vector describing the functional coupling characteristics, including information such as phase synchronization, energy correlation, and frequency coupling. This detailed overlap representation provides an important basis for understanding the dynamic organization of the neural network. For example, in the decision-making task, when the subject needs to make a choice among multiple options, the system can detect the multiple band couplings between the prefrontal cortex and the parietal lobe: there is a significant modulation relationship between the phase of the theta wave and the amplitude of the high-frequency gamma wave, and this spatio-temporal coupling pattern reflects the information integration process in the decision-making network. By analyzing the distribution characteristics and time evolution laws of the overlap regions, the neural information processing mechanism in the cognitive process can be deeply understood.

[0110] Use the landmark elements in the above steps as patterns, and the overlapping region information identified in the above steps as the content corresponding to the patterns to construct a pattern set for the target theme (spatiotemporal correlation).

[0111] Use the set of labeled regions L = {L1, L2,..., LP} obtained in the above steps as basic pattern nodes, and the set of overlapping regions O = {(Li, Lj, k, overlap, H(Li), H(Lj), Vk, Coupling)} obtained in the above steps as pattern relationship information to construct a pattern set representing spatiotemporal correlation. For each labeled region Li, construct its corresponding pattern node pattern_i = (boundingBox(Li), type(Li), freq(Li), time(Li)). Each attribute of the pattern node has a clear physical meaning: boundingBox defines the exact position and range of the region in the time-frequency plane; type labels the type of neuroelectrophysiological phenomenon corresponding to the region, such as event-related potential (ERP), evoked potential (EP), etc.; freq = [fl, fh] details the frequency range, where fl and fh are the lower and upper limit frequencies respectively; time = [ts, te] describes the time range, and ts and te represent the start and end time points respectively. In the sustained attention task, when the subject needs to maintain an attention state for a long time, a stable theta wave rhythm pattern will form in the prefrontal region, which is accurately expressed as freq = [4Hz, 8Hz] in pattern_i, and time may span several seconds to several minutes, accompanied by the suppression feature of the parietal alpha wave (freq = [8Hz, 13Hz]).

[0112] The pattern relationships constructed on this basis are more richly defined. For the overlapping region pairs Li and Lj, their relationship relation(Li,Lj)=(k,overlap,Coupling) contains information in multiple dimensions. First, k represents the hierarchical level at which the relationship occurs, reflecting the scale characteristics of information interaction. Overlap not only calculates the spatial overlap degree, but also considers the temporal overlap rate and the frequency overlap bandwidth. Coupling is a multi-dimensional vector that describes multiple aspects of functional coupling: phase_sync measures the degree of temporal phase locking of oscillatory activities, energy_corr reflects the co-variation of signal intensities, and freq_coupling characterizes the modulation relationship across frequency bands. In the working memory task, this fine-grained relationship description can fully characterize the dynamic characteristics of the information maintenance network. For example, as the memory load increases, the phase coupling between the parietal alpha wave and the prefrontal theta wave gradually strengthens, phase_sync shows a stable phase difference of π / 4, energy_corr shows a significant positive correlation, and freq_coupling reflects the modulation relationship between the theta wave envelope and the alpha wave amplitude.

[0113] The construction of the pattern graph PS(Patterns,Relations) introduces an innovative network metric mechanism. Each component in the node importance index Importance(pattern_i)=α*Energy(Li)+β*Centrality(Li)+γ*Connectivity(Li) is carefully designed: Energy calculates the total energy contribution of the signal using time-frequency integration, Centrality evaluates the network centrality of the node based on the geodesic distance, and Connectivity calculates the connection strength through the normalized adjacency matrix. The setting of the weight coefficients (α,β,γ) takes into account the physical characteristics of the signal and the network topology. Usually, α∈[0.3,0.5] emphasizes the energy contribution, β∈[0.2,0.4] focuses on the positional importance, and γ∈[0.2,0.4] emphasizes the network integration degree. In the emotion recognition task, this multi-dimensional importance assessment can accurately identify the core processing units. When processing fear stimuli, the theta wave pattern in the amygdala region not only shows a significant energy enhancement (Energy≈0.85), but also forms stable connections with multiple brain regions including the prefrontal and temporal lobes (Connectivity≈0.78), and is at the core of the network (Centrality≈0.72), finally obtaining a high comprehensive importance score.

[0114] The hierarchical structure of pattern relations H_relation={R1,R2,...,RM} is organized in a bottom-up manner, and each level corresponds to a specific scale of information processing. R1 and R2 mainly contain local neural synchronization features, such as high-frequency synchronized activities in primary sensory areas. These relations usually have strong spatial limitations and short time duration. R3 and R4 correspond to the coordination patterns of functional subnetworks, such as the loop activity of the frontoparietal network in attention tasks. These relations often span multiple brain regions and last for a long time. R5 and higher levels reflect the integration characteristics of large-scale cognitive networks, involving the coordinated work of multiple functional systems. In language comprehension tasks, this hierarchical structure clearly shows the progressive process of information processing: the bottom-level relations capture the fast gamma wave activity (60-80Hz) in the temporal auditory area, reflecting the encoding of speech features; the middle-level relations show the theta wave synchronization (4-8Hz) of the frontoparietal loop, corresponding to syntactic analysis and working memory maintenance; the high-level relations involve the beta wave network (13-30Hz) in a wide range of brain areas, reflecting the integration process of deep semantics.

[0115] The evolutionary characteristics of the time dimension are expressed by E(t)={(pattern_i, pattern_j, relation_ij(t))|t∈T}, where relation_ij(t) is a time-varying function that describes the dynamic changes of pattern relations over time. This description not only includes the changing curve of the overlap intensity, but also records the dynamic evolution of the phase relationship and the time-varying characteristics of functional coupling. In the visual search task, this dynamic description can fully record the temporal characteristics of information processing: first, the instantaneous activation of the gamma wave (60-80Hz) in the occipital visual area lasts for about 100ms; followed by the continuous activity of the beta wave (13-30Hz) in the intraparietal sulcus area of the parietal lobe, which manifests as a stable oscillation of 200-500ms; finally, the prefrontal area is recruited to form a stable α-θ coupling network after 500ms. E(t) not only records the dominant patterns at these key time points, but also accurately describes the evolutionary relationship between patterns, including energy transfer paths, phase synchronization dynamics, and network topology changes.

[0116] The finally constructed pattern set PS = (P, R, H, I, E) provides a multi-level and multi-scale spatio-temporal correlation expression framework. P stores all pattern nodes and their complete attributes; R contains detailed descriptions of all pattern relationships; H maintains the hierarchical organizational structure of the relationships; I records the importance indicators of the nodes; E describes the time evolution characteristics. In the analysis of decision-making tasks, this structured representation can comprehensively depict the complete process from perception to decision-making: specific pattern nodes reflect the process of evidence accumulation, such as the progressive enhancement of parietal beta wave activity; other nodes correspond to value evaluation, such as the phase encoding of orbital frontal theta waves; the dynamic associations between nodes describe the temporal pattern of information integration, from early perceptual encoding to late decision-making formation.

[0117] Step S105, construct a pattern set of a preset target theme with the iconic element as the pattern and the overlapping region information as the pattern content; use the pattern set to train a preset optimization model for extracting visual spatio-temporal correlation features in combination with a two-dimensional grayscale image, and complete the spatio-temporal correlation feature analysis of the electroencephalogram signal data.

[0118] Specifically, use the pattern set PS constructed in the above steps to train an optimization model to obtain the target model for spatio-temporal correlation analysis and its output; combine the model output with the original images {I1, I2,..., IM} in the above steps, and apply image processing algorithms (such as segmentation, texture analysis, etc.) to extract and visualize spatio-temporal correlation features.

[0119] In some embodiments, the constructing a pattern set of a preset target theme with the iconic element as the pattern and the overlapping region information as the pattern content includes: constructing pattern nodes including bounding box positions, neuroelectrophysiological types, frequency ranges, and time ranges according to the iconic element; converting the overlapping region information into relationship edges including overlap degrees, coupling characteristics, and hierarchical levels; forming a pattern set representing spatio-temporal correlations according to the pattern nodes and pattern edges.

[0120] Construct a graph neural network model based on a pattern set. This model adopts a multi-layer graph attention network structure to achieve automatic learning and analysis of spatio-temporal correlation patterns. For each node pattern_i in the pattern set, construct an initial node feature vector X_v = [boundingBox, type_embedding, freq, time, importance], where type_embedding is the vector representation obtained by mapping the category label through an embedding layer. For the pattern relationship relation(Li,Lj), construct an edge feature vector X_e = [overlap, coupling, hierarchy_level]. This feature representation method is of great significance in cognitive state analysis. For example, in a visual working memory task, when a subject needs to remember multiple visual items, specific spatio-temporal correlation patterns will be formed between different brain regions, and these patterns can be fully expressed through node and edge features.

[0121] In some embodiments, training a preset optimization model using the pattern set for extracting visual spatio-temporal correlation features in combination with a two-dimensional grayscale image includes: constructing a multi-layer graph attention network, where the node features of the multi-layer graph attention network include type embedding vectors, frequency, and time information, and the edge features include coupling characteristics and hierarchy levels; calculating the inter-node association weights of the multi-layer graph attention network through an attention mechanism for inputting the two-dimensional grayscale image to output visual spatio-temporal correlation features including an association intensity distribution map and a temporal evolution sequence.

[0122] In the graph attention network, the update and aggregation of node features are achieved through an attention mechanism. For the central node v, calculate its attention coefficient with the neighbor node u: α_uv = f_att([W*h_u || W*h_v || X_e]), and normalize it through the softmax function: α'_uv = exp(α_uv) / ∑k∈N_v exp(α_vk). The node feature update adopts a weighted aggregation method: h'_v = σ(∑u∈N_v α'_uv*W*h_u). This attention-based feature aggregation mechanism can adaptively capture the dynamic features in the cognitive process. In a sustained attention task, when a subject needs to switch attention between multiple targets, the information flow between different brain regions will change dynamically, and the attention mechanism can automatically adjust the importance of different connections according to the current task state.

[0123] The training objective of the model is designed as a combination of multiple loss terms: L_total = L_node + λ1*L_edge + λ2*L_structure, where L_node is responsible for predicting the type and importance of the pattern nodes, L_edge is used to predict the strength of the pattern relationships, and L_structure ensures that the hierarchical structure of the pattern set is maintained. The training process adopts a learning rate schedule with cosine annealing and optimizes the model parameters through batch gradient descent. In the emotion recognition task, this multi-objective training strategy can simultaneously learn the local features and global patterns of the emotional state. For example, when analyzing the happy emotion, the model can not only identify the local feature of enhanced prefrontal gamma band activity but also capture the large-scale coordination pattern between the prefrontal and limbic systems.

[0124] The training objective of the model is designed as a combination of multiple loss terms: L_total = L_node + λ1*L_edge + λ2*L_structure, where L_node is responsible for predicting the type and importance of the pattern nodes, L_edge is used to predict the strength of the pattern relationships, and L_structure ensures that the hierarchical structure of the pattern set is maintained. The training process adopts a learning rate schedule with cosine annealing and optimizes the model parameters through batch gradient descent. In the emotion recognition task, this multi-objective training strategy can simultaneously learn the local features and global patterns of the emotional state. For example, when analyzing the happy emotion, the model can not only identify the local feature of enhanced prefrontal gamma band activity but also capture the large-scale coordination pattern between the prefrontal and limbic systems.

[0125] When applying the trained model to new data, first, the predicted pattern associations are obtained, and then this association information is combined with the original time-frequency mapping image. An association strength map is constructed based on the predicted edge weights: S(x,y) = ∑(i,j)∈Ew_ij*K((x,y),(xi,yi)), where w_ij is the predicted edge weight and K is a two-dimensional Gaussian kernel function. In the construction of the association strength map, an adaptive kernel bandwidth selection strategy is adopted, and the bandwidth parameter σ is automatically adjusted according to the scale of the association region: σ = median(d_ij)*β, where d_ij is the feature distance of the association edge and β is the scale factor. On this basis, a multi-layer feature extraction algorithm is designed, including spatial domain features (position distribution, shape description, regional connectivity), temporal domain features (duration, phase relationship, temporal pattern), and frequency domain features (energy distribution, band coupling, spectral coherence). In the data analysis of the language understanding task, when dealing with complex language information, the system can automatically detect and locate the θ-γ band coupling between the temporal auditory region and the frontal language region, and this coupling pattern appears as an active region with a clear boundary and internal structure on the association strength map.

[0126] The final analysis results include three main parts: model prediction results, feature extraction results, and visualization results. The model prediction results include node attributes (type, importance), edge attributes (association strength, functional type), and temporal evolution features; the feature extraction results contain quantitative metrics in the spatial domain, temporal domain, and frequency domain; the visualization results provide distribution maps of association strength, temporal evolution sequences, and multi-scale feature maps. Taking the continuous visual recognition task as an example, when the target appears in the field of view, the system first captures the enhanced activity in the γ band (30 - 50 Hz) of the occipital visual area, and simultaneously detects the synchronous changes in the θ band (4 - 8 Hz) of the prefrontal region. This cross-frequency spatio-temporal coupling is manifested in the output as follows: the model prediction shows that the association strength between the two regions reaches 0.85, with a time delay of 200 ms; the feature extraction results show that the γ-band energy rapidly rises within 100 ms, and the phase locking of the θ band gradually increases between 200 - 400 ms; the visualization results intuitively display this information transfer process from back to front and from high frequency to low frequency through gradient color coding. Such multi-dimensional analysis results completely depict the dynamic characteristics of the neural network during the visual information processing process.

[0127] The provided method has the following beneficial effects:

[0128] 1. An innovative graph neural network-based modeling method is proposed, which can automatically discover and represent the spatio-temporal correlation patterns and their interrelationships contained in EEG signals, overcoming the defect that traditional methods are difficult to capture spatio-temporal detailed structures. This model automatically aggregates node and neighbor node representations through an attention mechanism, makes full use of the topological structure information of the pattern set, and simultaneously optimizes the pattern node classification loss and the associated edge regression loss during the training process, prompting the network to learn the semantic labels of spatio-temporal patterns and their association strengths, thereby obtaining a representation model that can capture spatio-temporal correlations.

[0129] 2. A novel hierarchical density clustering algorithm is proposed, which can automatically divide different EEG state types according to data distribution characteristics, such as resting state, attention state, memory processing state, etc. This algorithm combines the advantages of hierarchical clustering and density clustering, does not require pre-setting the number and shape of clusters, and adaptively discovers and identifies the characteristic patterns of different cognitive states from the data by calculating local density and relative density.

[0130] 3. Combining image processing techniques, innovative methods such as automatic extraction and hierarchical annotation of landmark elements, dynamically determining the optimal hierarchical model, and multi-scale spatio-temporal feature mapping are proposed to prepare for spatio-temporal correlation analysis. Finally, by combining the model output with the original image, an intuitive visualization analysis method is proposed to present the specific distribution map of spatio-temporal correlation features, greatly improving the interpretability and usability of the results.

[0131] To implement the electroencephalogram (EEG) signal spatio-temporal correlation feature analysis method based on a graph neural network corresponding to the above method embodiments, so as to achieve the corresponding functions and technical effects. Refer to Figure 2 , Figure 2 FIG. shows a structural block diagram of an EEG signal spatio-temporal correlation feature analysis device 200 provided by an embodiment of the present application. For ease of description, only the parts related to this embodiment are shown. The EEG signal spatio-temporal correlation feature analysis device 200 provided by the embodiment of the present application includes:

[0132] A signal acquisition module 201, configured to acquire a target EEG signal dataset, and obtain the brain state categories of the EEG signal data corresponding to the target EEG signal dataset through a hierarchical density clustering algorithm;

[0133] A signal extraction module 202, configured to extract signal segments of the EEG data corresponding to each brain state category in a target time period; map the signal segments into two-dimensional grayscale images; extract landmark elements and corresponding levels from the two-dimensional grayscale images;

[0134] A model construction module 203, configured to construct a multi-scale hierarchical model according to the signal segments, and extract multi-scale features reflecting spatio-temporal correlation from the landmark elements;

[0135] A saliency calculation module 204, configured to calculate the salient region of the two-dimensional grayscale image, and map the multi-scale features to the hierarchical model according to the salient region to identify the overlapping regions;

[0136] An analysis completion module 205, configured to construct a pattern set of a preset target theme with the landmark elements as patterns and the overlapping region information as pattern contents; train a preset optimization model with the pattern set, and extract visual spatio-temporal correlation features in combination with the two-dimensional grayscale image to complete the spatio-temporal correlation feature analysis of the EEG signal data.

[0137] The above-mentioned EEG signal spatio-temporal correlation feature analysis device 200 can implement the EEG signal spatio-temporal correlation feature analysis method based on a graph neural network of the above method embodiments. The optional items in the above method embodiments are also applicable to this embodiment, which will not be elaborated here. The remaining content of the embodiment of the present application can refer to the content of the above method embodiments, and will not be repeated in this embodiment.

[0138] Figure 3 FIG. is a schematic structural diagram of a computer device provided by an embodiment of the present application. As Figure 3 shown, the computer device 3 of this embodiment includes: at least one processor 30 ( Figure 3(only one is shown), a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above method embodiments are implemented.

[0139] The computer device 3 may be a computing device such as a smart phone, a tablet computer, a desktop computer, and a cloud server. The computer device may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art can understand that Figure 3 merely examples of the computer device 3, which do not constitute a limitation on the computer device 3, may include more or fewer components than those shown in the figure, or combine some components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0140] The so-called processor 30 may be a central processing unit (CPU), and the processor 30 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0141] In some embodiments, the memory 31 may be an internal storage unit of the computer device 3, such as the hard disk or memory of the computer device 3. In other embodiments, the memory 31 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 3. Further, the memory 31 may also include both the internal storage unit and the external storage device of the computer device 3. The memory 31 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0142] In addition, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0143] An embodiment of the present application provides a computer program product. When the computer program product runs on a computer device, the computer device is caused to execute the steps in each of the above method embodiments.

[0144] In several embodiments provided by the present application, it can be understood that each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the block may occur in a different order from that marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.

[0145] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0146] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present application. It should be understood that the above are only specific embodiments of the present application and are not used to limit the protection scope of the present application. In particular, for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing spatio-temporal correlation features of electroencephalogram signals based on graph neural networks, characterized in that Including: Collecting a target electroencephalogram (EEG) signal dataset, and obtaining the brain state categories of the EEG signal data corresponding to the target EEG signal dataset through a hierarchical density clustering algorithm, including: calculating a distance matrix between EEG signal data samples to determine a cut-off distance; screening candidate cluster centers based on the local density and relative distance of the EEG signal data; determining the final number of clusters according to the candidate cluster centers through a density distribution histogram, constructing a hierarchical tree according to the final number of clusters for cluster merging, and retaining a stable cluster structure, and outputting brain state categories including a resting state, an attention state, and a memory processing state; Extracting signal segments of the EEG data corresponding to each brain state category in a target time period; mapping the signal segments into two-dimensional grayscale images; extracting landmark elements and corresponding levels from the two-dimensional grayscale images; Constructing a multi-scale hierarchical model according to the signal segments, and extracting multi-scale features reflecting spatio-temporal correlation in the landmark elements; Calculating the significant region of the two-dimensional grayscale image for mapping the multi-scale features to the hierarchical model according to the significant region to identify overlapping regions; Constructing a pattern set of a preset target theme with the landmark elements as patterns and the overlapping region information as pattern contents; training a preset optimization model with the pattern set for combining two-dimensional grayscale images to extract visual spatio-temporal correlation features, and completing the spatio-temporal correlation feature analysis of the EEG signal data.

2. The method according to claim 1, wherein The mapping of the signal segments into two-dimensional grayscale images includes: Performing time-frequency analysis on the signal segments to generate a time-frequency energy distribution diagram, in which the time-domain sampling points are mapped to the horizontal axis and the frequency components are mapped to the vertical axis in the time-frequency energy distribution diagram; Normalizing the energy amplitude to a grayscale value and then generating a two-dimensional grayscale image with a preset number of pixels.

3. The method according to claim 1, wherein The extracting of landmark elements and corresponding levels from the two-dimensional grayscale images includes: Performing frequency-domain residual significance detection on the two-dimensional grayscale image to generate a significance map; Setting three-level significance thresholds based on the mean and standard deviation of the significance map; labeling the regions with significant values higher than the first threshold as first-level elements, those between the first and second thresholds as second-level elements, and those between the second and third thresholds as third-level elements, forming a set of hierarchically labeled regions.

4. The method according to claim 1, characterized in that The constructing of a multi-scale hierarchical model according to the signal segments includes: Constructing a hierarchical model including an effective node set, an inter-layer connection set, and a node feature set according to the signal segments; Dynamically determining the number of nodes in each layer of the hierarchical model through weighted reconstruction error and node number penalty terms.

5. The method according to claim 1, wherein The extracting of multi-scale features reflecting spatio-temporal correlation in the landmark elements includes: Calculating the gray-level co-occurrence matrix of the landmark elements based on multiple preset displacement directions for extracting texture features, where the texture features include contrast, correlation, energy, and homogeneity; Extracting the shape features of the landmark elements, where the shape features include perimeter, area, and invariant moments; Concatenating the texture features and the shape features to form the multi-scale features.

6. The method according to claim 1, characterized in that, The calculating of the significant region of the two-dimensional grayscale image includes: Extracting the significant region by using the spectral residual method, and calculating the overlapping area ratio corresponding to multiple significant regions; When the overlapping degree corresponding to the overlapping area ratio exceeds a set threshold, a mapping relationship is established, and a coupling vector containing phase synchronization, energy correlation, and frequency coupling information is recorded in combination with the hierarchical model.

7. The method according to claim 1, wherein The mode set of the preset target theme constructed according to the iconic element as the mode and the overlapping area information as the mode content includes: Construct a mode node containing the boundary box position, neuroelectrophysiological type, frequency range, and time range according to the iconic element; Convert the overlapping area information into relationship edges containing the overlapping degree, coupling characteristics, and hierarchical levels; Form a mode set representing spatio-temporal correlation according to the mode node and the mode edge.

8. The method according to claim 1, characterized in that, Using the mode set to train a preset optimization model for extracting visual spatio-temporal correlation features in combination with a two-dimensional grayscale image, including: Construct a multi-layer graph attention network, where the node features of the multi-layer graph attention network include type embedding vectors, frequency, and time information, and the edge features include coupling characteristics and hierarchical levels; Calculate the inter-node correlation weights of the multi-layer graph attention network through the attention mechanism, and use them to input the two-dimensional grayscale image to output visual spatio-temporal correlation features including the correlation intensity distribution map and the time series evolution sequence.

9. A computer device, characterized in that, It includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the method according to any one of claims 1 to 8 when executing the computer program.

Images