Neural function feature recognition method and system based on multi-modal signal processing

Abstract

The invention discloses a neural function feature recognition method and system based on multi-modal signal processing, and relates to the technical field of multi-modal signal processing, and the method comprises the following steps: carrying out the preprocessing of an original behavior data stream and electroencephalogram data of a user, obtaining multi-modal data segments, extracting behavioral features and electroencephalogram micro-state features, and obtaining a multi-modal data segment; performing adaptive weighted fusion to obtain neural function feature vectors, associating the neural function feature vectors with corresponding metadata, adding labels, storing the neural function feature vectors in a database, establishing an index structure to obtain a neural function feature database, and when the feature vectors to be stored in the database are obtained, judging a new different mode through a preset similarity threshold value. And automatically storing the new different patterns into a database and updating an index structure, performing similarity retrieval on the to-be-recognized neural function feature vector and the database, and outputting a recognition result if the recognition confidence reaches a preset recognition confidence threshold according to the recognition confidence of label distribution in a retrieval result. Otherwise, transferring the query feature vector into a dynamic updating process.

Description

Technical Field

[0001] This invention relates to the field of multimodal signal processing technology, and in particular to a method and system for identifying neural functional features based on multimodal signal processing. Background Technology

[0002] In recent years, multimodal signal processing technology has received widespread attention in the field of neural function feature recognition. This technology can comprehensively characterize an individual's neural function status from two dimensions: task evoked and resting baseline, by simultaneously collecting and fusing behavioral data and EEG signals. It has shown unique advantages in key applications such as early warning of cognitive impairment, evaluation of neural rehabilitation effects, and adaptive control of brain-computer interfaces. Compared with single-modal analysis, multimodal fusion can provide a more comprehensive and stable representation of neural function and has become an important development direction in the fields of intelligent medicine and cognitive neural engineering.

[0003] However, existing technologies still have significant shortcomings in practical applications. Most methods use simple splicing or fixed-weight fusion to combine behavioral and EEG features, ignoring the dynamic fluctuations in the quality of data from different modalities. For example, decreased user attention can lead to increased noise in behavioral data, or motion artifacts can be mixed into EEG signals, reducing the reliability of EEG features. Fixed-weight fusion methods are difficult to adapt to such real-time changes, resulting in insufficient robustness of feature representation. Traditional recognition relies on static databases and single similarity matching, lacking an automatic discovery and storage mechanism for novel neural functional patterns. Furthermore, it cannot provide effective feedback and iterative updates when the retrieval results are uncertain, limiting its dynamic adaptability to individual differences and disease evolution. Summary of the Invention

[0004] The technical problem solved by this invention is that existing technologies still have significant shortcomings in practical applications. Most methods use simple splicing or fixed-weight fusion to combine behavioral features and EEG features, ignoring the dynamic fluctuations in the quality of data from different modalities. For example, decreased user attention leads to increased noise in behavioral data, or motion artifacts mixed into EEG signals reduce the reliability of EEG features. Fixed-weight fusion methods are difficult to adapt to such real-time changes, resulting in insufficient robustness of feature expression. Traditional recognition relies on static databases and single similarity matching, lacking an automatic discovery and storage mechanism for novel neural functional patterns. Furthermore, it cannot provide effective feedback and iterative updates when the retrieval results are uncertain, limiting its dynamic adaptability to individual differences and disease evolution.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a neural functional feature recognition method based on multimodal signal processing, comprising the following steps: Step S1: Preprocess the user's raw behavioral data stream and EEG data to obtain multimodal data segments; Step S2: Extract behavioral features and EEG microstate features from the multimodal data segments, and perform adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector; Step S3: After associating the neural function feature vector with the corresponding metadata and adding tags, store it in the database and establish an index structure to obtain the neural function feature database. When obtaining the feature vector to be stored in the database, determine the novel pattern through a preset similarity threshold, automatically store the novel pattern in the database and update the index structure. Step S4: Perform a similarity search between the neural function feature vector to be identified and the database. Based on the identification confidence of the label distribution in the search results, if the identification confidence reaches the preset identification confidence threshold, the identification result is output; otherwise, the query feature vector is transferred to the dynamic update process.

[0006] As a preferred embodiment of the neural functional feature recognition method based on multimodal signal processing described in this invention, step S1 includes steps S101, S102, S103 and S104. Step S101: When the user performs a preset cognitive task paradigm, collect the user's raw behavioral data stream and EEG data; The collection of raw behavioral data streams specifically includes: Present a visual sequence to the user and record the user's behavioral response to the visual sequence to obtain the raw behavioral data stream; The raw behavioral data stream includes timestamps, stimulus event markers, and response records; The collection of EEG data specifically includes: During the user's task execution and resting state, electrical activity signals are collected using a multi-channel EEG device to obtain raw multi-channel EEG time-series signals.

[0007] As a preferred embodiment of the neural functional feature recognition method based on multimodal signal processing described in this invention, step S102, preprocessing includes: The original behavioral data stream and the original multi-channel EEG time series signal are aligned based on a unified timestamp to obtain the aligned original behavioral data stream and the aligned original multi-channel EEG time series signal. Based on the preset time markers, the aligned original multi-channel EEG time series signal is divided into multiple time segments, and corresponding cognitive state labels are added to each time segment to obtain multimodal data segments; Step S103, the preprocessing also includes: The EEG signals in the multimodal data segments are filtered and rereferenced to obtain filtered and rereferenced EEG signals, specifically including: The filtering process includes using a high-pass filter to remove low-frequency drift below 0.5Hz and using a low-pass filter to remove high-frequency noise above 50Hz. The filtered EEG signal is rereferenced to obtain an average reference EEG signal, and the average reference EEG signal is downsampled to obtain a filtered rereference EEG signal. Step S104: The filtered rereference EEG signal is decomposed into multiple independent source components using the independent component analysis algorithm. Based on the topographic distribution characteristics of each independent source component, noise components corresponding to eye movement, blinking, and electromyographic physiological activities are removed to obtain an artifact-free EEG signal.

[0008] As a preferred embodiment of the neural function feature recognition method based on multimodal signal processing described in this invention, step S2 includes steps S201, S202 and S203; Step S201, extracting behavioral features from multimodal data segments, specifically including: Obtain behavioral data streams from multimodal data segments. Based on stimulus event labels and response records in the behavioral data streams, calculate the average accuracy and average reaction time of users within each time segment. Construct behavioral feature vectors using the average accuracy and average reaction time. Step S202 involves extracting EEG microstate features from multimodal data segments, specifically including: EEG signals with resting state labels are obtained from multimodal data segments. Based on the potential values ​​of each electrode in the EEG signal at each time point, the standard deviation of the potential values ​​of all electrodes is calculated as the global field strength value at that time point. Construct a global field strength time series based on the global field strength value; Detect local maxima in the global field strength time series and take the time corresponding to the local maxima as the peak time of the global field strength; The topographic map at the peak of the global field strength is spatially clustered using the K-means clustering algorithm to generate multiple micro-state templates; Calculate the cosine similarity between the topographic map and each microstate template of the EEG signal at each time point, take the category of the microstate template with the highest similarity as the microstate category at that time point, obtain the microstate category corresponding to each time point, and combine the microstate categories in chronological order to generate a microstate time series. Based on the microstate time series, calculate the dynamic characteristics of each type of microstate; Dynamic characteristics include average duration, frequency per second, time coverage ratio, and transition probability between different microstates; Multi-scale coarsening processing of micro-state time series is performed, specifically including: Multiple scale factors are set. For each scale factor, the microstate time series is divided into multiple non-overlapping time windows. The length of each window is equal to the current scale factor. The mode of the microstate category in each window is calculated. The modes calculated for each window are arranged in window order to obtain the coarse-grained microstate time series at this scale. Based on the coarse-grained microstate time series at each scale, the number of transitions from one type of microstate to another between adjacent time points is counted to obtain the transition number matrix corresponding to each scale. Calculate the row sum of each element in the transition number matrix; Divide each element in the row by the corresponding row sum and value to obtain the transition probability matrix for each scale. Calculate the transition entropy corresponding to each scale based on the transition probability matrix of each scale; The dynamic features and the transformation entropy corresponding to each scale are used as the EEG microstate feature vectors.

[0009] As a preferred embodiment of the neural function feature recognition method based on multimodal signal processing described in this invention, step S203, fusing the behavioral feature vector with the EEG microstate feature vector to obtain the neural function feature vector, specifically includes: Obtain the user's reaction time series within each time segment, calculate the ratio of the standard deviation to the mean of the reaction time series, and obtain the reaction time coefficient of variation; The coefficient of variation during reaction time is normalized to obtain a normalized value. The difference between 1 and the normalized value is calculated to obtain the first confidence level. Acquire artifact-free EEG signals, calculate the ratio of signal power to noise power of the EEG signals to obtain the signal-to-noise ratio (SNR), normalize the SNR to obtain the second confidence level; Based on the first and second confidence levels, the behavioral feature vector and the EEG microstate feature vector are weighted and fused to obtain the neural function feature vector.

[0010] As a preferred embodiment of the neural functional feature recognition method based on multimodal signal processing described in this invention, step S3 includes steps S301, S302 and S303; Step S301: Obtain metadata corresponding to the neural function feature vector. The metadata includes paradigm information, user information, and cognitive state labels. The neural function feature vectors are associated with the corresponding metadata, and an index label is added to each feature vector to obtain feature vector records with metadata labels. Metadata includes paradigm information, user information, and cognitive state tags; Step S302: Store multiple feature vector records with metadata tags into the database and establish an index structure for the feature vectors to obtain the neural function feature database.

[0011] As a preferred embodiment of the neural function feature recognition method based on multimodal signal processing described in this invention, step S303 involves obtaining the feature vector to be added to the database and calculating the similarity between the feature vector to be added to the database and the feature vector in the database. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is less than the preset similarity threshold, the feature vector to be added to the database is determined to be a novel pattern. After adding the cognitive state label to be labeled to the feature vector to be added to the database, it is stored in the database and the index structure is updated. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is greater than or equal to the preset similarity threshold, then the cognitive state label corresponding to the feature vector with the highest similarity will be used as the recognition result and output after being marked as low reliability.

[0012] As a preferred embodiment of the neural functional feature recognition method based on multimodal signal processing described in this invention, step S4 includes steps S401, S402 and S403; Step S401: Obtain the neural function feature vector to be identified as the query feature vector; The neural function feature vector to be identified is obtained using the same method as in steps S201 to S203; Step S402: Input the query feature vector into the neural function feature database, perform similarity retrieval using the index structure of the database, calculate the similarity between the query feature vector and each feature vector in the database, and return the K feature vector records with the highest similarity.

[0013] As a preferred embodiment of the neural function feature recognition method based on multimodal signal processing described in this invention, in step S403, the distribution of cognitive state labels corresponding to K feature vector records is statistically analyzed, and the proportion of the label with the highest votes is calculated as the recognition confidence level. If the recognition confidence is greater than or equal to the preset recognition confidence threshold, the label with the highest vote will be output as the neural function state corresponding to the query feature vector. If the recognition confidence level is less than the preset recognition confidence level threshold, the query feature vector will be used as the feature vector to be added to the database, and step S303 will be executed to enter the dynamic update process.

[0014] A neural functional feature recognition system based on multimodal signal processing includes a processing module, a fusion module, a decision module, and a recognition module; The processing module preprocesses the user's raw behavioral data stream and EEG data to obtain multimodal data segments; The fusion module extracts behavioral features and EEG microstate features from multimodal data segments, and performs adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector. The judgment module associates neural function feature vectors with corresponding metadata and adds tags, stores them in the database and establishes an index structure to obtain a neural function feature database. When obtaining feature vectors to be stored in the database, it judges novel patterns by a preset similarity threshold, automatically stores novel patterns in the database and updates the index structure. The recognition module performs a similarity search between the neural function feature vector to be recognized and the database. Based on the recognition confidence of the label distribution in the search results, if the recognition confidence reaches the preset recognition confidence threshold, the recognition result is output; otherwise, the query feature vector is transferred to the dynamic update process.

[0015] The beneficial effects of this invention are as follows: Addressing the shortcomings of existing neurofunctional recognition technologies, such as fixed multimodal fusion weights, static and closed databases, and a lack of confidence assessment, this solution proposes an adaptive weighted fusion strategy based on reaction time coefficient of variation and signal-to-noise ratio. This overcomes the problem of feature distortion in traditional methods when behavioral or EEG signal quality fluctuates, significantly improving the robustness and individual adaptability of neurofunctional feature vectors. Simultaneously, it constructs a dynamic database supporting automatic novel pattern identification and index updates, and introduces a recognition and judgment mechanism based on label distribution confidence. This solves the problems of existing technologies being unable to automatically discover novel patterns and having high uncertainty in single similarity matching. This enables the solution to possess self-learning capabilities while effectively reducing the risk of misjudgment, achieving adaptive fusion of multimodal features and dynamic closed-loop updates for recognition. While ensuring recognition accuracy, it significantly enhances the generalization ability to individual differences and clinical evolution. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a neural functional feature recognition method based on multimodal signal processing, provided as an embodiment of the present invention.

[0017] Figure 2 This is a basic flowchart of a neural function feature recognition system based on multimodal signal processing, provided as an embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0019] Example 1, referring to Figure 1 As an embodiment of the present invention, a neural functional feature recognition method based on multimodal signal processing is provided, comprising the following steps: Step S1: Preprocess the user's raw behavioral data stream and EEG data to obtain multimodal data segments; Step S2: Extract behavioral features and EEG microstate features from the multimodal data segments, and perform adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector; Step S3: After associating the neural function feature vector with the corresponding metadata and adding tags, store it in the database and establish an index structure to obtain the neural function feature database. When obtaining the feature vector to be stored in the database, determine the novel pattern through a preset similarity threshold, automatically store the novel pattern in the database and update the index structure. Step S4: Perform a similarity search between the neural function feature vector to be identified and the database. Based on the identification confidence of the label distribution in the search results, if the identification confidence reaches the preset identification confidence threshold, the identification result is output; otherwise, the query feature vector is transferred to the dynamic update process.

[0020] In one embodiment, step S1 establishes multimodal data segments by aligning, segmenting, filtering, rereferencing, and removing artifacts from independent component analysis of the original behavioral data stream and EEG data based on a unified timestamp, providing a temporally consistent and signal-clean data foundation for subsequent feature extraction; step S2 addresses the deficiency of fixed multimodal fusion weights in existing technologies by extracting behavioral features and EEG microstate features from the multimodal data segments, and performs adaptive weighted fusion based on reaction time coefficient of variation and signal-to-noise ratio to obtain neural function feature vectors, effectively solving the problem of feature distortion when the quality of data from different modalities fluctuates, and significantly improving the robustness of feature expression; step S3 forms neural function feature vectors by associating neural function feature vectors with metadata and establishing an index structure. The system utilizes a functional feature database and, when acquiring feature vectors to be added to the database, uses a preset similarity threshold to determine novel patterns. Novel patterns are automatically stored in the database, and the index structure is updated, enabling dynamic expansion and self-learning capabilities of the database. Step S4 performs a similarity search between the neural function feature vectors to be identified and the database. Based on the recognition confidence of the label distribution in the search results, the system decides whether to output the recognition result or proceed to the dynamic update process. This overcomes the limitations of high uncertainty in traditional single similarity matching and effectively reduces the risk of misjudgment. The entire architecture, through the synergistic effect of multimodal adaptive fusion and dynamic closed-loop update mechanism, constructs an accurate, self-learning, and evolvable recognition of neural function features. While ensuring recognition accuracy, it significantly enhances the generalization ability to individual differences and clinical evolution.

[0021] Step S1 includes steps S101, S102, S103 and S104; Step S101: When the user performs a preset cognitive task paradigm, collect the user's raw behavioral data stream and EEG data; The collection of raw behavioral data streams specifically includes: Present a visual sequence to the user and record the user's behavioral response to the visual sequence to obtain the raw behavioral data stream; The raw behavioral data stream includes timestamps, stimulus event markers, and response records; The collection of EEG data specifically includes: During the user's task execution and resting state, electrical activity signals are collected using a multi-channel EEG device to obtain raw multi-channel EEG time-series signals.

[0022] Step S102, preprocessing includes: The original behavioral data stream and the original multi-channel EEG time series signal are aligned based on a unified timestamp to obtain the aligned original behavioral data stream and the aligned original multi-channel EEG time series signal. Based on the preset time markers, the aligned original multi-channel EEG time series signal is divided into multiple time segments, and corresponding cognitive state labels are added to each time segment to obtain multimodal data segments; Step S103, the preprocessing also includes: The EEG signals in the multimodal data segments are filtered and rereferenced to obtain filtered and rereferenced EEG signals, specifically including: The filtering process includes using a high-pass filter to remove low-frequency drift below 0.5Hz and using a low-pass filter to remove high-frequency noise above 50Hz. The filtered EEG signal is rereferenced to obtain an average reference EEG signal, and the average reference EEG signal is downsampled to obtain a filtered rereference EEG signal. Step S104: The filtered rereference EEG signal is decomposed into multiple independent source components using the independent component analysis algorithm. Based on the topographic distribution characteristics of each independent source component, noise components corresponding to eye movement, blinking, and electromyographic physiological activities are removed to obtain an artifact-free EEG signal.

[0023] In one embodiment, step S101 involves synchronously collecting behavioral and EEG data through a standardized cognitive task paradigm and a resting state, providing a temporally aligned and cognitively labeled raw data foundation for subsequent multimodal fusion. Specifically, step S101 involves collecting raw behavioral data streams and EEG data when the user performs a preset cognitive task paradigm. The cognitive task paradigm refers to a standardized task used to induce specific cognitive activities. In this embodiment, a visual sequence task is used as an example, specifically presenting a visual sequence to the user and recording behavioral responses to obtain a raw behavioral data stream including timestamps, stimulus event markers, and response records. Simultaneously, during the user's task execution and resting state, a multi-channel EEG device is used to collect electrical activity signals to obtain raw multi-channel EEG time-series signals. The resting state refers to the period during which the user remains awake, fully relaxed, fixates on the crosshair in the center of the screen, avoids blinking and body movements as much as possible, and does not engage in any specific thinking activities. This step ensures the synchronization of the raw behavioral data and EEG data in the temporal dimension through a unified acquisition paradigm, laying the foundation for subsequent precise alignment. Step S102 involves precisely aligning the original heterogeneous data on the time axis and structurally segmenting it according to cognitive task stages, so that each data segment carries a corresponding cognitive state label, thereby constructing multimodal data segments with clear physiological and behavioral semantics. Specifically, step S102 aligns the original behavioral data stream and the original multi-channel EEG time series signal based on a unified timestamp, obtaining the aligned original behavioral data stream and the aligned original multi-channel EEG time series signal. According to the preset time markers, namely the time points of each stimulus presentation time and the start and end times of the resting period in the task paradigm, the aligned original multi-channel EEG time series signal is segmented into multiple time segments, and corresponding cognitive state labels are added to each time segment, such as the task state label corresponding to the visual stimulus presentation time and the resting state label corresponding to the resting period, thereby obtaining multimodal data segments. This step achieves accurate matching and semantic annotation of heterogeneous data on the time axis, providing a clear data foundation for subsequent submodal feature extraction. Step S103 involves removing low-frequency drift, high-frequency noise, and power frequency interference from the EEG signal through filtering, re-reference, and downsampling, and unifying the signal reference standard to improve signal quality and computational efficiency, providing a clean signal input for subsequent artifact removal and feature extraction. Specifically, step S103 filters and re-references the EEG signal in the multimodal data segment to obtain a filtered and re-referenced EEG signal. The filtering includes using a high-pass filter to remove low-frequency drift below 0.5Hz to prevent baseline fluctuations from affecting feature extraction, and using a low-pass filter to remove high-frequency noise above 50Hz to eliminate electromyography and environmental high-frequency interference. The cutoff frequency of Hz is set according to the effective frequency band of EEG, which can preserve the main neural activity bands of Delta, Theta, Alpha, and Beta while effectively suppressing non-neurophysiological noise. Then, the filtered EEG signal is averaged and rereferenced to eliminate global common-mode interference, and the averaged reference EEG signal is downsampled, for example, from the original 1000Hz to 250Hz. This reduces the amount of data while preserving effective neural information, thereby improving the efficiency of subsequent processing, and finally obtains the filtered and rereferenced EEG signal. This step significantly improves the signal-to-noise ratio of the EEG signal and the timeliness of data processing through precise frequency band selection and reference unification. Step S104 involves using Independent Component Analysis (ICA) to decompose the mixed EEG signal into independent source components. Based on the spatial topographic distribution characteristics of each source component, physiological artifacts are automatically identified and removed to obtain a pure neural electrical activity signal, ensuring the accuracy and reliability of subsequent EEG microstate feature extraction. Specifically, step S104 uses an ICA algorithm to decompose the filtered rereference EEG signal into independent source components. According to the topographic distribution characteristics of each independent source component, eye movement artifacts exhibit a smooth gradient from the prefrontal cortex to the occipital lobe, contributing most significantly to prefrontal electrodes such as Fp1 and Fp2. Blink artifacts are concentrated in the central region of the prefrontal cortex, symmetrically distributed, and have the greatest contribution at Fz and... The electrodes at Fpz were the most prominent. EMG artifacts were characterized by high-frequency, focal, high-amplitude distributions in the topographic map, mostly concentrated at electrode locations corresponding to the temporalis or frontalis muscles, such as T7, T8, F7, and F8. ECG artifacts were characterized by synchronous fluctuations throughout the brain in the topographic map without obvious focality. Electrode loosening artifacts were characterized by isolated high-amplitude regions in the topographic map, limited to a single electrode or no more than three adjacent electrodes. After removing noise components corresponding to physiological activities such as eye movements, blinking, EMG, ECG, and electrode loosening, artifact-free EEG signals were obtained. This step, through precise artifact identification and removal based on topographic map features, effectively removed non-neurogenic interference, providing a high-quality signal foundation for subsequent extraction of micro-state features reflecting the dynamics of the real brain network. Step S1 constructs a complete preprocessing chain from data acquisition and alignment, task segmentation, signal purification to artifact removal, thereby constructing a multimodal data segmentation that is temporally aligned, semantically clear, and signal-pure. This provides reliable data input for the accurate extraction of behavioral features and EEG microstate features in subsequent steps, ensuring the accuracy and robustness of neural function feature recognition.

[0024] Step S2 includes steps S201, S202 and S203; Step S201, extracting behavioral features from multimodal data segments, specifically including: Obtain behavioral data streams from multimodal data segments. Based on stimulus event labels and response records in the behavioral data streams, calculate the average accuracy and average reaction time of users within each time segment. Construct behavioral feature vectors using the average accuracy and average reaction time. Step S202 involves extracting EEG microstate features from multimodal data segments, specifically including: EEG signals with resting state labels are obtained from multimodal data segments. Based on the potential values ​​of each electrode in the EEG signal at each time point, the standard deviation of the potential values ​​of all electrodes is calculated as the global field strength value at that time point. Construct a global field strength time series based on the global field strength value; Detect local maxima in the global field strength time series and take the time corresponding to the local maxima as the peak time of the global field strength; The topographic map at the peak of the global field strength is spatially clustered using the K-means clustering algorithm to generate multiple micro-state templates; Calculate the cosine similarity between the topographic map and each microstate template of the EEG signal at each time point, take the category of the microstate template with the highest similarity as the microstate category at that time point, obtain the microstate category corresponding to each time point, and combine the microstate categories in chronological order to generate a microstate time series. Based on the microstate time series, calculate the dynamic characteristics of each type of microstate; Dynamic characteristics include average duration, frequency per second, time coverage ratio, and transition probability between different microstates; Multi-scale coarsening processing of micro-state time series is performed, specifically including: Multiple scale factors are set. For each scale factor, the microstate time series is divided into multiple non-overlapping time windows. The length of each window is equal to the current scale factor. The mode of the microstate category in each window is calculated. The modes calculated for each window are arranged in window order to obtain the coarse-grained microstate time series at this scale. Based on the coarse-grained microstate time series at each scale, the number of transitions from one type of microstate to another between adjacent time points is counted to obtain the transition number matrix corresponding to each scale. Calculate the row sum of each element in the transition number matrix; Divide each element in the row by the corresponding row sum and value to obtain the transition probability matrix for each scale. Calculate the transition entropy corresponding to each scale based on the transition probability matrix of each scale; The dynamic features and the transformation entropy corresponding to each scale are used as the EEG microstate feature vectors.

[0025] Step S203 involves fusing the behavioral feature vector with the EEG microstate feature vector to obtain the neural function feature vector, specifically including: Obtain the user's reaction time series within each time segment, calculate the ratio of the standard deviation to the mean of the reaction time series, and obtain the reaction time coefficient of variation; The coefficient of variation during reaction time is normalized to obtain a normalized value. The difference between 1 and the normalized value is calculated to obtain the first confidence level. Acquire artifact-free EEG signals, calculate the ratio of signal power to noise power of the EEG signals to obtain the signal-to-noise ratio (SNR), normalize the SNR to obtain the second confidence level; Based on the first and second confidence levels, the behavioral feature vector and the EEG microstate feature vector are weighted and fused to obtain the neural function feature vector.

[0026] In one embodiment, step S201 involves extracting quantitative indicators of the user's behavioral performance in the task state from the multimodal data segments, providing behavioral dimension features reflecting cognitive control and executive functions for subsequent fusion. Specifically, step S201 acquires the behavioral data stream from the multimodal data segments, and calculates the user's average accuracy rate (the percentage of correct responses out of the total number of stimuli in that segment) and average reaction time (the average time interval between stimulus presentation and user response) for each time segment based on the stimulus event labels and response records in the behavioral data stream. The average accuracy rate and average reaction time are used to construct a behavioral feature vector. This step transforms the original behavioral records into quantifiable feature indicators, providing a basic input for behavioral dimensions in subsequent fusion with EEG features. Step S202 involves extracting micro-state features reflecting the baseline state of the neural network and the dynamic characteristics of the brain network from the resting-state EEG signal. Multi-scale transformation entropy is used to capture the dynamic transformation complexity of the micro-state sequence at different time scales, thereby constructing a feature representation of the EEG dimension. Specifically, step S202 involves obtaining EEG signals with resting-state labels from multimodal data segments. Based on the potential values ​​of each electrode at each time point in the EEG signal, the standard deviation of the potential values ​​of all electrodes is calculated as the global field strength value at that time point, and a global field strength time series is constructed based on the global field strength value. Local maxima points are detected in the global field strength time series, i.e., the value at a point is greater than the values ​​at the two adjacent time points. The time corresponding to the local maxima point is then recorded as... The peak global field strength was used as the time point. The topographic map at the peak global field strength was spatially clustered using the K-means clustering algorithm, with the number of clusters K set to 4. This value is based on the classic research paradigm of EEG microstates and can cover most typical microstate types under resting state, generating 4 microstate templates. The cosine similarity between the topographic map of the EEG signal at each time point and each microstate template was calculated. The category of the microstate template with the highest similarity was taken as the microstate category at that time point, obtaining the microstate categories corresponding to each time point. The microstate categories were combined in chronological order to generate a microstate time series. Based on the microstate time series, the dynamic characteristics of each microstate category were calculated, including average duration, frequency per second, time coverage ratio, and other characteristics. The transition probabilities between microstates are calculated. The microstate time series undergoes multi-scale coarse-graining processing, specifically by setting multiple scale factors. These scale factors range from integers 1 to 10, covering multi-level time histories from local fine dynamics to global macro-dynamics. For each scale factor, the microstate time series is divided into multiple non-overlapping time windows, each with a length equal to the current scale factor. The mode of each microstate category within each window is calculated, and the modes obtained from each window are arranged in window order to obtain the coarse-grained microstate time series at that scale. Based on the coarse-grained microstate time series at each scale, the number of transitions from one type of microstate to another between adjacent time points is counted, yielding the transition probabilities corresponding to each scale. The transition probability matrix is ​​calculated by summing the row sum of each element in each row and dividing each element in that row by the corresponding row sum to obtain the transition probability matrix for each scale. Based on the transition probability matrix for each scale, the transition entropy for each scale is calculated by multiplying the negative logarithmic probability of each element in the transition probability matrix by the sum of the probabilities. This is used to quantify the uncertainty and dynamic complexity of the microstate sequence at that scale. The dynamic features and the transition entropy for each scale are used as the EEG microstate feature vector. This step, through the introduction of multi-scale transition entropy, effectively captures the dynamic transition features of the microstate sequence at different temporal granularities, giving the EEG microstate features richer temporal structure information and providing robust EEG dimensional features for subsequent fusion. Step S203 addresses the deficiency of fixed weights in existing multimodal fusion technologies by introducing an adaptive weighting mechanism based on data quality. This dynamically adjusts the fusion weights according to the real-time reliability of behavioral and EEG data, thereby constructing a more comprehensive and robust neural function feature vector. Specifically, step S203 involves acquiring the user's reaction time series within each time segment, calculating the ratio of the standard deviation to the mean of the reaction time series to obtain the reaction time coefficient of variation (RCV). A smaller RCV indicates higher consistency in the user's response and more reliable behavioral data quality. The RCV is then mapped to the 0-1 interval using a minimum-maximum normalization method to obtain a normalized value. The difference between 1 and the normalized value is calculated to obtain a first confidence level, which is positively correlated with behavioral data quality. Finally, artifact-free EEG signals are acquired, and the confidence level of the EEG signals is calculated. The signal-to-noise ratio (SNR) is obtained by calculating the ratio of signal power to noise power. The SNR is then mapped to the 0-1 range using a minimum-maximum normalization method to obtain a second confidence level, which is positively correlated with EEG signal quality. Based on the first and second confidence levels, the behavioral feature vector and the EEG microstate feature vector are weighted and fused. Specifically, a weighted summation method is used: the behavioral feature vector multiplied by the first confidence level is added to the EEG microstate feature vector multiplied by the second confidence level, and then divided by the sum of the first and second confidence levels to obtain the neural function feature vector. This step dynamically adjusts the fusion weights, allowing high-quality modalities to contribute more to the fusion and low-quality modalities to contribute less, effectively solving the problem of feature distortion in fixed-weight fusion when data quality fluctuates, and significantly improving the robustness and individual adaptability of the neural function feature vector. Step S2 extracts behavioral features from the task state and EEG microstate features from the resting state, and constructs a neural function feature vector using adaptive weighted fusion. Behavioral features are derived from the user's performance during cognitive task execution, reflecting cognitive control and executive functions in the task state. EEG microstate features are derived from resting-state EEG signals, reflecting the individual's neural baseline state and brain network dynamics. Behavioral and EEG microstate features characterize neural function states from the two dimensions of task induction and resting baseline, respectively, and are complementary. Through adaptive weighted fusion, a more comprehensive and robust neural function feature vector is constructed for subsequent identification and database construction. This step provides core feature representations that integrate the advantages of both task and resting states and can adapt to data quality fluctuations, laying the feature foundation for the entire neural function feature recognition process.

[0027] Step S3 includes steps S301, S302 and S303; Step S301: Obtain metadata corresponding to the neural function feature vector. The metadata includes paradigm information, user information, and cognitive state labels. The neural function feature vectors are associated with the corresponding metadata, and an index label is added to each feature vector to obtain feature vector records with metadata labels. Metadata includes paradigm information, user information, and cognitive state tags; Step S302: Store multiple feature vector records with metadata tags into the database and establish an index structure for the feature vectors to obtain the neural function feature database.

[0028] Step S303: Obtain the feature vector to be added to the database, and calculate the similarity between the feature vector to be added to the database and the feature vector in the database. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is less than the preset similarity threshold, the feature vector to be added to the database is determined to be a novel pattern. After adding the cognitive state label to be labeled to the feature vector to be added to the database, it is stored in the database and the index structure is updated. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is greater than or equal to the preset similarity threshold, then the cognitive state label corresponding to the feature vector with the highest similarity will be used as the recognition result and output after being marked as low reliability.

[0029] In one embodiment, step S301 establishes a traceable metadata tagging system for neural function feature vectors, enabling each feature vector to have clear source information and semantic annotation, providing a classification basis for subsequent retrieval and identification. Specifically, step S301 involves acquiring metadata corresponding to the neural function feature vectors, where the metadata includes paradigm information (i.e., the name of the cognitive task paradigm used during collection and the configuration of key paradigm parameters), user information (i.e., user identifier and demographic attributes), and cognitive state label (i.e., the standard neural function state category corresponding to the feature vector). The configuration of key paradigm parameters includes stimulus presentation time, stimulus interval, and sequence length. The neural function feature vectors are associated with the corresponding metadata, and an index label is added to each feature vector to obtain a feature vector record with metadata labels. This step, by binding feature vectors with multi-dimensional metadata, provides structured information support for accurate database retrieval and result tracing in the future. Step S302 involves organizing and storing feature vector records with complete labels using an efficient indexing method to form a neural function feature database that enables rapid similarity retrieval, providing a data foundation for identification queries. Specifically, step S302 includes storing multiple feature vector records with metadata labels into the database and establishing an index structure for the feature vectors. Vector space indexes such as KD trees or product quantization indexes are used to support rapid nearest neighbor retrieval of feature vectors, resulting in the neural function feature database. This step significantly improves the computational efficiency of subsequent similarity retrieval by constructing an efficient index structure, ensuring the real-time nature of the identification process. Step S303 automatically determines the novelty of the feature vector to be added to the database by using a preset similarity threshold, realizing automatic identification of novel patterns and dynamic expansion of the database. Simultaneously, it outputs the similarity matching results with low reliability, providing a closed-loop entry point. Specifically, step S303 obtains the feature vector to be added to the database, where the feature vector to be added refers to neural function feature vectors not yet stored in the database. These vectors can be obtained from newly extracted feature vectors from steps S201 to S203, or from query feature vectors transferred from subsequent step S403. The similarity between the feature vector to be added and the feature vectors in the database is calculated using cosine similarity. If the maximum similarity between the feature vector to be added and the feature vectors in the database is less than the preset similarity threshold, this preset similarity threshold is set to 0.85. The threshold is set based on the similarity distribution among similar feature vectors in a typical neural function feature database, using the lower quartile of the similarity distribution among similar samples as the threshold, which can effectively distinguish novel patterns. The process involves identifying novel patterns and comparing them with known patterns. Feature vectors to be added to the database are classified as novel patterns. Cognitive state labels are added to these vectors before they are stored in the database, and the index structure is updated. If the maximum similarity between a feature vector to be added and a feature vector in the database is greater than or equal to a preset similarity threshold, the cognitive state label corresponding to the feature vector with the highest similarity is used as the recognition result. This result is then output with a low-reliability label, meaning it was not obtained through the high-confidence label distribution statistics in step S403, but rather based on a single similarity match. The reliability of this recognition result is lower than the normal output threshold and is only for reference or subsequent manual review. This step, by introducing a preset similarity threshold, enables automatic identification and storage of novel patterns, allowing the database to continuously expand its coverage as data accumulates. Simultaneously, the low-reliability labeling mechanism clarifies the output confidence boundary based on a single matching result, providing a clear basis for subsequent manual review and iteration. Step S3, through the collaborative design of metadata association, index building, and novel pattern automatic judgment mechanism, constructs a neural function feature database with self-expanding capabilities. It can continuously enrich the feature coverage while accumulating new data, providing dynamically updated data support for similarity retrieval in step S4, and realizing the evolution of recognition from static query to dynamic self-learning.

[0030] Step S4 includes steps S401, S402 and S403; Step S401: Obtain the neural function feature vector to be identified as the query feature vector; The neural function feature vector to be identified is obtained using the same method as in steps S201 to S203; Step S402: Input the query feature vector into the neural function feature database, perform similarity retrieval using the index structure of the database, calculate the similarity between the query feature vector and each feature vector in the database, and return the K feature vector records with the highest similarity.

[0031] Step S403: Statistically analyze the distribution of cognitive state labels corresponding to the K feature vector records, and calculate the proportion of the label with the highest votes as the recognition confidence. If the recognition confidence is greater than or equal to the preset recognition confidence threshold, the label with the highest vote will be output as the neural function state corresponding to the query feature vector. If the recognition confidence level is less than the preset recognition confidence level threshold, the query feature vector will be used as the feature vector to be added to the database, and step S303 will be executed to enter the dynamic update process.

[0032] In one embodiment, step S401 ensures that the feature vector to be identified and the feature vector in the database construction stage are completely consistent in the extraction process, eliminating the retrieval bias introduced by the difference in feature calculation methods and laying a consistent foundation for subsequent similarity comparison. Specifically, step S401 obtains the neural function feature vector to be identified as the query feature vector. The neural function feature vector to be identified is obtained using the same method as steps S201 to S203, that is, following the same preprocessing, behavioral feature extraction, EEG microstate feature extraction and adaptive weighted fusion process, ensuring that the query feature and the features in the database have a unified dimension and scale. This step ensures that the query vector and the vector in the database are homologous and isomorphic in the representation space by reusing the feature extraction process, providing standardized input for accurate retrieval. Step S402 utilizes the existing index structure of the database to achieve efficient similarity retrieval of the query feature vector, quickly returning the K most similar candidate records, providing a statistically significant sample basis for subsequent confidence statistics. Specifically, step S402 inputs the query feature vector into the neural function feature database, and uses the index structure in the database, namely the vector space index established in step S302, such as a KD-tree, to perform similarity retrieval, calculating the similarity between the query feature vector and each feature vector in the database. The similarity calculation uses cosine similarity, consistent with step S303, and returns the K feature vector records with the highest similarity, where K is set to 5. This value is set based on the requirements of retrieval stability and statistical significance, balancing the representativeness of candidate samples and computational efficiency, avoiding statistical bias due to an excessively small K value or noise introduced due to an excessively large K value. This step, through index acceleration and K value optimization, significantly improves the recognition response speed while ensuring retrieval accuracy. Step S403 involves calculating the recognition confidence by statistically analyzing the label distribution of K nearest neighbor records, achieving highly reliable recognition based on group decision-making, and triggering a dynamic update process for low-confidence queries, forming a closed loop of recognition and self-learning. Specifically, step S403 involves statistically analyzing the distribution of cognitive state labels corresponding to K feature vector records, calculating the proportion of the label with the highest votes as the recognition confidence. For example, if a label appears 3 times when K=5, the confidence is 0.6. If the recognition confidence is greater than or equal to a preset recognition confidence threshold, which is set to 0.6, the basis for which is that when K=5, a simple majority is satisfied when the number of votes reaches 3 or more. The preset recognition confidence threshold can effectively distinguish between high-confidence recognition results with clear consensus and low-confidence results with disagreement. In this case, the label with the highest votes is used as the output of the neural function state corresponding to the query feature vector. If the recognition confidence is less than the preset recognition confidence threshold, the query feature vector is used as a feature vector to be stored in the database. The feature vector to be stored in the database refers to the neural function state that has not yet been stored in the database. The functional feature vectors, which are derived from newly extracted feature vectors in steps S201 to S203 or query feature vectors transferred from this step, are executed in step S303 to enter the dynamic update process. In this process, if the maximum similarity between the query feature vector and the feature vector in the database is greater than or equal to the preset similarity threshold of 0.85 set in step S303, the cognitive state label corresponding to the feature vector with the highest similarity is used as the recognition result and output with low reliability. This low reliability label means that the recognition result is not obtained through the statistical distribution of high-confidence labels in this step, but is based on a single similarity match. Its reliability is lower than the output confidence corresponding to the preset recognition confidence threshold in step S403, and is only for reference or subsequent manual review. This step realizes the direct output of high-confidence recognition and the secondary processing of low-confidence queries through the confidence threshold mechanism. This not only ensures the reliability of the recognition result, but also continuously absorbs new patterns by transferring low-confidence queries into the dynamic update process, forming a positive cycle of recognition and self-learning.

[0033] Step S4, through the collaborative design of isomorphic feature acquisition, indexed similarity retrieval, identification judgment based on label distribution confidence, and dynamic update triggering, can output reliable identification results at high confidence levels and automatically trigger the entry of novel patterns into the database and index updates at low confidence levels, thus achieving a balance between identification accuracy and scalability.

[0034] Example 2, refer to Figure 2 In another embodiment of the present invention, which differs from the first embodiment, a neural functional feature recognition system based on multimodal signal processing is provided, including a processing module, a fusion module, a determination module, and a recognition module; The processing module preprocesses the user's raw behavioral data stream and EEG data to obtain multimodal data segments; The fusion module extracts behavioral features and EEG microstate features from multimodal data segments, and performs adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector. The judgment module associates neural function feature vectors with corresponding metadata and adds tags, stores them in the database and establishes an index structure to obtain a neural function feature database. When obtaining feature vectors to be stored in the database, it judges novel patterns by a preset similarity threshold, automatically stores novel patterns in the database and updates the index structure. The recognition module performs a similarity search between the neural function feature vector to be recognized and the database. Based on the recognition confidence of the label distribution in the search results, if the recognition confidence reaches the preset recognition confidence threshold, the recognition result is output; otherwise, the query feature vector is transferred to the dynamic update process.

[0035] In one embodiment, multimodal data segments are established by aligning, segmenting, filtering, rereferencing, and removing artifacts through independent component analysis of the original behavioral data stream and EEG data based on a unified timestamp. This provides a temporally consistent and signal-clean data foundation for subsequent feature extraction. Addressing the limitation of fixed multimodal fusion weights in existing technologies, behavioral features and EEG microstate features are extracted from the multimodal data segments and adaptively weighted fusion is performed based on reaction time coefficient of variation and signal-to-noise ratio to obtain neural function feature vectors. This effectively solves the problem of feature distortion during data quality fluctuations across different modalities and significantly improves the robustness of feature representation. Finally, neural function features are formed by associating neural function feature vectors with metadata and establishing an index structure. The database is used to identify novel patterns by using a preset similarity threshold when acquiring feature vectors to be added to the database. Novel patterns are automatically stored in the database and the index structure is updated, realizing the dynamic expansion and self-learning capabilities of the database. The neural function feature vectors to be identified are compared with the database for similarity retrieval. The recognition result is output or the dynamic update process is initiated based on the recognition confidence of the label distribution in the retrieval results. This overcomes the limitation of high uncertainty in traditional single similarity matching and effectively reduces the risk of misjudgment. The entire architecture, through the synergistic effect of multimodal adaptive fusion and dynamic closed-loop update mechanism, constructs an accurate, self-learning, and evolvable recognition of neural function features. While ensuring recognition accuracy, it significantly enhances the generalization ability to individual differences and clinical evolution.

[0036] This invention addresses the shortcomings of existing neurofunctional recognition technologies, such as fixed multimodal fusion weights, static and closed databases, and a lack of confidence assessment. It proposes an adaptive weighted fusion strategy based on reaction time coefficient of variation and signal-to-noise ratio, overcoming the distortion of fused features when behavioral or EEG signal quality fluctuates. This significantly improves the robustness and individual adaptability of neurofunctional feature vectors. Simultaneously, it constructs a dynamic database supporting automatic novel pattern identification and index updates, and introduces a recognition and judgment mechanism based on label distribution confidence. This solves the problems of existing technologies' inability to automatically discover novel patterns and high uncertainty in single similarity matching. This approach enables self-learning capabilities while effectively reducing the risk of misjudgment, achieving adaptive fusion of multimodal features and dynamic closed-loop updates for recognition. While maintaining recognition accuracy, it significantly enhances the generalization ability to individual differences and clinical evolution.

[0037] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0038] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A method for identifying neural functional features based on multimodal signal processing, characterized in that, Includes the following steps: Step S1: Preprocess the user's raw behavioral data stream and EEG data to obtain multimodal data segments; Step S2: Extract behavioral features and EEG microstate features from the multimodal data segments, and perform adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector; Step S3: After associating the neural function feature vector with the corresponding metadata and adding tags, store it in the database and establish an index structure to obtain the neural function feature database. When obtaining the feature vector to be stored in the database, determine the novel pattern through a preset similarity threshold, automatically store the novel pattern in the database and update the index structure. Step S4: Perform a similarity search between the neural function feature vector to be identified and the database. Based on the identification confidence of the label distribution in the search results, if the identification confidence reaches the preset identification confidence threshold, the identification result is output; otherwise, the query feature vector is transferred to the dynamic update process.

2. The neural functional feature recognition method based on multimodal signal processing as described in claim 1, characterized in that: Step S1 includes steps S101, S102, S103 and S104; Step S101: When the user performs a preset cognitive task paradigm, collect the user's raw behavioral data stream and EEG data; The collection of raw behavioral data streams specifically includes: Present a visual sequence to the user and record the user's behavioral response to the visual sequence to obtain the raw behavioral data stream; The raw behavioral data stream includes timestamps, stimulus event markers, and response records; The collection of EEG data specifically includes: During the user's task execution and resting state, electrical activity signals are collected using a multi-channel EEG device to obtain raw multi-channel EEG time-series signals.

3. The neural functional feature recognition method based on multimodal signal processing as described in claim 2, characterized in that: Step S102, preprocessing includes: The original behavioral data stream and the original multi-channel EEG time series signal are aligned based on a unified timestamp to obtain the aligned original behavioral data stream and the aligned original multi-channel EEG time series signal. Based on the preset time markers, the aligned original multi-channel EEG time series signal is divided into multiple time segments, and corresponding cognitive state labels are added to each time segment to obtain multimodal data segments; Step S103, the preprocessing also includes: The EEG signals in the multimodal data segments are filtered and rereferenced to obtain filtered and rereferenced EEG signals, specifically including: The filtering process includes using a high-pass filter to remove low-frequency drift below 0.5Hz and using a low-pass filter to remove high-frequency noise above 50Hz. The filtered EEG signal is rereferenced to obtain an average reference EEG signal, and the average reference EEG signal is downsampled to obtain a filtered rereference EEG signal. Step S104: The filtered rereference EEG signal is decomposed into multiple independent source components using the independent component analysis algorithm. Based on the topographic distribution characteristics of each independent source component, noise components corresponding to eye movement, blinking, and electromyographic physiological activities are removed to obtain an artifact-free EEG signal.

4. The neural functional feature recognition method based on multimodal signal processing as described in claim 3, characterized in that: Step S2 includes steps S201, S202 and S203; Step S201, extracting behavioral features from multimodal data segments, specifically including: Obtain behavioral data streams from multimodal data segments. Based on stimulus event labels and response records in the behavioral data streams, calculate the average accuracy and average reaction time of users within each time segment. Construct behavioral feature vectors using the average accuracy and average reaction time. Step S202 involves extracting EEG microstate features from multimodal data segments, specifically including: EEG signals with resting state labels are obtained from multimodal data segments. Based on the potential values ​​of each electrode in the EEG signal at each time point, the standard deviation of the potential values ​​of all electrodes is calculated as the global field strength value at that time point. Construct a global field strength time series based on the global field strength value; Detect local maxima in the global field strength time series and take the time corresponding to the local maxima as the peak time of the global field strength; The topographic map at the peak of the global field strength is spatially clustered using the K-means clustering algorithm to generate multiple micro-state templates; Calculate the cosine similarity between the topographic map and each microstate template of the EEG signal at each time point, take the category of the microstate template with the highest similarity as the microstate category at that time point, obtain the microstate category corresponding to each time point, and combine the microstate categories in chronological order to generate a microstate time series. Based on the microstate time series, calculate the dynamic characteristics of each type of microstate; Dynamic characteristics include average duration, frequency per second, time coverage ratio, and transition probability between different microstates; Multi-scale coarsening of micro-state time series is performed, specifically including: Multiple scale factors are set. For each scale factor, the microstate time series is divided into multiple non-overlapping time windows. The length of each window is equal to the current scale factor. The mode of the microstate category in each window is calculated. The modes calculated for each window are arranged in window order to obtain the coarse-grained microstate time series at this scale. Based on the coarse-grained microstate time series at each scale, the number of transitions from one type of microstate to another between adjacent time points is counted to obtain the transition number matrix corresponding to each scale. Calculate the row sum of each element in the transition number matrix; Divide each element in the row by the corresponding row sum and value to obtain the transition probability matrix for each scale. Calculate the transition entropy corresponding to each scale based on the transition probability matrix of each scale; The dynamic features and the transformation entropy corresponding to each scale are used as the EEG microstate feature vectors.

5. The neural functional feature recognition method based on multimodal signal processing as described in claim 4, characterized in that: Step S203 involves fusing the behavioral feature vector with the EEG microstate feature vector to obtain the neural function feature vector, specifically including: Obtain the user's reaction time series within each time segment, calculate the ratio of the standard deviation to the mean of the reaction time series, and obtain the reaction time coefficient of variation; The coefficient of variation during reaction time is normalized to obtain a normalized value. The difference between 1 and the normalized value is calculated to obtain the first confidence level. Obtain an artifact-free EEG signal, calculate the ratio of signal power to noise power of the EEG signal to obtain the signal-to-noise ratio, normalize the signal-to-noise ratio to obtain the second confidence level; Based on the first and second confidence levels, the behavioral feature vector and the EEG microstate feature vector are weighted and fused to obtain the neural function feature vector.

6. The neural functional feature recognition method based on multimodal signal processing as described in claim 5, characterized in that: Step S3 includes steps S301, S302 and S303; Step S301: Obtain metadata corresponding to the neural function feature vector. The metadata includes paradigm information, user information, and cognitive state labels. The neural function feature vectors are associated with the corresponding metadata, and an index label is added to each feature vector to obtain feature vector records with metadata labels. Metadata includes paradigm information, user information, and cognitive state tags; Step S302: Store multiple feature vector records with metadata tags into the database and establish an index structure for the feature vectors to obtain the neural function feature database.

7. The neural functional feature recognition method based on multimodal signal processing as described in claim 6, characterized in that: Step S303: Obtain the feature vector to be added to the database, and calculate the similarity between the feature vector to be added to the database and the feature vector in the database. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is less than the preset similarity threshold, the feature vector to be added to the database is determined to be a novel pattern. After adding the cognitive state label to be labeled to the feature vector to be added to the database, it is stored in the database and the index structure is updated. If the maximum similarity between the feature vector to be added to the database and the feature vector in the database is greater than or equal to the preset similarity threshold, then the cognitive state label corresponding to the feature vector with the highest similarity will be used as the recognition result and output after being marked as low reliability.

8. The neural functional feature recognition method based on multimodal signal processing as described in claim 7, characterized in that: Step S4 includes steps S401, S402 and S403; Step S401: Obtain the neural function feature vector to be identified as the query feature vector; The neural function feature vector to be identified is obtained using the same method as in steps S201 to S203; Step S402: Input the query feature vector into the neural function feature database, perform similarity retrieval using the index structure of the database, calculate the similarity between the query feature vector and each feature vector in the database, and return the K feature vector records with the highest similarity.

9. The neural functional feature recognition method based on multimodal signal processing as described in claim 8, characterized in that: Step S403: Statistically analyze the distribution of cognitive state labels corresponding to the K feature vector records, and calculate the proportion of the label with the highest votes as the recognition confidence. If the recognition confidence is greater than or equal to the preset recognition confidence threshold, the label with the highest vote will be output as the neural function state corresponding to the query feature vector. If the recognition confidence level is less than the preset recognition confidence level threshold, the query feature vector will be used as the feature vector to be added to the database, and step S303 will be executed to enter the dynamic update process.

10. A neural functional feature recognition system based on multimodal signal processing, applied to a neural functional feature recognition method based on multimodal signal processing as described in any one of claims 1-9, characterized in that, It includes a processing module, a fusion module, a judgment module, and a recognition module; The processing module preprocesses the user's raw behavioral data stream and EEG data to obtain multimodal data segments; The fusion module extracts behavioral features and EEG microstate features from multimodal data segments, and performs adaptive weighted fusion of behavioral features and EEG microstate features to obtain a neural function feature vector. The judgment module associates neural function feature vectors with corresponding metadata and adds tags, stores them in the database and establishes an index structure to obtain a neural function feature database. When obtaining feature vectors to be stored in the database, it judges novel patterns by a preset similarity threshold, automatically stores novel patterns in the database and updates the index structure. The recognition module performs a similarity search between the neural function feature vector to be recognized and the database. Based on the recognition confidence of the label distribution in the search results, if the recognition confidence reaches the preset recognition confidence threshold, the recognition result is output; otherwise, the query feature vector is transferred to the dynamic update process.

Images