Brain function evaluation method and system based on multi-modal data fusion

Abstract

The application discloses a brain function evaluation method and system based on multi-modal data fusion, and belongs to the technical field of brain function evaluation. The method comprises the following steps: receiving a voice signal, an eye movement trajectory coordinate sequence and a handwriting trace pressure array which are synchronously collected when a subject performs a preset cognitive task; extracting a latent feature tensor through multi-view recursive canonical tensor decomposition; generating a covariance feature matrix located on a symmetric positive definite manifold through Lie algebra manifold mapping; calculating a multi-dimensional evaluation feature vector in a tangent space based on Riemann logarithmic mapping; solving and obtaining a convergence trajectory reflecting the cooperative state of each cognitive domain by using a residual dynamics system constrained by a Hamiltonian; and outputting a quantitative evaluation indication of each cognitive domain through adaptive threshold segmentation of a regularization path. The scheme realizes full-process analysis from multi-modal time series data to cognitive state dynamics characteristics, and directly outputs a quantitative evaluation result.

Description

Technical Field

[0001] This invention relates to the field of brain function assessment technology, specifically to a brain function assessment method and system based on multimodal data fusion. Background Technology

[0002] Existing brain function assessment techniques largely rely on single-modality data, such as electroencephalography (EEG) or functional magnetic resonance imaging (fMRI). These methods are limited by equipment and experimental conditions, making them difficult to implement conveniently in everyday cognitive tasks. Some solutions attempt to combine behavioral data such as speech, eye movement, or handwriting for assessment, but the fusion processing of multimodal temporal data remains at the feature concatenation level, failing to delve into the potential higher-order interaction structures and temporal evolution patterns between different modalities. Conventional tensor decomposition methods, when processing multi-view temporal data, struggle to guarantee the dynamic consistency of the factor matrix over time, resulting in insufficient representational ability of extracted features to depict changes in cognitive states.

[0003] Another problem is that feature vectors extracted from multimodal data are mostly located in high-dimensional Euclidean space. Conventional linear discriminant analysis or Euclidean distance analysis ignores the inherent nonlinear manifold geometry of the covariant relationship between brain functional connectivity and cognitive states. Even when manifold learning methods are used, the data is often projected onto a general manifold instead of specifically modeling the Riemannian manifold composed of symmetric positive definite matrices. This leads to distortion in the measurement of covariance features, making it impossible to accurately reflect the interaction strength between different cognitive domains.

[0004] Furthermore, existing technologies tend to simplify the modeling of the dynamic evolution of cognitive states, typically employing static classification or univariate trend analysis. They fail to incorporate multiple cognitive variables into a unified energy dynamics framework to simulate their nonlinear coupling and co-evolution process, making it difficult to fully characterize the internal dynamics of a cognitive system as it transitions from one functional state to another.

[0005] Therefore, how to fully preserve the inherent multi-view low-rank cyclic structure and non-Euclidean geometric characteristics of multimodal behavioral time series data during the fusion process, and on this basis, establish a dynamic evolution model of cognitive state variables to obtain a quantitative representation reflecting the cognitive collaborative state, is an urgent problem to be solved. Summary of the Invention

[0006] The brain function assessment method and system based on multimodal data fusion aims to solve the problem of how to preserve the non-Euclidean geometric characteristics of the inherent multi-view low-rank cyclic structure and symmetric positive definite manifold when fusing multimodal behavioral time-series data. On this basis, a dynamic evolution model of cognitive state variables is constructed to obtain a quantitative convergence trajectory reflecting the collaborative state of cognitive domains, and then output standardized assessment results for each cognitive domain.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a brain function assessment method based on multimodal data fusion, comprising: The system receives a set of multimodal temporal data collected synchronously when the subject performs a preset cognitive task. The set of multimodal temporal data includes at least speech signals, eye movement trajectory coordinate sequences, and handwriting pressure arrays.

[0008] The first step of the multimodal time series data set is performed based on multi-view recursive canonical tensor decomposition to extract latent feature tensors related to cognitive states. In this first step, preferably, the speech signal, eye-tracking coordinate sequence, and handwritten pressure array are arranged in chronological order as a first time-series vector, a second time-series vector, and a third time-series vector, respectively. Each time-series vector is mapped to a high-dimensional Hilbert space, with radial basis function kernels used for embedding during mapping, and the embedding dimension is adaptively determined using a pseudo nearest neighbor method, thereby obtaining a first reconstructed phase space trajectory matrix, a second reconstructed phase space trajectory matrix, and a third reconstructed phase space trajectory matrix. These reconstructed phase space trajectory matrices are stacked along the third dimension to construct a third-order source tensor. A recursive canonical decomposition constraint is applied to the third-order source tensor, decomposing it into a product of a core tensor and three factor matrices. The recursive canonical decomposition constraint forces the core tensor to form a low-rank cyclic structure along the time dimension, thereby effectively capturing the cyclic dynamic features in multimodal time-series data. After expanding the core tensor along the time dimension, the principal singular vectors are taken and reconstructed into a latent feature tensor, realizing high-order cross-fusion and dimensionality reduction of multimodal time-series information.

[0009] A second step is performed on the latent feature tensor using a Lie algebraic manifold mapping based on non-Euclidean space to generate a covariance feature matrix located on a symmetric positive definite manifold. In this second step, each time slice of the latent feature tensor is expanded into a sequence of eigenvectors, and the Gram matrix between the eigenvectors of different time slices is calculated to obtain multiple instantaneous covariance matrices. These instantaneous covariance matrices are then projected onto the symmetric positive definite manifold. The projection operation is performed by performing eigenvalue decomposition on each instantaneous covariance matrix, replacing its negative eigenvalues ​​with their absolute values, and then reconstructing the matrix to ensure the positive definiteness of the covariance matrix. Finally, the multiple projected instantaneous covariance matrices are Riemann-weightedly averaged along the time dimension to generate the covariance feature matrix. The weight coefficients of the Riemann weighted average are inversely proportional to the Mahalanobis distance between the eigenvectors of adjacent time slices, and the weight coefficients decay exponentially, resulting in smaller weights for adjacent slices with larger Mahalanobis distances from the current time slice. The decay rate is controlled by a preset time constant. By using manifold mapping and Riemann weighted averaging, we can robustly fuse temporal evolution information to obtain a feature matrix that represents the covariance structure of the global cognitive state.

[0010] A third step based on Riemann logarithmic mapping is performed on the covariance feature matrix to calculate multi-dimensional evaluation feature vectors within the tangent space of a symmetric positive definite manifold. Preferably, the identity matrix on the symmetric positive definite manifold is selected as the reference point of the tangent space. The Riemann logarithmic mapping of the covariance feature matrix relative to the reference point is calculated by solving the geodesic equation between the covariance feature matrix and the reference point. The resulting matrix of the Riemann logarithmic mapping is then subjected to lower triangular element extraction and vectorization to obtain the original tangent space vector. Dimensionality reduction based on chi-square distance is performed on the original tangent space vector, and the principal component components obtained after dimensionality reduction are sequentially concatenated to form the multi-dimensional evaluation feature vector. The kernel function used in the kernel principal component analysis dimensionality reduction is a Gaussian radial basis kernel, and the kernel parameters are optimized by maximizing the ratio of the inter-class scatter matrix to the intra-class scatter matrix of the dimensionality-reduced feature vector, further enhancing the discriminative ability of the features across different cognitive states.

[0011] The fourth step of the process, which involves solving a residual dynamics system based on Hamiltonian constraints, is performed on the multi-dimensional evaluation feature vector to obtain a convergence trajectory reflecting the collaborative state of each cognitive domain. In this fourth step, each dimension of the multi-dimensional evaluation feature vector is treated as a cognitive state variable, constructing a state space composed of multiple cognitive state variables. A total energy function is defined in the state space, which is the sum of the potential energy term of each cognitive state variable and the coupling potential energy term between any two cognitive state variables. Based on the partial derivatives of the total energy function with respect to each cognitive state variable, a system of equations of motion under Hamiltonian constraints is constructed. Residual terms are introduced into the system of equations of motion to describe external observation noise and unmodeled dynamics. A pre-defined symplectic integrator is used to numerically solve the system of equations of motion after introducing the residual terms, iterating from the initial state until the rate of change of the total energy function is lower than a pre-defined energy gradient threshold. The sequence of state space points recorded during the iteration is used as the convergence trajectory. Through Hamiltonian constraints and residual dynamics, the energy interaction and collaborative evolution process between multiple cognitive domains can be characterized, resulting in a more stable and interpretable convergence trajectory. Furthermore, two state space points corresponding to the initial and final states in the convergence trajectory are extracted, the Riemann geodesic distance between the two state space points is calculated, and the average evolution rate is obtained by dividing the distance by the number of iterations. This average evolution rate is output as an additional convergence dynamic feature along with the convergence trajectory to characterize the overall evolution speed of the cognitive system.

[0012] A fifth step, adaptive threshold segmentation based on regularized paths, is performed on the convergence trajectory to output quantitative evaluation indicators for each cognitive domain. In this fifth step, the projection sequences of the convergence trajectory onto each cognitive state variable dimension are subjected to Fourier transforms to obtain the frequency domain energy spectrum corresponding to each cognitive state variable. Regularized path analysis based on minimum absolute contraction and selection operators is applied to each frequency domain energy spectrum, extracting frequency components with energy values ​​higher than the energy threshold corresponding to the inflection point of the regularized path as feature frequency bands. The ratio of the average energy value within the feature frequency band of each cognitive state variable to the reference baseline energy value is calculated to obtain the original evaluation score for that cognitive state variable. The original evaluation scores of each cognitive state variable are then input into their respective pre-trained adaptive logical threshold functions to output quantitative evaluation indicators for each cognitive domain. When constructing the adaptive logical threshold function, for each cognitive state variable, the quantile parameters of the corresponding healthy population reference distribution are obtained, and a tilt coefficient and a shift coefficient are constructed based on the quantile parameters. The original assessment score is input into the constructed adaptive logical threshold function, which is a composite function of an exponential function with the natural constant as the base. The numerator contains the exponential operation of the difference between the original assessment score and the shift coefficient multiplied by the negative value of the tilt coefficient, and the denominator is the sum of the constant and the exponential operation results. The calculation result is mapped to a preset numerical range to obtain the quantitative assessment indicator corresponding to the cognitive state variable. This achieves adaptive threshold adjustment based on an individualized reference distribution, making the assessment results more meaningful for individualized calibration.

[0013] As a preferred embodiment of the present invention, after receiving the multimodal time-series data set, the data is further preprocessed, including: performing cascaded preprocessing based on blind source separation and Wiener filtering on the speech signal to separate the clean speech component and the environmental noise component, and calculating the short-time energy envelope of the clean speech component; performing gaze point recognition based on Kalman filtering and velocity thresholding on the eye movement trajectory coordinate sequence to segment the eye movement trajectory coordinate sequence into a gaze point sequence and an eye sac segment sequence; performing time-frequency analysis based on empirical mode decomposition on the handwriting pressure array to decompose it into multiple intrinsic mode function components, and selecting the dominant component related to the handwriting action; and aligning the short-time energy envelope, the dwell time statistics of the gaze point sequence, and the Hilbert spectrum of the dominant component with timestamps to form a preprocessed multimodal time-series data set. Further detection is performed on the aligned dataset to check for missing data segments or misaligned timestamps. When missing data segments are found, Gaussian process regression-based interpolation is used to fill them in. When misaligned timestamps are found, the maximum cross-correlation delay between each pair of the speech signal, eye-tracking coordinate sequence, and handwritten pressure array is calculated. Using the timestamp of the speech signal as a reference, the timestamps of the eye-tracking coordinate sequence and the handwritten pressure array are resampled and aligned according to the maximum cross-correlation delay. The aligned multimodal time-series dataset is then divided into multiple continuous time-series data segments according to fixed-duration time windows. Each time-series data segment is used in the subsequent first-step processing. These preprocessing operations effectively remove noise, synchronize multimodal data, and maintain temporal continuity, improving the accuracy and robustness of subsequent fusion analysis.

[0014] This invention also provides a brain function assessment system based on multimodal data fusion, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the aforementioned brain function assessment method based on multimodal data fusion. This system can automatically acquire multimodal data, perform tensor decomposition, manifold mapping, dynamics solving, and adaptive assessment, outputting quantitative assessment indicators for each cognitive domain, providing objective and accurate technical support for clinical cognitive function assessment.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: When processing speech, eye-tracking, and handwriting data using multi-view recursive canonical tensor decomposition, the temporal vectors of the three modalities are mapped to a high-dimensional Hilbert space to obtain reconstructed phase space trajectory matrices, which are then stacked into third-order source tensors. Recursive canonical decomposition constraints are applied to force the core tensor to maintain a low-rank cyclic structure along the temporal dimension slices. This decomposition method enables factorized mutual constraints and cyclic alignment of temporal dynamic information from different modalities in the latent space. The extracted latent feature tensors simultaneously preserve high-order interaction patterns between modalities and temporal evolution consistency within modalities. Compared to conventional feature concatenation or independent component analysis, this method can more completely capture the coupled features of cognitive states in multimodal behavior.

[0016] When processing latent feature tensors using Lie algebraic manifold mapping and Riemannian logarithmic mapping, the instantaneous covariance matrix is ​​first projected onto a symmetric positive definite manifold and then weighted by Riemann to generate a covariance feature matrix that meets positive definite constraints. This matrix is ​​then transformed to the tangent space using the identity matrix as the base point via Riemannian logarithmic mapping, followed by lower triangular element vectorization and chi-square distance kernel principal component analysis for dimensionality reduction. This process maps high-dimensional covariance features from a curved manifold to a flat tangent space while preserving their inherent non-Euclidean geometric distance relationships. During dimensionality reduction, it maximizes the between-class and within-class scatter ratios, ensuring that the final multi-dimensional evaluation feature vectors contain more accurate cognitive domain difference information, effectively mitigating the measurement distortion problem caused by Euclidean space processing.

[0017] When constructing a residual dynamic system by introducing Hamiltonian constraints to solve for the trajectories of each cognitive state variable, each dimension of the multi-dimensional evaluation feature vector is treated as a cognitive state variable, and a total energy function containing potential energy terms and coupled potential energy terms is defined. Based on this, a system of motion equations containing residual terms is established, and a symplectic integrator is used for numerical solution until the rate of change of the total energy function is lower than a preset gradient threshold. This solution process generates convergent trajectories that reflect the energy distribution and cooperative migration of multiple cognitive domains while maintaining the system's geometric structure. It overcomes the limitation of static analysis in describing the dynamic evolution of cognitive processes, and the output results more realistically present the functional interactions and equilibrium states of each cognitive domain. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0019] Figure 1 This is a flowchart of a brain function assessment method based on multimodal data fusion; Figure 2 This is a flowchart of a method for generating the covariance feature matrix based on a symmetric positive definite manifold; Figure 3 This is a schematic diagram of the feature dimensionality reduction process based on Riemannian manifolds and kernel principal component analysis; Figure 4 This is a flowchart of the cognitive state evolution evaluation based on Hamiltonian constraints; Figure 5 This is a flowchart of a multimodal time-series data cognitive function assessment method; Figure 6 It is the time series variation curve of the principal singular vector of the latent feature tensor; Figure 7 It is a multi-dimensional evaluation feature vector principal component distribution map based on kernel principal component analysis; Figure 8 It is a multi-dimensional evaluation curve of the principal component eigenvalues ​​and cumulative variance contribution rate of the eigenvector; Figure 9 It is the mapping curve between the original evaluation score of the cognitive state variable and the adaptive logical threshold function. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1This invention provides a brain function assessment method based on multimodal data fusion. In specific implementation, a processor executes a computer program to store and process a set of multimodal temporal data synchronously collected by the subject while performing a preset cognitive task. The multimodal temporal data set includes at least speech signals, eye-tracking coordinate sequences, and handwriting pressure arrays. The computer program first performs a first-step processing on the multimodal temporal data set based on multi-view recursive canonical tensor decomposition, extracting latent feature tensors related to cognitive states from high-dimensional phase space reconstruction and tensor decomposition. Then, a second-step processing is performed on the extracted latent feature tensors based on Lie algebraic manifold mapping in non-Euclidean space, generating a covariance feature matrix located on a symmetric positive definite manifold through instantaneous Gram matrix calculation and Riemann weighted averaging on the symmetric positive definite manifold. Next, a third-step processing is performed on the covariance feature matrix based on Riemann logarithmic mapping, calculating a multidimensional assessment feature vector in the tangent space of the symmetric positive definite manifold through lower triangular element extraction, vectorization, and kernel principal component analysis. Next, the fourth step of the process, solving the residual dynamics system based on Hamiltonian constraints, is performed on the multi-dimensional evaluation feature vectors. A total energy function and a set of equations of motion are constructed in the state space, and residual terms are introduced. A symplectic integrator is used for numerical solution to obtain the convergence trajectory reflecting the collaborative state of each cognitive domain. Finally, the fifth step, adaptive threshold segmentation based on a regularized path, is performed on the convergence trajectory. Feature frequency bands are extracted through frequency domain energy spectrum analysis and LASSO regularization path, the original evaluation score is calculated, and input into the adaptive logical threshold function to output a quantitative evaluation indicator for each cognitive domain.

[0022] Example 1:

[0023] In practice, a multimodal temporal data set is received synchronously while the subject performs a preset cognitive task. This multimodal temporal data set includes at least speech signals, eye-tracking coordinate sequences, and handwriting pressure arrays. The speech signals are arranged in chronological order as a first temporal vector, the eye-tracking coordinate sequences as a second temporal vector, and the handwriting pressure arrays as a third temporal vector. The arrangement maintains the monotonically increasing order of the original sampling timestamps. Each element in the first temporal vector corresponds to the amplitude of a speech sampling point, each element in the second temporal vector corresponds to an eye-tracking coordinate point, and each element in the third temporal vector corresponds to a handwriting pressure value.

[0024] In the specific implementation, the first, second, and third time-series vectors are mapped to a high-dimensional Hilbert space to obtain the first, second, and third reconstructed phase space trajectory matrices. Radial basis function kernels are used for embedding during the mapping process, and for any two time points... and The expression for the radial basis function kernel for: in: Represents the time points in the first time series vector The corresponding element value, Represents the time points in the first time series vector The corresponding element value, express and The absolute value of the difference The width parameter of the radial basis function kernel. The value is the median of the standard deviations of all elements in the first time series vector. Represented by natural constant The second and third time series vectors are mapped using the same form of radial basis function kernel, where the width parameter corresponds to the median of the standard deviations of all elements in the second and third time series vectors, respectively.

[0025] In practice, the embedding dimension is adaptively determined using a pseudo nearest neighbor method. For the first time-series vector, the embedding dimension is determined... Initially, for each phase point in the reconstructed phase space, find its nearest neighbor and calculate the distance between the nearest neighbors; gradually increase the embedding dimension. When the embedding dimension increases to When, if the two phase points are at In 3D space, it is the nearest neighbor and If two points are no longer nearest neighbors in the dimensional space, they are marked as false nearest neighbors; when the proportion of false nearest neighbors is less than a preset threshold, the current embedded dimension is changed. The embedding dimension of the first time-series vector is set with a false nearest neighbor ratio threshold of 0.05. The embedding dimensions of the second and third time-series vectors are determined in the same manner. Using the determined embedding dimensions and kernel mapping results, the first, second, and third reconstructed phase space trajectory matrices are constructed. The number of rows in each reconstructed phase space trajectory matrix is ​​the embedding dimension, and the number of columns is the length of the time-series vector minus the embedding dimension plus one. Each column represents the image of a delayed embedding vector in a high-dimensional Hilbert space.

[0026] In the specific implementation, the first reconstructed phase space trajectory matrix, the second reconstructed phase space trajectory matrix, and the third reconstructed phase space trajectory matrix are stacked along the third dimension to construct a third-order source tensor. The first order of the third-order source tensor corresponds to the embedding dimension, the second order corresponds to the time index, and the third order corresponds to the modality type, with a total of 3 modality types.

[0027] In practice, a recursive canonical decomposition constraint is applied to the third-order source tensor, decomposing it into a product of the core tensor and three factor matrices. Decomposed into ,in For the core tensor, It is a first-order factor matrix. It is a second-order factor matrix. It is a third-order factor matrix. Represents tensors and matrices along the first... The modal product of order 3. Recursive canonical decomposition constraints force the core tensor to form a low-rank cyclic structure along the time dimension. Specifically, during the alternating least squares solution of the factor matrix and the core tensor, low-rank constraints and cyclic structure constraints are applied to the matrix obtained by expanding the core tensor along the time dimension (i.e., the second-order expansion). The low-rank constraint uses nuclear norm regularization, and the cyclic structure constraint is applied by circularly shifting each row of the expansion matrix one position to the right of the previous row. Simultaneously, the rank of the expansion matrix is ​​limited to a preset low-rank value, which is the result of rounding down one-quarter of the product of the embedding dimension and the number of modal types. The coefficient of nuclear norm regularization is set to one-tenth of the spectral norm of the third-order source tensor along the second-order expansion matrix.

[0028] In practice, after completing the recursive canonical decomposition, the core tensor is expanded along the time dimension to obtain the expanded matrix of the core tensor along the time dimension. Singular value decomposition is then performed on this expanded matrix to extract the principal singular vectors. The number of principal singular vectors is set to the minimum number that ensures the cumulative singular value ratio first exceeds 0.95. The extracted principal singular vectors are then reorganized into a latent feature tensor according to their correspondence with the time dimension slices of the original core tensor. The first order of the latent feature tensor is the principal singular vector index, the second order is the time index, and the third order is the same as the third order dimension of the third-order source tensor.

[0029] See Figure 6 The horizontal axis of the graph represents the time index, and the vertical axis represents the principal singular vector values. It shows the changing trends of the first three principal singular vectors extracted by expanding the core tensor along the time dimension and performing singular value decomposition after recursive canonical decomposition in Example 1 over time. The curves represent the first principal singular vector (solid line), the second principal singular vector (dashed line), and the third principal singular vector (dotted line), respectively.

[0030] As shown in the figure, the three principal singular vectors exhibit significant oscillatory changes throughout the entire time index range, with amplitudes roughly concentrated between ±0.8, reflecting the dynamic characteristics of the latent feature tensor in the time dimension. The first principal singular vector curve shows relatively regular periodic fluctuations, with alternating peaks and troughs, and its amplitude reaches a large value in multiple time intervals, indicating that this principal singular vector captures strong time-related features in the data. The second principal singular vector curve shows a certain phase difference from the first principal singular vector at the peak and trough positions, exhibiting a different dynamic pattern, and its amplitude changes are relatively balanced. The third principal singular vector curve has a higher oscillation frequency and denser fluctuations, reflecting more subtle dynamic characteristics.

[0031] Overall, the three principal singular vectors exhibit continuous fluctuations between time indices 0 and 580, without any obvious stable regions. This indicates that the latent feature tensors obtained through multi-view recursive canonical tensor decomposition can effectively reflect the dynamic changes in cognitive states implied in multimodal time-series data, providing rich time-varying feature information for subsequent construction of Lie algebraic manifold mappings and covariance feature matrices based on non-Euclidean space. The figure visually illustrates the temporal evolution of latent cognitive state features after multimodal data fusion in Example 1, providing fundamental data support for brain function assessment.

[0032] Example 2:

[0033] In specific implementation, please refer to Figure 2 Each time slice of the latent feature tensor is expanded into a sequence of feature vectors. The latent feature tensor has three dimensions: a first-order principal singular vector index, a second-order time index, and a third-order modality type index. For each fixed time point in the second-order time index of the latent feature tensor, the matrix formed by the first and third orders corresponding to that fixed time point is flattened into a column vector in row-major order. This column vector is the feature vector corresponding to that time point in the feature vector sequence. The feature vectors corresponding to all time points are arranged in chronological order to form the feature vector sequence.

[0034] In practice, the Gram matrix between eigenvectors of every two different time slices in the eigenvector sequence is calculated, resulting in multiple instantaneous covariance matrices. For a eigenvector sequence with time index... eigenvectors and time index eigenvectors The formula for calculating the Gram matrix is: in: Indicates the time index is eigenvectors, Indicates the time index is eigenvectors, express transpose, Indicates by and The calculated Gram matrix, and All are time-indexed and The Gram matrices corresponding to all pairwise distinct time slices in the feature vector sequence are combined to form a set, and each Gram matrix is ​​an instantaneous covariance matrix.

[0035] In practice, multiple instantaneous covariance matrices are projected onto a symmetric positive definite manifold. For each instantaneous covariance matrix, eigenvalue decomposition is performed to obtain an eigenvalue diagonal matrix and an eigenvector matrix. Each eigenvalue in the eigenvalue diagonal matrix is ​​examined; if a negative eigenvalue is found, it is replaced with its absolute value; if an eigenvalue is zero, it is replaced with a preset minimum positive value, which is 10 to the power of -8. The replaced eigenvalue diagonal matrix is ​​then multiplied by the original eigenvector matrix to reconstruct the matrix, which is the instantaneous covariance matrix projected onto the symmetric positive definite manifold. The same projection operation is performed on all instantaneous covariance matrices.

[0036] In practice, the projected instantaneous covariance matrices are subjected to a Riemann weighted average along the time dimension to generate a covariance feature matrix. For a time index of... For the target time slice, the weight coefficients of the Riemann weighted average are inversely proportional to the Mahalanobis distance between the feature vectors of adjacent time slices. The Mahalanobis distance is calculated as follows: for a time slice... With time slices Mahalanobis distance is The square root of, where The covariance matrix of the eigenvectors over the entire time period. For time index The corresponding feature vector, For time index The corresponding feature vector.

[0037] In practice, the weighting coefficients of the Riemann weighted average adopt an exponentially decaying form. For time indexes... Corresponding central slice, time index Weight coefficients of corresponding adjacent slices Determined by the following formula: in: Indicates time index With time index Mahalanobis distance between the corresponding feature vectors The preset time constant. The value is the median of the Mahalanobis distance between adjacent time index feature vectors in all feature vector sequences. Represented by natural constant The weighted average is calculated using an exponential function with base 0. The Riemann mean iterative algorithm is applied to a symmetric positive definite manifold. The initial value is set as the instantaneous covariance matrix corresponding to the projected center time slice. The iterative update rule is as follows: using the current mean point as the tangent space base point, each instantaneous covariance matrix is ​​mapped to the tangent space through a Riemann logarithmic mapping. After calculating the weighted Euclidean average, the manifold is pulled back through an exponential mapping. Iteration stops when the change in Riemann distance between two adjacent iterations is less than 10 to the power of -5, yielding the covariance characteristic matrix. The slice range for the weighted average of the instantaneous covariance matrices of all time slices is a window centered on the center slice, with 10 time slices before and after it, for a total window size of 21 time slices. When the center slice is close to the time series boundary, resulting in a window of less than 21 slices, only the actual existing slices are used.

[0038] See Figure 7 In the figure, the horizontal axis represents the score of the first principal component after dimensionality reduction by principal component analysis of the multi-dimensional evaluation feature vector, and the vertical axis represents the score of the corresponding second principal component. In the legend, circles "〇" represent sample points of category one, and squares "□" represent sample points of category two. The scatter plot shows the distribution of the two classes of samples in the first and second principal component spaces.

[0039] Category 1 sample points are mainly concentrated in the positive interval of the first principal component, with values ​​ranging from approximately 0 to 1.5. The scores of the second principal component are mostly distributed between -1 and 1, showing an overall upward-right shift. Category 1 sample points are relatively concentrated, and the clustering area of ​​this category's samples is marked by dashed ellipses in the figure, showing a relatively obvious concentration trend in the scores of the first and second principal components.

[0040] The sample points of category two are mainly distributed in the negative to near-zero range of the first principal component, approximately -1.0 to 0.5, while the scores of the second principal component are concentrated between -1.0 and 0, with their overall position biased towards the lower left. The clustering regions of category two are marked with dotted ellipses, showing a distribution center location and range significantly different from category one. Furthermore, the distribution range of category two samples is slightly wider than that of category one, indicating that this type of cognitive state variable exhibits greater variability in the principal component space.

[0041] The two types of samples exhibit good discriminatory power in the two-dimensional space of the first and second principal components, with a clear interval between the two clustering regions. The difference in principal component scores between category one and category two reflects that the multi-dimensional evaluation features extracted after multimodal data fusion processing based on the latent feature tensor can effectively distinguish samples of different cognitive state categories. This is consistent with the technical solution in Example 2, which expands the latent feature tensor into a sequence of feature vectors and calculates the covariance feature matrix using the Gram matrix before performing dimensionality reduction and feature extraction, verifying the effectiveness of the feature vectors obtained in identifying cognitive state categories.

[0042] Example 3:

[0043] In specific implementation, please refer to Figure 3 We choose the identity matrix on a symmetric positive definite manifold as the reference point for the tangent space. The dimension of the identity matrix is ​​the same as the dimension of the covariance eigenma matrix, and all diagonal elements of the identity matrix are 1, while all off-diagonal elements are 0. We denote this identity matrix as... .

[0044] In practice, the Riemann logarithmic mapping of the covariance eigenvalue matrix relative to the reference point is calculated. Let the covariance eigenvalue matrix be... The Riemannian logarithmic mapping is performed by solving the geodesic equation between the covariance eigenma matrix and a reference point. The solution uses matrix logarithmic operations, as shown in the formula: in: This represents the matrix resulting from the Riemann logarithmic mapping. Indicated by reference point The covariance characteristic matrix based on the base point The Riemann logarithmic mapping, Indicates the reference point The square root of the matrix, through The eigenvalues ​​are obtained by taking the square root of the eigenvalues ​​after eigenvalue decomposition and then reconstructing the eigenvalues. Indicates the reference point The inverse matrix of the square root of the matrix is ​​obtained by... After eigenvalue decomposition, the square root and reciprocal of the eigenvalues ​​are taken to reconstruct the eigenvalues. This represents the matrix logarithm operation, which is obtained by first performing eigenvalue decomposition on the matrix, then taking the natural logarithm of the eigenvalues, and finally reconstructing the matrix.

[0045] In practice, the lower triangular elements of the Riemann logarithm mapping matrix are extracted and vectorized to obtain the original tangent space vector. During extraction, the lower triangular portion of the resulting matrix, including diagonal elements, is traversed in row-major order. The extracted lower triangular elements are then arranged into a column vector, which is the original tangent space vector. The dimension of the original tangent space vector is... ,in denoted as the order of the covariance characteristic matrix.

[0046] In practice, dimensionality reduction is performed on the original tangent space vectors using kernel principal component analysis (KPI) based on chi-square distance. The kernel function used in KPI is the Gaussian radial basis function (RBF) kernel, and its expression is as follows: ,in and They represent the first The and the first The original tangent space vector of each sample This represents the Euclidean distance between two original tangent space vectors. Let be the kernel parameters of the Gaussian radial basis function kernel. The kernel parameters are optimized by maximizing the ratio of the inter-class scatter matrix to the intra-class scatter matrix of the eigenvectors after dimensionality reduction.

[0047] In the process of optimizing the kernel parameters of the Gaussian radial basis kernel, the kernel parameters are set. The candidate values ​​range from the tenths to the ninetieths of the Euclidean distance between any two pairs of vectors in the original tangent space to an equally spaced grid, with 20 grid points. For each candidate kernel parameter... Kernel principal component analysis was performed for dimensionality reduction, and the top performers with a cumulative variance contribution rate exceeding 0.95 were selected. For each principal component, the ratio of the trace of the inter-class scatter matrix to the trace of the intra-class scatter matrix of the dimensionality-reduced feature vector is calculated. The candidate kernel parameter that maximizes this ratio is selected as the optimal kernel parameter. The inter-class scatter matrix is ​​calculated based on the different cognitive state categories to which the samples belong, and the intra-class scatter matrix is ​​calculated based on the deviation of samples of the same category from their category mean.

[0048] In practice, the principal component components obtained after dimensionality reduction are sequentially concatenated into a multi-dimensional evaluation feature vector. The selected principal component components are then ordered from largest to smallest variance contribution rate. The projection values ​​of the principal component components onto the original tangent space vector are arranged sequentially to form a multidimensional evaluation feature vector. The dimension of the multidimensional evaluation feature vector is equal to the number of selected principal component components. .

[0049] See Figure 8In the graph, the horizontal axis represents the principal component index, with a total of 480 principal components. The left side of the vertical axis uses a logarithmic scale to represent the eigenvalues ​​of each principal component, gradually decreasing from a maximum eigenvalue of approximately 10 to a minimum of approximately 0.001. The right side of the vertical axis uses a linear scale to represent the variance contribution rate obtained by the cumulative algorithm; the curve shows that the cumulative variance contribution rate gradually increases from 0 to close to 1. The gray bars represent the eigenvalues ​​of each principal component, showing a decreasing trend. The eigenvalues ​​of the first 100 principal components are significantly larger than those of subsequent principal components, indicating that the main information is concentrated in the earlier principal components. The black solid line curve represents the change in the cumulative variance contribution rate as the principal component index increases. The curve first exceeds the threshold of 0.95 at approximately the 140th principal component (indicated by the black dashed line), indicating that selecting the first 140 principal components can explain more than 95% of the variance contribution. This figure illustrates the process of dimensionality reduction of the original tangent space vector based on chi-square distance kernel principal component analysis in Example 3. By selecting the number of principal components with a cumulative variance contribution rate exceeding 0.95, effective representation and feature extraction of the potential feature tensor after dimensionality reduction are achieved, laying the foundation for the subsequent construction and analysis of cognitive state variables based on multi-dimensional evaluation feature vectors.

[0050] Example 4:

[0051] In specific implementation, please refer to Figure 4 This approach treats each dimension of the multi-dimensional evaluation feature vector as a cognitive state variable. The multi-dimensional evaluation feature vector is composed of... The principal components are assembled sequentially, and the first one is... The principal component is considered as the first principal component. The cognitive state variable will be the first one. The principal component is considered as the first principal component. The cognitive state variable is denoted as _____, and so on, until the _____ cognitive state variable is denoted as _____. The principal component is considered as the first principal component. There are 1 cognitive state variables. Let all cognitive state variables be denoted as... All cognitive state variables together constitute A 3D state space, where the coordinates of each point are determined by a... dimensional vector express.

[0052] In practical implementation, a total energy function is defined in the state space. The total energy function is the sum of the potential energy term of each cognitive state variable and the coupling potential energy term between any two cognitive state variables. For the ... Cognitive state variables Its potential energy term Using the double-well potential function form, ,in For the first The potential energy intensity coefficient of each cognitive state variable. The value is the first The variance contribution rate of the principal component corresponding to each cognitive state variable in the multidimensional evaluation feature vector is multiplied by 10. Cognitive state variables and the Cognitive state variables The coupling potential term between the two Adopting a harmonic coupling form, ,in For the first The cognitive state variable and the first The coupling strength coefficient between cognitive state variables The value of is obtained by calculating the th in the multi-dimensional evaluation feature vector. The principal component and the first principal component The absolute values ​​of the Pearson correlation coefficients between the principal component components are obtained. Total energy function. The definition of is: in: Represents all cognitive state variables The total energy function constituted This represents the total number of cognitive state variables. Indicates the first The potential energy intensity coefficient of each cognitive state variable. Indicates the first A cognitive state variable, Indicates the first Operation of the double-well potential function for each cognitive state variable. Indicates the first The cognitive state variable and the first The coupling strength coefficient between cognitive state variables Indicates the first A cognitive state variable, Indicates the first A cognitive state variable, Indicates the first The cognitive state variable and the first The square of the difference between the cognitive state variables.

[0053] In practical implementation, a system of equations of motion under Hamiltonian constraints is constructed based on the partial derivatives of the total energy function with respect to each cognitive state variable. For the th Cognitive state variables Introduce a momentum variable that is conjugate to this cognitive state variable. Momentum variable The initial value is set to zero. The equations of motion under Hamiltonian constraints are given by... It consists of several first-order differential equations, for The equations of motion are in the form of: in This represents the evolution time variable.

[0054] In practice, residual terms are introduced into the equations of motion. These residual terms describe external observation noise and unmodeled dynamics. One residual term is added to the differential equation for each momentum variable. residual term The intensity of the modeled zero-mean Gaussian white noise process is... By calculating the first eigenvector in the multi-dimensional evaluation feature vector The estimation error variance of each principal component is determined by using the ratio of the sample variance of the principal component to the signal-to-noise ratio (SNR) estimate. The SNR estimate is taken as the ratio of the absolute value of the sample mean to the sample standard deviation of the principal component. The momentum variable differential equation after introducing the residual term is: in: This indicates that the mean is zero and the autocorrelation function is Dirac. The function is a unit-intensity Gaussian white noise process. For the first The noise intensity of the residual term corresponding to each cognitive state variable.

[0055] In practice, a pre-defined symplectic integrator is used to numerically solve the equations of motion after introducing the residual terms. The pre-defined symplectic integrator adopts a second-order stochastic symplectic scheme, with each iteration containing two half-steps. The first half-step updates the momentum variable in the following manner: ,in The integration step size is set to 0.01. Indicates the first The first iteration One momentum variable value, Indicates the first A vector consisting of all positional variable values ​​during each iteration. This represents an independent random number drawn from a standard normal distribution. The second step updates the position variable using a half-step size, and the update method is as follows: ,in Indicates the first The first iteration The position variable value. The third step, the half-step size, updates the momentum variable again, using the following update method: ,in Indicates and Independent and identically distributed standard normal random numbers, Indicates the first The vector consists of all position variable values ​​during each iteration. The position variables in the initial state vector are taken as values ​​from the corresponding dimensions of the multi-dimensional evaluation feature vector, normalized to the interval. The result is normalized by dividing the value of each dimension in the multi-dimensional evaluation feature vector by the maximum absolute value and then multiplying by 0.9. Starting from the initial state, numerical solutions are obtained according to the above three-step iterative format. After each iteration, the rate of change of the total energy function value relative to the previous iteration is calculated. The rate of change of the total energy function is defined as the ratio of the absolute value of the difference between the total energy function values ​​of two adjacent iterations to the integration step size. Iteration stops when the rate of change of the total energy function is lower than a preset energy gradient threshold, which is set to 10 to the power of -6. The sequence of state space points recorded at each step of the iteration process is used as the convergence trajectory. Each point in the convergence trajectory is determined by... A vector consisting of positional variable values This indicates that the state space points of all iteration steps, including the initial state point and the final state point, are recorded.

[0056] In practice, two state space points corresponding to the initial and final states are extracted from the convergence trajectory. The state space point corresponding to the initial state is the point in the iteration step. The state space point recorded in each step is the state space point corresponding to the final state, which is the state space point recorded in the last step when the iteration satisfies the stopping condition. The state space point corresponding to the initial state is denoted as... The state space point corresponding to the termination state is denoted as .

[0057] In practice, the Riemann geodesic distance between two points in the state space is calculated. The state space is a space containing a metric tensor. 3D space, the metric tensor is taken as The identity matrix, under which the Riemannian geodesic distance degenerates into the Euclidean distance. Riemannian geodesic distance The calculation method is as follows ,in This represents the first point in the state space corresponding to the initial state. The values ​​of the positional variables, This represents the state space point corresponding to the termination state. The values ​​of each location variable are calculated. The average evolution rate is obtained by dividing the calculated Riemann geodesic distance by the number of iterations, which is the total number of iterations required to satisfy the stopping condition. The average evolution rate is output as an additional convergence dynamic feature along with the convergence trajectory. The output data includes the sequence of all state space points in the convergence trajectory, the average evolution rate, and the total number of iterations.

[0058] Example 5:

[0059] In specific implementation, please refer to Figure 5 After receiving the multimodal temporal data set synchronously collected by the subjects while performing a preset cognitive task, the speech signal undergoes cascaded preprocessing based on blind source separation and Wiener filtering. Blind source separation employs an adaptive online algorithm based on second-order statistics to separate multiple independent source signal components from the collected multi-channel speech signal. Wiener filtering uses one of the source signal components obtained from blind source separation as the desired signal, and the sum of the remaining source signal components as the noise reference. A Wiener gain is constructed by estimating the ratio of the noise power spectrum to the noisy signal power spectrum, and adaptive noise reduction is applied to the desired signal component to output a clean speech component. Simultaneously, the filtered noise component is recorded as the environmental noise component. When calculating the short-time energy envelope of the clean speech component, the clean speech component is divided into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. The sum of squares of the clean speech sample values ​​within each frame is calculated to obtain the short-time energy envelope sequence.

[0060] In practice, gaze point recognition based on Kalman filtering and velocity thresholding is performed on the eye movement trajectory coordinate sequence. The Kalman filter employs a constant velocity motion model, where the state vector consists of four components: horizontal coordinates of the eye movement trajectory, vertical coordinates of the eye movement trajectory, horizontal velocity, and vertical velocity. The observation vector consists of the horizontal and vertical coordinates of each sampling time point in the eye movement trajectory coordinate sequence. The process noise covariance matrix is ​​set as a diagonal matrix with diagonal elements of 0.01, and the observation noise covariance matrix is ​​also set as a diagonal matrix with diagonal elements representing the square of the nominal measurement error of the pupil tracking device. The instantaneous velocity is calculated point-by-point for the eye movement trajectory coordinate sequence smoothed by Kalman filtering. The instantaneous velocity is defined as the Euclidean distance between two adjacent sampling points divided by the sampling time interval. Continuous coordinate point segments with instantaneous velocities below a preset velocity threshold are marked as gaze point sequences, and continuous coordinate point segments with instantaneous velocities above the preset velocity threshold are marked as saccade sequence sequences. The preset velocity threshold is set to 30 degrees per second.

[0061] In practice, time-frequency analysis based on empirical mode decomposition (EMD) is performed on the handwriting pressure array. EMD employs a screening process to extract multiple intrinsic mode function (IMF) components and remainder terms from the handwriting pressure array that satisfy the IMF conditions. The screening process terminates when the number of extreme points is less than or equal to 2. The current order of screening stops when the standard deviation between two consecutive IMF components obtained from the screening is less than 0.3. From the multiple IMF components obtained from the decomposition, the Pearson correlation coefficient between the power spectral density of each IMF component and the power spectral density of the original handwriting pressure array is calculated. The IMF component with the largest correlation coefficient is selected as the dominant component related to the handwriting action.

[0062] In practice, the short-time energy envelope, the dwell time statistics of the fixation sequence, and the Hilbert spectrum of the dominant component are timestamped. The dwell time statistics of the fixation sequence are defined as the length of time spanned by each fixation sequence; the Hilbert spectrum is constructed on the time axis by performing a Hilbert transform on the dominant component to obtain the instantaneous amplitude and instantaneous frequency. Using the timestamp of the short-time energy envelope as a reference, the fixation dwell time statistics are linearly interpolated using the time interval between adjacent fixations to transform them into a sequence aligned with the time point of the short-time energy envelope; the Hilbert spectrum of the dominant component is expanded on the time axis in equally spaced frequency slices with a frequency slice interval of 0.5 Hz, and the time signal of each frequency slice is also linearly interpolated according to the timestamp of the short-time energy envelope to form an aligned multimodal time series data set.

[0063] In practice, the system detects whether there are missing data segments or misaligned timestamps in the preprocessed multimodal time-series dataset. Missing data segments are detected by traversing the timestamp sequences in the aligned multimodal time-series dataset. If the difference between adjacent timestamps exceeds a preset maximum allowable interval, a missing data segment is determined to exist between two adjacent timestamps. The maximum allowable interval is three times the median of the preset cognitive task sampling period. Misaligned timestamps are detected by calculating the cross-correlation coefficient between the timestamp sequence corresponding to the speech signal and the timestamp sequence corresponding to the eye-tracking coordinate sequence, as well as the cross-correlation coefficient between the timestamp sequence corresponding to the speech signal and the timestamp sequence corresponding to the handwritten pressure array. A timestamp misalignment is determined to exist when the cross-correlation coefficient is below 0.85.

[0064] In practice, when missing data segments are detected, Gaussian process regression is used to interpolate and fill in the missing segments. The kernel function for Gaussian process regression is a superposition of a radial basis function kernel and a white noise kernel. The length scale parameter of the radial basis function kernel is set to the mean of the time interval between the actual data before and after the missing segment, and the noise variance of the white noise kernel is set to one-tenth of the variance of the actual data before and after the missing segment. Using actual data from 50 time points before and after the missing segment as training samples, the hyperparameters of the Gaussian process regression model are optimized. The optimization objective is to maximize the logarithmic marginal likelihood, and the conjugate gradient method is used. The maximum number of iterations is set to 200. After training convergence, the posterior mean and posterior variance of the Gaussian process are calculated for each time point within the missing segment. The posterior mean is used to fill in the missing segment, while the posterior variance is recorded as a measure of interpolation uncertainty.

[0065] In practice, when a timestamp misalignment is detected, the maximum cross-correlation delay between each pair of the speech signal, eye-tracking coordinate sequence, and handwritten pressure array is calculated. For the short-time energy envelope sequence corresponding to the timestamp of the speech signal and the gaze dwell time statistics sequence corresponding to the timestamp of the eye-tracking coordinate sequence, the normalized cross-correlation function between them is calculated. The delay corresponding to the maximum value of the cross-correlation function is the maximum cross-correlation delay between the speech signal and the eye-tracking coordinate sequence. The maximum cross-correlation delay between the speech signal and the handwritten pressure array is calculated in the same way. Using the timestamp of the speech signal as a reference, the timestamps of the eye-tracking coordinate sequence are resampled and aligned according to the maximum cross-correlation delay between the speech signal and the eye-tracking coordinate sequence. The timestamps of the handwritten pressure array are also resampled and aligned according to the maximum cross-correlation delay between the speech signal and the handwritten pressure array. Resampling is achieved using cubic spline interpolation.

[0066] In practice, the aligned multimodal time-series data set is divided into multiple consecutive time-series data segments according to a fixed-duration time window. The fixed duration is 30 seconds, and there is no overlap between adjacent time windows. Each time-series data segment contains the short-time energy envelope sequence, gaze dwell time statistics sequence, and frequency band energy sequences of the dominant component Hilbert spectrum within the corresponding time window. The time-series data segment serves as the input unit for the multimodal time-series data set in the subsequent first step of processing.

[0067] In practice, the projection sequence of the convergence trajectory output from the fourth step is subjected to Fourier transform on each cognitive state variable dimension. The convergence trajectory contains a sequence of state space points, each state space point consisting of a... The dimensional positional variable vector representation, the first Cognitive state variables The projection sequence is the first step recorded at each step of the convergent trajectory. The positional variable values ​​are arranged in iterative order. A discrete Fourier transform is performed on this projection sequence, and the square of the energy amplitude at each frequency point is calculated to obtain the nth... The frequency domain energy spectrum corresponding to each cognitive state variable.

[0068] In practice, regularized path analysis based on minimum absolute contraction and selection operators is applied to the frequency domain energy spectrum corresponding to each cognitive state variable. The frequency domain energy spectra are then arranged in ascending order of frequency to form the response vector. Construct an identity matrix as the design matrix. The number of columns in the design matrix is ​​the same as the number of frequency points in the frequency domain energy spectrum; solve for the regularization path for the following objective function: in: This represents the frequency domain energy spectrum vector corresponding to a certain cognitive state variable. Each element corresponds to an energy value at a specific frequency point. Indicates size is The identity matrix, This represents the number of frequency points in the frequency domain energy spectrum. This represents the coefficient vector to be solved. Each element corresponds to a regression coefficient at a frequency point. Represents the square of the Euclidean norm. Represents the coefficient vector The sum of the absolute values ​​of all elements in the set. This refers to the regularization penalty parameter. The regularization penalty parameter is used during the solution process. Start with a large initial value and gradually decrease it. Set as The minimum value is set to 0.001 times. A series of coefficient solutions are generated along the regularization path. The regularization penalty parameter is then transferred from... During the descent, the regularization penalty parameter value at which the coefficient corresponding to each frequency point changes from zero to non-zero is used as the entry order marker for that frequency point; the regularization path inflection point is defined as the regularization penalty parameter value corresponding to a jump in the number of vanishing coefficients. The energy threshold corresponding to this inflection point is calculated, i.e., the coefficient vector is taken when the regularization penalty parameter equals the inflection point value. The minimum energy value of the frequency point corresponding to the non-zero element is used as the energy threshold.

[0069] In practice, frequency components with energy values ​​higher than the energy threshold corresponding to the inflection point of the regularization path are extracted as feature frequency bands. A feature frequency band contains a continuous frequency interval in the frequency domain energy spectrum where all energy values ​​are greater than the energy threshold. The average energy value within the feature frequency band for each cognitive state variable is calculated, which is the arithmetic mean of the energy values ​​at all frequency points within the feature frequency band. The reference baseline energy value is obtained by collecting the frequency domain energy spectrum of the corresponding cognitive state variable in the same cognitive domain under resting state or baseline task conditions, and extracting the average energy value within the feature frequency band in the same way. The ratio of the average energy value within the feature frequency band to the reference baseline energy value is calculated and used as the raw evaluation score for that cognitive state variable.

[0070] In practice, the raw evaluation scores of each cognitive state variable are input into their respective pre-trained adaptive logical threshold functions. For each cognitive state variable, the quantile parameters of the corresponding healthy population reference distribution are obtained. Specifically, a sample of raw evaluation scores of the cognitive state variables calculated from a group of healthy individuals performing the same preset cognitive task is collected. The raw evaluation score samples are arranged in ascending order, and the 25th percentile is taken as the first quantile, and the 75th percentile is taken as the third quantile. Based on the quantile parameters, the tilt coefficient and translation coefficient of the adaptive logical threshold function are constructed. The construction method is to take the value 4 and divide it by the difference between the third quantile and the first quantile, and the skewness coefficient is... The shift coefficient characterizes the central tendency of the distribution of original assessment scores in healthy individuals. The median of the original assessment scores of the healthy population is taken as the threshold. The adaptive logical threshold function has the following form: in: Represents the raw assessment score of a certain cognitive state variable. This represents the skewness coefficient of the adaptive logical threshold function corresponding to the cognitive state variable. This represents the shift coefficient of the adaptive logical threshold function corresponding to the cognitive state variable. Represented by natural constant Exponential operations with base 0.05. The result of the adaptive logic threshold function calculation. Mapped to a preset numerical range, where the preset numerical range is set to... The mapping method is to Multiply by 100 to obtain the quantitative assessment indicator corresponding to this cognitive state variable. The magnitude of the quantitative assessment indicator reflects the degree to which the cognitive state deviates from the reference distribution of healthy individuals; the closer the value is to 100, the closer the cognitive function is to the typical level of healthy individuals.

[0071] See Figure 9 In the graph, the horizontal axis represents the original assessment score, ranging from approximately 55 to 85, and the vertical axis represents the quantitative assessment indicator, ranging from 0 to 100. The curve represents a mapping relationship based on an adaptive logical threshold function, exhibiting an overall S-shaped trend that rapidly decreases from high to low values, with the most dramatic changes occurring in the range of approximately 63 to 70 for the original assessment score. The slope coefficient and location parameters of this logical threshold function are adaptively determined based on the distribution characteristics of the healthy population.

[0072] The vertical lines in the graph indicate the key quantile positions of the raw assessment scores in healthy individuals: the dashed line represents the 25th percentile, located at approximately 63.8; the dotted-dashed line represents the median, located at approximately 66.4; and the dotted-dotted line represents the 75th percentile, located at approximately 69.1. The current participant's raw assessment score is marked with a diamond, and its value is approximately 67.5, placing it between the median and the 75th percentile in healthy individuals.

[0073] According to the figure, although the current subject's original assessment score is close to or above the median of the healthy population, it is still within the rapidly decreasing range of the adaptive logic threshold function. The mapped quantitative assessment indicator is approximately 20, significantly lower than the full score of 100, indicating that the subject's cognitive function status deviates to some extent from that of the healthy reference group. The overall mapping curve and quantile indicators fully demonstrate the technical solution in Embodiment 5 of this invention, which uses a pre-trained adaptive logic threshold function to convert the original assessment score into a quantitative assessment indicator, effectively achieving quantitative assessment of cognitive status and precise differentiation of health status.

[0074] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A brain function assessment method based on multimodal data fusion, characterized in that, include: The system receives a set of multimodal temporal data collected synchronously when the subject performs a preset cognitive task. The set of multimodal temporal data includes at least speech signals, eye movement trajectory coordinate sequences, and handwriting pressure arrays. The first step of the multimodal temporal data set is performed based on multi-view recursive canonical tensor decomposition to extract latent feature tensors related to cognitive states. A second step of processing is performed on the latent feature tensor based on a Lie algebraic manifold mapping in non-Euclidean space to generate a covariance feature matrix on a symmetric positive definite manifold. A third step based on the Riemann logarithmic mapping is performed on the covariance feature matrix to compute multidimensional evaluation feature vectors in the tangent space of the symmetric positive definite manifold; The fourth step of the process, which is to solve the residual dynamics system based on Hamiltonian constraints, is performed on the multidimensional evaluation feature vector to obtain the convergence trajectory that reflects the collaborative state of each cognitive domain. The convergence trajectory is processed in a fifth step, which involves adaptive threshold segmentation based on regularized paths, to output quantitative assessment indicators for each cognitive domain.

2. The brain function assessment method based on multimodal data fusion according to claim 1, characterized in that, The first step of processing the multimodal temporal data set based on multi-view recursive canonical tensor decomposition to extract latent feature tensors related to cognitive states includes: The speech signal, the eye-track coordinate sequence, and the handwriting pressure array are arranged in chronological order as a first time-series vector, a second time-series vector, and a third time-series vector, respectively. The first time-series vector, the second time-series vector, and the third time-series vector are respectively mapped to a high-dimensional Hilbert space to obtain a first reconstructed phase space trajectory matrix, a second reconstructed phase space trajectory matrix, and a third reconstructed phase space trajectory matrix. The first reconstructed phase space trajectory matrix, the second reconstructed phase space trajectory matrix, and the third reconstructed phase space trajectory matrix are stacked along the third dimension to construct a third-order source tensor. A recursive canonical decomposition constraint is applied to the third-order source tensor to decompose the third-order source tensor into a product of a core tensor and three factor matrices, wherein the recursive canonical decomposition constraint forces the core tensor to form a low-rank cyclic structure along the time dimension. After expanding the core tensor along the time dimension, the principal singular vector is extracted and recombined into the latent feature tensor.

3. The brain function assessment method based on multimodal data fusion according to claim 2, characterized in that, When mapping the first time-series vector, the second time-series vector, and the third time-series vector to a high-dimensional Hilbert space, a radial basis function kernel is used for embedding, and the embedding dimension is adaptively determined by the pseudo nearest neighbor method.

4. The brain function assessment method based on multimodal data fusion according to claim 2, characterized in that, The second step of processing the latent feature tensor, based on a Lie algebraic manifold mapping in non-Euclidean space, to generate a covariance feature matrix located on a symmetric positive definite manifold, includes: Each time slice of the latent feature tensor is expanded into a sequence of feature vectors; Calculate the Gram matrix between the feature vectors of every two different time slices in the feature vector sequence to obtain multiple instantaneous covariance matrices; The multiple instantaneous covariance matrices are projected onto a symmetric positive definite manifold. The projection operation is performed by performing eigenvalue decomposition on each instantaneous covariance matrix, replacing its negative eigenvalues ​​with their absolute values, and then reconstructing it. The projected instantaneous covariance matrices are subjected to a Riemann weighted average along the time dimension to generate the covariance feature matrix, wherein the weight coefficients of the Riemann weighted average are inversely proportional to the Mahalanobis distance between the feature vectors of adjacent time slices.

5. The brain function assessment method based on multimodal data fusion according to claim 4, characterized in that, The weight coefficients of the Riemann weighted average adopt an exponential decay form, which makes the weight of adjacent slices with a larger Mahalanobis distance from the current time slice smaller, and the decay rate is controlled by a preset time constant.

6. The brain function assessment method based on multimodal data fusion according to claim 4, characterized in that, A third step based on the Riemann logarithmic mapping is performed on the covariance eigenma to compute multidimensional evaluation eigenvectors within the tangent space of the symmetric positive definite manifold, including: The identity matrix on the symmetric positive definite manifold is selected as the reference point of the tangent space; The Riemann logarithmic mapping of the covariance feature matrix relative to the reference base point is calculated, and the Riemann logarithmic mapping is performed by solving the geodesic equation between the covariance feature matrix and the reference base point; The resulting matrix of the Riemann logarithmic mapping is subjected to lower triangular element extraction and vectorization to obtain the original tangent space vector. The original tangent space vector is subjected to kernel principal component analysis based on chi-square distance for dimensionality reduction. The principal component components obtained after dimensionality reduction are then concatenated sequentially to form the multidimensional evaluation feature vector.

7. The brain function assessment method based on multimodal data fusion according to claim 6, characterized in that, The kernel function used in the kernel principal component analysis dimensionality reduction is the Gaussian radial basis kernel, and the kernel parameters are optimized by maximizing the ratio of the inter-class scatter matrix to the intra-class scatter matrix of the eigenvectors after dimensionality reduction.

8. The brain function assessment method based on multimodal data fusion according to claim 6, characterized in that, The fourth step of the process, which involves solving the residual dynamics system based on Hamiltonian constraints, is performed on the multi-dimensional evaluation feature vector to obtain the convergence trajectory reflecting the collaborative state of each cognitive domain, including: Each dimension in the multi-dimensional evaluation feature vector is regarded as a cognitive state variable, and a state space composed of multiple cognitive state variables is constructed. A total energy function is defined on the state space, which is composed of the sum of the potential energy term of each cognitive state variable and the coupling potential energy term between any two cognitive state variables; Based on the partial derivatives of the total energy function with respect to each cognitive state variable, a system of motion equations under Hamiltonian constraints is constructed. A residual term is introduced into the equations of motion, which is used to describe external observation noise and unmodeled dynamics; A preset symplectic integrator is used to numerically solve the system of motion equations after introducing residual terms. The process iterates from the initial state until the rate of change of the total energy function is lower than the preset energy gradient threshold. The sequence of state space points recorded during the iteration process is used as the convergence trajectory.

9. The brain function assessment method based on multimodal data fusion according to claim 8, characterized in that, A fifth step of processing, based on adaptive threshold segmentation using regularized paths, is performed on the convergence trajectory to output quantitative evaluation indicators for each cognitive domain, including: The projection sequence of the convergence trajectory on each cognitive state variable dimension is subjected to Fourier transform to obtain the frequency domain energy spectrum corresponding to each cognitive state variable. Apply regularized path analysis based on minimum absolute contraction and selection operators to the frequency domain energy spectrum corresponding to each cognitive state variable, and extract the frequency components with energy values ​​higher than the energy threshold corresponding to the inflection point of the regularized path as feature frequency bands; The ratio of the average energy value within the characteristic frequency band of each cognitive state variable to the reference baseline energy value is calculated to obtain the original assessment score of that cognitive state variable; The original evaluation scores of each cognitive state variable are input into their respective pre-trained adaptive logic threshold functions, and the output of the adaptive logic threshold functions is the quantitative evaluation indicator of each cognitive domain.

10. A brain function assessment system based on multimodal data fusion, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the brain function assessment method based on multimodal data fusion as described in any one of claims 1 to 9.

Images