Immersive VR psychological detection system and method based on multi-modal AI

Abstract

The invention relates to an immersive VR psychological detection system and method based on multi-modal AI. The system comprises a data acquisition and processing module which is used for acquiring a multi-modal data set of a user in a virtual reality scene based on unified clock synchronization, performing time-space alignment and noise reduction standardization processing on the multi-modal data set, and extracting key biological characteristics. And the correlation model construction module performs space-time correlation mapping through a spatial transformation network, constructs a three-dimensional space attention model, and generates a real-time fluctuation curve after inputting the key biological characteristics into the trained model. And the state report generation module identifies a real-time fluctuation curve by using a time sequence analysis model, performs backtracking analysis in combination with the psychological state conversion node and a multi-modal cross validation result, and finally generates a three-dimensional interactive report. By adopting the method, multi-modal data fusion can be realized, the dynamic change of the psychological state of the user can be effectively captured, the psychological state of the user can be comprehensively and deeply analyzed, and a scientific basis is provided for psychological health assessment and intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent mental health detection, and in particular relates to an immersive VR psychological detection system and method based on multimodal AI. Background Art

[0002] With the development of intelligent mental health testing technology, immersive VR psychological testing based on multimodal AI has emerged. Current mental health testing primarily relies on traditional questionnaires and single biosignal acquisition instruments, which present significant limitations. Traditional questionnaires require participants to actively complete them, which can easily distort results due to subjective misinterpretation and misunderstandings, and they fail to capture momentary emotional fluctuations. Single EEG and eye movement monitoring devices offer limited data dimensions and weak interference resistance. While multi-device combinations can expand data types, they suffer from timeline misalignment caused by asynchronous acquisition frequencies (e.g., 1000Hz for EEG signals and 120Hz for eye movement data), which distorts analysis results. Furthermore, the equipment costs, which can reach hundreds of thousands of yuan, hinder widespread adoption in primary care settings. While existing VR psychological testing systems can create immersive scenarios, they only record superficial data, such as operational responses, lacking in-depth integration and analysis with physiological signals. Furthermore, data transmission in the cloud poses privacy risks. These technical bottlenecks hinder traditional solutions from meeting the practical needs of clinical assisted diagnosis, large-scale mental health screening, and personalized interventions in terms of accuracy, cost-effectiveness, adaptability, and user experience. Summary of the Invention

[0003] Based on this, it is necessary to provide an immersive VR psychological detection system and method based on multimodal AI that can more accurately, efficiently and safely evaluate the user's psychological state in response to the above technical problems.

[0004] In the first aspect, this application provides an immersive VR psychological detection system based on multimodal AI, including:

[0005] The data acquisition and processing module is used to obtain the user's multimodal dataset in the virtual reality scene based on a unified clock synchronization; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data and operation behavior data; it is also used to perform spatiotemporal alignment and noise reduction standardization on the multimodal dataset to extract key biometric features.

[0006] The association model construction module is used to build a three-dimensional spatial attention model based on the response time difference of key biometric features in the scene, and to realize spatiotemporal association mapping through a spatial transformation network; it is also used to input key biometric features into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve; the real-time fluctuation curve includes a pressure index curve and a concentration index curve.

[0007] The status report generation module is used to identify real-time fluctuation curves using a time series analysis model, and to generate a three-dimensional interactive report based on retrospective analysis of psychological state transition nodes and multimodal cross-validation results.

[0008] In one embodiment, a multimodal dataset is subjected to spatiotemporal alignment and noise reduction normalization to extract key biometric features, including:

[0009] Obtain EEG signal data and pupil data from a multimodal dataset; the EEG signal data contains a raw potential sequence marked with a timestamp.

[0010] The original potential sequence and pupil data are matched in time window according to the timestamp mark, and the synchronized signal sequence after time and space alignment is obtained by using the interpolation algorithm.

[0011] A noise distribution model is generated based on the synchronous signal sequence through preset frequency domain filtering parameters and dynamic threshold calibration.

[0012] The noise distribution model is used to optimize the synchronization signal sequence and output the noise reduction weight matrix.

[0013] The spatial coordinates of the gaze trajectory data and the operation behavior data are normalized according to the denoising weight matrix to obtain a standardized set of biological feature vectors.

[0014] Feature dimensions that meet preset association rules are extracted from the set of biometric feature vectors and fused to generate key biometric features; key biometric features include EEG waveform patterns, pupil diameter change characteristics, gaze trajectory spatial distribution characteristics, and operation reaction speed characteristics.

[0015] In one embodiment, feature dimensions that meet preset association rules are extracted from a set of biometric feature vectors and fused to generate key biometric features, including:

[0016] The set of biological feature vectors is screened according to the support threshold in the preset association rules to obtain a high-confidence feature subset; the preset association rules are established based on the dynamic coupling relationship between multimodal data.

[0017] The feature weights are calculated for the high-confidence feature subset to generate a fusion coefficient sequence for each dimension.

[0018] A dynamic weighted algorithm is used to perform dot product operations on the fusion coefficient sequence and the high-confidence feature subset to obtain the initial key biometric features.

[0019] The initial key biometric features are input into the preset association rule verification model to obtain the final key biometric features that meet the confidence threshold.

[0020] In one embodiment, based on the response time difference of key biometric features in a scene, a three-dimensional spatial attention model is constructed by implementing spatiotemporal correlation mapping through a spatial transformer network, including:

[0021] Obtain time series data of biometric features in the scene; time series data includes response time differences at different spatial locations.

[0022] Obtain the sight focus coordinate sequence, and use the coordinate difference between adjacent timestamps to calculate the coordinate offset in the sight focus coordinate sequence.

[0023] The time series data and coordinate offsets are input to the spatial transformer network, which outputs an encoded tensor containing spatiotemporal correlation features.

[0024] The frequency band energy values of the EEG active areas are obtained by decomposing the EEG signal data through wavelet transform.

[0025] The coding tensor and the frequency band energy value are fused to generate a multimodal feature vector.

[0026] The path curvature in the gaze trajectory data is calculated by the rate of change of the tangent angle between adjacent movement coordinates.

[0027] The multimodal feature vector is associated with the path curvature to obtain the attention weight matrix.

[0028] The three-dimensional spatial coordinates are weighted according to the attention weight matrix to generate a three-dimensional spatial attention model of the activation area distribution.

[0029] In one embodiment, the association model building module further includes:

[0030] The key biological features are deconstructed and reconstructed to obtain the corresponding multi-channel physiological signal sequence; the physiological signal sequence includes heart rate variability and brain wave frequency band energy value.

[0031] The time-frequency domain feature vector is extracted based on the physiological signal sequence; the time-frequency domain feature vector includes the energy proportion of the pressure-related frequency band and the phase difference of the concentration-related frequency band.

[0032] The time-frequency domain feature vector is input into the three-dimensional spatial attention model to generate an attention weight distribution map.

[0033] The time-frequency domain feature vectors are weightedly fused according to the attention weight distribution map to obtain the fused feature tensor.

[0034] A bidirectional long short-term memory network is used to process the fused feature tensor and output the time series of stress indicators and concentration indicators.

[0035] The pressure index time series is normalized using a sliding window to generate a pressure index curve.

[0036] Perform dynamic threshold segmentation processing on the concentration index time series to generate a concentration index curve.

[0037] In one embodiment, a time series analysis model is used to identify real-time fluctuation curves, and a retrospective analysis is performed based on psychological state transition nodes and multimodal cross-validation results to generate a three-dimensional interactive report, including:

[0038] A sliding time window is used to segment the time series data stream in the real-time fluctuation curve, and the time domain and frequency domain feature vectors in each time window are extracted.

[0039] According to the preset mental state classification threshold, it is judged whether the feature vector triggers the state transition node and a transition event sequence with a timestamp is generated.

[0040] The conversion event sequence is timestamp-aligned with the multimodal cross-validation result dataset to obtain the aligned validation event set.

[0041] Based on the verification event set, the association rules of state transition are mined and the rule confidence matrix is generated.

[0042] The rule confidence matrix is visualized in three dimensions to obtain a thermal layer that dynamically renders the psychological state transition path.

[0043] According to the spatial mapping relationship between the thermal layer and the real-time fluctuation curve, the interactive viewing angle parameters are adjusted and a three-dimensional interactive report is output; the three-dimensional interactive report includes a spatial overlay map of EEG active areas and gaze trajectories, a temporal correlation map of physiological data and scene events, and a color-coded continuous distribution map of psychological states.

[0044] In one embodiment, the status report generation module further includes:

[0045] The three-dimensional interactive report is screened according to the preset key event detection model to obtain the time points that meet the dynamic threshold and obtain the target time point set.

[0046] The 3D scene rendering frame data corresponding to the target time point is intercepted to generate a scene screenshot sequence with a time stamp.

[0047] Extract the emotion feature vector matching the target time point from the continuous distribution map of mental states.

[0048] The visual feature vectors and emotional feature vectors of the scene screenshot sequence are multimodally aligned to obtain the fused state association data.

[0049] The preset intervention rule library is queried based on the state-related data. If there is a feature combination that continuously triggers the intervention conditions in the state-related data, an intervention strategy list with priority labels is generated; the intervention rule library contains the mapping relationship between scene features and psychological states.

[0050] The intervention strategy list is matched with the user's historical feedback data for similarity, and a set of personalized intervention suggestions is output.

[0051] Secondly, this application also provides an immersive VR psychological detection method based on multimodal AI, the method comprising:

[0052] Based on unified clock synchronization, a multimodal dataset of users in virtual reality scenes is obtained; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data and operation behavior data; the multimodal dataset is subjected to spatiotemporal alignment and noise reduction standardization processing to extract key biological features.

[0053] Based on the response time difference of key biometric features in the scene, a three-dimensional spatial attention model is constructed through spatiotemporal correlation mapping via a spatial transformation network. The key biometric features are input into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve. The real-time fluctuation curve includes a pressure index curve and a concentration index curve.

[0054] The time series analysis model is used to identify real-time fluctuation curves, and retrospective analysis is performed based on the psychological state transition nodes and multimodal cross-validation results to generate a three-dimensional interactive report.

[0055] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above system when executing the computer program.

[0056] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the above system when executed by a processor.

[0057] The aforementioned multimodal AI-based immersive VR psychological assessment system, method, apparatus, computer device, and storage medium utilize a data acquisition and processing module, which uses a unified clock synchronization to acquire multimodal datasets including EEG signal data, pupil data, gaze trajectory data, and operational behavior data from users in virtual reality scenarios. These datasets are then subjected to spatiotemporal alignment, noise reduction, and normalization to extract key biometric features. The correlation model construction module uses a spatial transformation network to perform spatiotemporal correlation mapping based on the response time differences of key biometric features within the scenario, constructing a three-dimensional spatial attention model. After inputting the key biometric features into this trained model, real-time fluctuation curves are generated, including stress and concentration index curves. The status report generation module utilizes a time series analysis model to identify real-time fluctuation curves, performs retrospective analysis based on psychological state transition nodes and multimodal cross-validation results, and ultimately generates a three-dimensional interactive report. This multimodal data fusion effectively captures the dynamic changes in the user's psychological state, facilitates comprehensive and in-depth analysis of the user's psychological state, and provides a scientific basis for psychological health assessment and intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 This is a structural block diagram of the multimodal AI-based immersive VR psychological detection system provided by an embodiment of the present invention;

[0060] Figure 2 A flowchart of an embodiment of the present invention for identifying real-time fluctuation curves using a time series analysis model, performing retrospective analysis based on psychological state transition nodes and multimodal cross-validation results, and generating a three-dimensional interactive report;

[0061] Figure 3 This is a flowchart of the multimodal AI-based immersive VR psychological detection method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] In one embodiment, Figure 1 As shown, this application provides an immersive VR psychological detection system based on multimodal AI, which may include:

[0064] The data acquisition and processing module 101 is used to synchronously acquire a multimodal dataset of the user in a virtual reality scene based on a unified clock; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data, and operation behavior data; and is also used to perform spatiotemporal alignment and noise reduction standardization on the multimodal dataset to extract key biometric features.

[0065] Specifically, the module initializes the VR headset, EEG detection headband, high-precision eye tracker, and motion capture sensor through a unified clock system. After the user enters the preset VR game scene, the detection level is automatically triggered according to the plot progress, and EEG signal data (raw potential sequence with timestamp mark), pupil data (pupil zoom trajectory), gaze trajectory data (spatial coordinate sequence of line of sight focus), and operation behavior data (reaction speed, operation accuracy, etc.) are synchronously collected. In view of the frequency difference of multimodal data (such as 1000Hz EEG signal and 60Hz eye movement data), an interpolation algorithm is used to align the time axis. Frequency domain filtering is used to remove the interference of head micro-movements in EEG signals, correct the eye movement data offset caused by VR lens distortion, and filter out unconscious operation noise in behavioral data. Finally, the metadata index library is used to structure the storage of device parameters, scene characteristics and other information, and extract key biometric features such as EEG emotion waveforms, pupil diameter changes, gaze trajectory spatial distribution, and operation reaction speed to form a standardized biometric feature vector set.

[0066] The association model construction module 102 is used to construct a three-dimensional spatial attention model based on the response time difference of key biometric features in the scene, and to implement spatiotemporal association mapping through a spatial transformation network; it is also used to input key biometric features into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve; the real-time fluctuation curve includes a pressure index curve and a concentration index curve.

[0067] Specifically, based on the time series data of key biometric features in VR scenes (including the response time difference of each modality) and the offset of the focus coordinate, a spatial transformation network is used to realize spatiotemporal correlation mapping. Combined with the energy value of the EEG active area frequency band decomposed by wavelet transform and the curvature of the gaze trajectory path (calculated by the rate of change of the tangent angle of adjacent coordinates), an attention weight matrix is generated and a three-dimensional spatial attention model is constructed (integrating the focus of sight and the EEG active area). After the key biometric features are input into the model, multi-channel physiological signals such as heart rate variability and EEG frequency band energy are reconstructed through feature inversion, and time-frequency domain features such as the energy proportion of pressure-related frequency bands and the phase difference of concentration-related frequency bands are extracted. After processing by a bidirectional long-short-term memory network, sliding window normalization, and dynamic threshold segmentation, a pressure index curve and a concentration index curve are generated, and game scene events corresponding to abnormal fluctuations (such as prop triggering and level switching) are annotated in real time.

[0068] The status report generation module 103 is used to identify the real-time fluctuation curve using the time series analysis model, and to generate a three-dimensional interactive report by performing a retrospective analysis based on the psychological state transition nodes and the multimodal cross-validation results.

[0069] Specifically, a time series analysis model is used to segment the real-time fluctuation curve into sliding time windows. Time-frequency feature vectors within each window are extracted. Based on preset psychological state classification thresholds, state transition nodes, such as calm-stress, are identified, generating a time-stamped transition event sequence. Calibration results are obtained through a multimodal cross-validation mechanism (retracing data from other modalities when a single sensor is abnormal). The validated event set is then aligned with the scene interaction log, and state transition association rules are mined to generate a three-dimensional heat map (dynamically rendering the psychological state transition path). Combining the heat map with the spatial mapping relationship between the heat map and the curve, a three-dimensional interactive report is generated, including a spatial overlay of EEG activity areas and gaze trajectories, a temporal correlation diagram of physiological data and game events (with timeline zooming), and a color-coded continuous distribution map of psychological states. Furthermore, a key event detection model is used to identify high-risk time points. Scene screenshots are captured and matched with emotional feature vectors, and an intervention rule library is queried to generate prioritized personalized intervention recommendations (such as identifying levels requiring relaxation and adjustment).

[0070] In this multimodal AI-based immersive VR psychological assessment system, the data acquisition and processing module uses a unified clock synchronization to acquire multimodal datasets, including EEG signal data, pupil data, gaze trajectory data, and operational behavior data, within the user's virtual reality environment. These datasets are then subjected to spatiotemporal alignment, noise reduction, and normalization to extract key biometric features. The correlation model construction module uses a spatial transformation network to perform spatiotemporal correlation mapping based on the response time differences of key biometric features within the environment, constructing a three-dimensional spatial attention model. After inputting the key biometric features into this trained model, real-time fluctuation curves, including stress and concentration index curves, are generated. The status report generation module uses a time series analysis model to identify real-time fluctuation curves, performs retrospective analysis based on psychological state transition nodes and multimodal cross-validation results, and ultimately generates a three-dimensional interactive report. This multimodal data fusion effectively captures the dynamic changes in the user's psychological state, facilitates comprehensive and in-depth analysis of the user's psychological state, and provides a scientific basis for psychological health assessment and intervention.

[0071] In one embodiment, performing spatiotemporal alignment and noise reduction normalization on a multimodal dataset to extract key biometric features may include the following steps:

[0072] Step S201 , obtaining EEG signal data and pupil data in a multimodal data set; the EEG signal data includes a raw potential sequence marked with a timestamp.

[0073] Step S202 : performing time window matching on the original potential sequence and the pupil data according to the timestamp mark, and obtaining a synchronization signal sequence after time and space alignment using an interpolation algorithm.

[0074] Step S203 : generating a noise distribution model based on the synchronization signal sequence by calibrating preset frequency domain filtering parameters and dynamic thresholds.

[0075] Step S204: Optimize the synchronization signal sequence using the noise distribution model and output a noise reduction weight matrix.

[0076] Step S205 , performing spatial coordinate normalization on the gaze trajectory data and the operation behavior data according to the noise reduction weight matrix to obtain a normalized set of biological feature vectors.

[0077] Step S206: extract feature dimensions that meet preset association rules from the biometric feature vector set and fuse them to generate key biometric features; the key biometric features include EEG waveform patterns, pupil diameter change features, gaze trajectory spatial distribution features, and operation reaction speed features.

[0078] Preferably, EEG signal data and pupil data in a multimodal data set are obtained, wherein the EEG signal data includes a raw potential sequence marked with a timestamp. Time window matching is performed on the raw potential sequence and the pupil data according to the timestamp mark, and the signals of different frequencies are unified to the same time scale using an interpolation algorithm to obtain a synchronized signal sequence after time and space alignment. Based on the synchronized signal sequence, a noise distribution model is generated by calibrating the preset frequency domain filter parameters and the dynamic threshold, and the synchronized signal sequence is optimized using the model to output a noise reduction weight matrix. According to the noise reduction weight matrix, the spatial coordinates of the gaze trajectory data and the operation behavior data are normalized to eliminate device errors and environmental interference, and a standardized set of biometric feature vectors is obtained. Finally, the feature dimensions that meet the preset association rules in the set are extracted, and the key biometric features including EEG waveform patterns, pupil diameter change features, gaze trajectory spatial distribution features, and operation reaction speed features are generated by fusion.

[0079] This embodiment solves the problem of asynchronous multimodal data acquisition frequencies through time axis alignment technology, ensuring that EEG, pupil, and other data accurately correspond in the time dimension. It uses frequency domain filtering and dynamic threshold calibration to construct a noise distribution model, effectively removing interference factors such as head micro-movements and VR lens distortion, thereby improving signal purity. It also standardizes gaze trajectory and operational behavior data through spatial coordinate normalization, providing a unified benchmark for cross-modal feature fusion. This improves the spatiotemporal consistency and reliability of multimodal data, ensuring that key biometric features can truly reflect the user's psychological changes in VR scenarios.

[0080] In one embodiment, extracting feature dimensions that meet preset association rules from a set of biometric feature vectors and fusing them to generate key biometric features may include the following steps:

[0081] Step S301 , screening a set of biometric feature vectors according to a support threshold in a preset association rule to obtain a high-confidence feature subset; the preset association rule is established based on a dynamic coupling relationship between multimodal data.

[0082] Step S302: Calculate feature weights for the high-confidence feature subset to generate a fusion coefficient sequence for each dimension.

[0083] Preferably, the corresponding feature weight is calculated using the following formula:

[0084]

[0085] Among them, ω j represents the feature weight of the jth feature dimension, E j represents the information entropy of the j-th feature dimension, represents the sum of information gains of m feature dimensions, ρ jk represents the Pearson correlation coefficient between feature dimensions j and k, |ρ jk | represents the linear correlation between feature dimension j and other features. The larger the value, the higher the redundancy between features. Measures the sum of the correlations between feature dimension j and all features, and its inverse is used to suppress the weights of high-correlation features.

[0086] In step S303 , a dynamic weighting algorithm is used to perform a dot product operation on the fusion coefficient sequence and the high confidence feature subset to obtain the initial key biometric features.

[0087] Step S304: input the initial key biometric features into a preset association rule verification model to obtain the final key biometric features that meet the confidence threshold.

[0088] Specifically, the set of biometric feature vectors is screened based on the support threshold in the preset association rules to obtain a high-confidence feature subset based on the dynamic coupling relationship between multimodal data. Feature weights are calculated for the high-confidence feature subset to generate a fusion coefficient sequence for each dimension. A dynamic weighting algorithm is used to perform a dot product operation on the fusion coefficient sequence and the high-confidence feature subset to obtain the initial key biometric features. The initial key biometric features are then input into the preset association rule verification model. Through model verification, the final key biometric features that meet the confidence threshold are selected to achieve an optimized combination of feature dimensions.

[0089] By using support thresholds to filter high-confidence feature subsets, low-correlation features in multimodal data are eliminated, reducing redundant information interference. Feature weights are calculated based on dynamic coupling relationships and fusion coefficients are generated to ensure that the weight distribution of feature dimensions conforms to the inherent correlation patterns of the data. Initial features are rechecked using an association rule verification model to eliminate feature combinations with accidental associations, ensuring the reliability of the final key biometric features. This improves the scientific nature and effectiveness of feature fusion, enabling extracted key biometric features such as EEG waveform patterns and pupil diameter change characteristics to more accurately represent the user's psychological state.

[0090] In one embodiment, based on the response time difference of key biometric features in a scene, a three-dimensional spatial attention model is constructed by implementing spatiotemporal correlation mapping through a spatial transformer network, which may include the following steps:

[0091] Step S401, obtaining time series data of the biometric feature in the scene; the time series data includes response time differences at different spatial positions.

[0092] Step S402: Obtain a sight focus coordinate sequence, and calculate a coordinate offset in the sight focus coordinate sequence using the coordinate difference between adjacent timestamps.

[0093] Step S403: input the time series data and coordinate offset into the spatial transformation network, and output a coding tensor containing spatiotemporal correlation features.

[0094] Step S404: Decomposing the EEG signal data by wavelet transform to obtain the frequency band energy value of the EEG active area.

[0095] Step S405 : performing feature fusion on the coding tensor and the frequency band energy value to generate a multimodal feature vector.

[0096] Step S406 , calculating the path curvature in the gaze trajectory data by the tangent angle change rate of adjacent movement coordinates.

[0097] Step S407: perform correlation mapping between the multimodal feature vector and the path curvature to obtain an attention weight matrix.

[0098] Step S408: weighting the three-dimensional spatial coordinates according to the attention weight matrix to generate a three-dimensional spatial attention model of the activation area distribution.

[0099] Specifically, the method first obtains the time series data of the biometric features in the scene, which contains the response time difference of different spatial positions. At the same time, it obtains the coordinate sequence of the gaze focus and calculates the coordinate offset of the gaze focus through the coordinate difference of adjacent timestamps. The time series data and the coordinate offset are input into the spatial transformation network, and the coding tensor containing the spatiotemporal correlation features is output. The frequency band energy value of the EEG active area is decomposed from the EEG signal data through wavelet transform, and the feature is fused with the coding tensor to generate a multimodal feature vector. At the same time, the path curvature of the gaze trajectory data is calculated by the tangent angle change rate of adjacent moving coordinates. The multimodal feature vector is associated with the path curvature and mapped to obtain the attention weight matrix. Finally, the three-dimensional spatial coordinates are weighted according to the matrix to generate a three-dimensional spatial attention model of the activation area distribution.

[0100] This embodiment achieves precise positioning of EEG active areas; the model construction method effectively integrates the dynamic changes of time series and the three-dimensional spatial distribution characteristics, so that the attention weight matrix can accurately reflect the correlation between the user's psychological state and spatial interaction in the VR scene, and provides a spatial dimension analysis basis for the real-time generation of psychological indicator curves such as stress and concentration, significantly improving the system's dynamic analysis capability and spatial positioning accuracy for psychological states in complex scenes.

[0101] In one embodiment, the association model building module may include:

[0102] Step S501 : performing feature inversion and signal reconstruction on key biological features to obtain corresponding multi-channel physiological signal sequences; the physiological signal sequences include heart rate variability and brain wave frequency band energy values.

[0103] Step S502 , extracting a time-frequency domain feature vector based on the physiological signal sequence; the time-frequency domain feature vector includes the energy proportion of the pressure-related frequency band and the phase difference of the concentration-related frequency band.

[0104] Step S503: Input the time-frequency domain feature vector into the three-dimensional spatial attention model to generate an attention weight distribution map.

[0105] Step S504: perform weighted fusion on the time-frequency domain feature vectors according to the attention weight distribution map to obtain a fused feature tensor.

[0106] Step S505: Use a bidirectional long short-term memory network to process the fused feature tensor and output a time series sequence of stress indicators and a time series sequence of concentration indicators.

[0107] Preferably, the fused feature tensor is dimensionally adapted to obtain three-dimensional sequence data suitable for the input of the bidirectional long short-term memory network; the fused feature tensor is adjusted to the format of "number of samples × time step × feature dimension" to ensure that it matches the network input dimension.

[0108] The three-dimensional sequence data is temporally characterized by forward and backward propagation paths of a bidirectional long short-term memory network, and a hidden state sequence containing contextual dependencies is obtained. The bidirectional propagation mechanism is used to capture the feature associations between past and future moments in the sequence and generate bidirectional hidden states.

[0109] Based on the hidden state sequence, the stress index time series sequence and the concentration index time series sequence are obtained respectively through full connection layer mapping and activation function conversion; the hidden state is mapped to the target indicator dimension through linear transformation and activation function (such as ReLU, Sigmoid), and the corresponding time series sequence is output.

[0110] Step S506 , performing sliding window normalization processing on the pressure index time series to generate a pressure index curve.

[0111] Step S506 , performing dynamic threshold segmentation processing on the concentration index time series to generate a concentration index curve.

[0112] Specifically, key biometric features are deconstructed and reconstructed to produce a multi-channel physiological signal sequence containing heart rate variability and brainwave frequency band energy values. A time-frequency domain feature vector is extracted from the physiological signal sequence. This vector contains the energy proportion of stress-related frequency bands and the phase difference of concentration-related frequency bands. The time-frequency domain feature vector is input into a three-dimensional spatial attention model to generate an attention weight distribution map. Based on this map, the time-frequency domain feature vectors are weightedly fused to obtain a fused feature tensor. A bidirectional long short-term memory network is used to process the fused feature tensor, outputting a time series sequence of stress indicators and a time series sequence of concentration indicators. The two types of time series are then subjected to sliding window normalization and dynamic threshold segmentation to generate stress index curves and concentration index curves, respectively.

[0113] This embodiment improves the temporal continuity and spatial accuracy of mental state analysis, enabling stress and concentration indicators to dynamically reflect changes in the user's mental workload in VR scenarios, and provides a scientific quantitative basis for the identification of mental state transition nodes and the formulation of intervention strategies.

[0114] In one embodiment, Figure 2 As shown, using a time series analysis model to identify real-time fluctuation curves, performing retrospective analysis based on psychological state transition nodes and multimodal cross-validation results to generate a three-dimensional interactive report can include the following steps:

[0115] Step S601 : Segment the time series data stream in the real-time fluctuation curve using a sliding time window, and extract the time domain and frequency domain feature vectors within each time window.

[0116] Step S602: determine whether the feature vector triggers a state transition node based on a preset mental state classification threshold, and generate a transition event sequence with a timestamp.

[0117] Preferably, the time domain and frequency domain feature vectors in each time window are judged window by window based on a preset psychological state classification threshold to obtain a Boolean value sequence indicating whether the feature vector meets the state transition condition; and whether the feature vector of each window corresponds to a psychological state change is determined by threshold comparison.

[0118] Timestamp the positions in the Boolean value sequence that meet the state transition conditions to obtain a set of time points that trigger the state transition; extract all time points that meet the conditions and associate them with the specific occurrence time.

[0119] A sequence of transition events with timestamps is generated based on a set of time points. The event content includes the type of mental state transition and the corresponding moment. The time points and state transition types are integrated into an ordered event sequence.

[0120] Step S603 , aligning the conversion event sequence with the multimodal cross-validation result dataset by timestamp to obtain an aligned validation event set.

[0121] Step S604: mining association rules of state transitions based on the verification event set to generate a rule confidence matrix.

[0122] Step S605 : Perform three-dimensional visualization on the rule confidence matrix to obtain a thermal map of the dynamically rendered psychological state transition path.

[0123] Step S606: Adjust the interactive viewing angle parameters based on the spatial mapping relationship between the thermal layer and the real-time fluctuation curve, and output a three-dimensional interactive report; the three-dimensional interactive report includes a spatial overlay map of EEG active areas and gaze trajectories, a temporal correlation map of physiological data and scene events, and a color-coded continuous distribution map of psychological states.

[0124] A sliding time window is used to segment the time series data stream from the real-time fluctuation curve. Time and frequency domain feature vectors within each window are extracted. Pre-set mental state classification thresholds are used to determine whether the feature vector triggers a state transition node, generating a time-stamped transition event sequence. This sequence is timestamp-aligned with the multimodal cross-validation dataset to obtain a validation event set. Based on this set, state transition association rules are mined and a rule confidence matrix is generated. This matrix is then visualized in three dimensions to form a heat map that dynamically renders the mental state transition paths. Finally, the interactive viewing angle parameters are adjusted based on the spatial mapping relationship between the heat map and the real-time fluctuation curve. The resulting report contains a spatial overlay of EEG activity areas and gaze trajectories, a temporal association diagram of physiological data and scene events, and a color-coded persistent mental state distribution map.

[0125] This embodiment uses sliding window technology to achieve granular analysis of time series data, combining it with preset thresholds to accurately identify psychological state transition nodes. Through association rule mining and 3D visualization, abstract psychological state changes are transformed into interactive thermal maps and spatial overlay maps. This improves the interpretability and visualization of psychological state analysis results, allowing medical staff to intuitively observe the temporal relationship between physiological data and scene events, the spatial distribution of psychological states, and transition paths through 3D interactive reports. This provides multidimensional data support and interactive analysis tools for clinical auxiliary diagnosis and the development of personalized intervention plans.

[0126] In one embodiment, the status report generation module may further include:

[0127] Step S701 : Screening the three-dimensional interactive report according to a preset key event detection model to obtain time points that meet a dynamic threshold, thereby obtaining a target time point set.

[0128] Step S702 : intercepting the 3D scene rendering frame data corresponding to the target time point to generate a scene screenshot sequence with a time stamp.

[0129] Step S703: extracting an emotion feature vector matching the target time point from the mental state continuous distribution map.

[0130] Step S704 , performing multimodal alignment on the visual feature vectors and the emotional feature vectors of the scene screenshot sequence to obtain fused state association data.

[0131] Step S705: query the preset intervention rule library based on the state association data. If there is a feature combination that continuously triggers the intervention condition in the state association data, generate an intervention strategy list with priority tags; the intervention rule library contains the mapping relationship between scene features and psychological states.

[0132] Step S706: perform similarity matching between the intervention strategy list and the user's historical feedback data, and output a personalized intervention suggestion set.

[0133] Preferably, a preset key event detection model is used to screen the three-dimensional interactive report, and target time points that meet the conditions are extracted based on dynamic thresholds to form a set of target time points. The three-dimensional scene rendering frame data corresponding to the target time point is intercepted to generate a sequence of scene screenshots with time tags, and at the same time, the emotional feature vector that matches the target time point is extracted from the continuous distribution map of the psychological state. The visual feature vector of the scene screenshot is multimodally aligned with the emotional feature vector to obtain the fused state association data, and the intervention rule library containing the mapping relationship between scene features and psychological states is queried based on the data. If there is a feature combination that continuously triggers the intervention condition, a list of intervention strategies with priority labels is generated. Finally, the intervention strategy list is matched with the user's historical feedback data for similarity, and a set of personalized intervention suggestions is output.

[0134] This embodiment improves the closed-loop service capabilities of the psychological testing system. It can not only provide diagnostic reports, but also generate targeted intervention suggestions based on real-time data and historical experience, such as locating VR levels that require relaxation and adjustment. It provides a scientific and efficient implementation path for personalized psychological intervention and enhances the practicality of the system in clinical auxiliary diagnosis and mental health management.

[0135] In one embodiment, Figure 3 As shown, this application also provides an immersive VR psychological detection method based on multimodal AI, which may include the following steps:

[0136] Step S801: Acquire a multimodal dataset of the user in a virtual reality scene based on a unified clock synchronization; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data, and operation behavior data; perform spatiotemporal alignment and noise reduction standardization on the multimodal dataset to extract key biometric features.

[0137] Step S802: Based on the response time difference of the key biometric features in the scene, a three-dimensional spatial attention model is constructed by implementing spatiotemporal correlation mapping through a spatial transformation network; the key biometric features are input into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve; the real-time fluctuation curve includes a pressure index curve and a concentration index curve.

[0138] Step S803 , using the time series analysis model to identify the real-time fluctuation curve, and performing a retrospective analysis based on the psychological state transition nodes and the multimodal cross-validation results to generate a three-dimensional interactive report.

[0139] This multimodal AI-based immersive VR psychological assessment method first uses a unified clock synchronization to acquire a multimodal dataset of users in a virtual reality scene, encompassing EEG signals, pupil data, gaze trajectory data, and action data. After performing spatiotemporal alignment and noise reduction normalization on this data, key biometric features are extracted. Next, based on the response time differences of key biometric features within the scene, a spatial transformation network is used to perform spatiotemporal correlation mapping, constructing a three-dimensional spatial attention model. Key biometric features are then input into this trained model to generate real-time fluctuation curves, including stress and concentration index curves. Finally, a time series analysis model is used to identify the real-time fluctuation curves. Retrospective analysis is then performed, combining psychological state transition nodes with multimodal cross-validation results, to generate a three-dimensional interactive report. This multimodal data fusion effectively captures the dynamic changes in the user's psychological state, facilitates comprehensive and in-depth analysis of the user's psychological state, and provides a scientific basis for psychological health assessment and intervention.

[0140] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the immersive VR psychological detection system and method based on multimodal AI as described above are implemented.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0143] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0144] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. An immersive VR psychological detection system based on multimodal AI, characterized by: The system comprises: A data acquisition and processing module is used to synchronously acquire a multimodal dataset of the user in a virtual reality scene based on a unified clock; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data, and operational behavior data; and is also used to perform spatiotemporal alignment and noise reduction standardization on the multimodal dataset to extract key biometric features; A correlation model construction module is configured to construct a three-dimensional spatial attention model based on the response time difference of the key biometric features in the scene by implementing spatiotemporal correlation mapping through a spatial transformation network; and is further configured to input the key biometric features into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve; the real-time fluctuation curve includes a stress index curve and a concentration index curve; The status report generation module is used to identify the real-time fluctuation curve using a time series analysis model, and to generate a three-dimensional interactive report by performing a retrospective analysis based on the psychological state transition nodes and multimodal cross-validation results.

2. The system according to claim 1, wherein: The multimodal dataset is subjected to spatiotemporal alignment and noise reduction standardization processing to extract key biometric features, including: Acquiring EEG signal data and pupil data in the multimodal data set; the EEG signal data includes a raw potential sequence marked with a timestamp; Performing time window matching on the original potential sequence and pupil data according to the timestamp mark, and obtaining a time-space aligned synchronization signal sequence using an interpolation algorithm; Generating a noise distribution model based on the synchronization signal sequence through preset frequency domain filtering parameters and dynamic threshold calibration; Performing signal optimization on the synchronization signal sequence using the noise distribution model and outputting a noise reduction weight matrix; Normalizing the spatial coordinates of the gaze trajectory data and the operation behavior data according to the noise reduction weight matrix to obtain a standardized set of biological feature vectors; Feature dimensions that meet preset association rules in the biometric feature vector set are extracted and fused to generate key biometric features; the key biometric features include EEG waveform patterns, pupil diameter change characteristics, gaze trajectory spatial distribution characteristics, and operation reaction speed characteristics.

3. The system according to claim 2, characterized in that The step of extracting feature dimensions that meet preset association rules from the set of biometric feature vectors and fusing them to generate key biometric features includes: The biometric feature vector set is screened according to the support threshold in the preset association rule to obtain a high-confidence feature subset; the preset association rule is established based on the dynamic coupling relationship between multimodal data; Calculating feature weights for the high-confidence feature subset to generate a fusion coefficient sequence for each dimension; Performing a dot product operation on the fusion coefficient sequence and the high-confidence feature subset using a dynamic weighting algorithm to obtain an initial key biometric feature; The initial key biometric feature is input into a preset association rule verification model to obtain a final key biometric feature that meets a confidence threshold.

4. The system according to claim 1, wherein: The method of constructing a three-dimensional spatial attention model by implementing spatiotemporal correlation mapping based on the response time difference of the key biometric features in the scene through a spatial transformation network includes: Acquiring time series data of the biometric feature in the scene; the time series data includes response time differences at different spatial locations; Obtaining a sight focus coordinate sequence, and calculating a coordinate offset in the sight focus coordinate sequence using coordinate differences between adjacent timestamps; Inputting the time series data and the coordinate offset into the spatial transformation network, and outputting an encoding tensor containing spatiotemporal correlation features; Decomposing the EEG signal data by wavelet transform to obtain frequency band energy values of EEG active areas; Performing feature fusion on the coding tensor and the frequency band energy value to generate a multimodal feature vector; Calculating the path curvature in the gaze trajectory data by the rate of change of the tangent angles of adjacent moving coordinates; Associating and mapping the multimodal feature vector with the path curvature to obtain an attention weight matrix; The three-dimensional spatial coordinates are weighted according to the attention weight matrix to generate a three-dimensional spatial attention model of the activation area distribution.

5. The system according to claim 1, wherein: The association model building module also includes: Performing feature inversion and signal reconstruction on the key biological features to obtain a corresponding multi-channel physiological signal sequence; the physiological signal sequence includes heart rate variability and brain wave frequency band energy value; Extracting a time-frequency domain feature vector based on the physiological signal sequence; the time-frequency domain feature vector includes the energy proportion of the pressure-related frequency band and the phase difference of the concentration-related frequency band; Inputting the time-frequency domain feature vector into the three-dimensional spatial attention model to generate an attention weight distribution map; Performing weighted fusion on the time-frequency domain feature vectors according to the attention weight distribution map to obtain a fused feature tensor; Using a bidirectional long short-term memory network to process the fused feature tensor, and outputting a time series sequence of stress indicators and a time series sequence of concentration indicators; Performing sliding window normalization processing on the pressure indicator time series to generate the pressure indicator curve; Dynamic threshold segmentation processing is performed on the concentration index time series to generate the concentration index curve.

6. The system according to claim 1, wherein: The method of using a time series analysis model to identify the real-time fluctuation curve and performing a retrospective analysis based on the psychological state transition nodes and multimodal cross-validation results to generate a three-dimensional interactive report includes: The sliding time window is used to segment the time series data stream in the real-time fluctuation curve, and the time domain and frequency domain feature vectors in each time window are extracted; Determining whether the feature vector triggers a state transition node based on a preset mental state classification threshold, and generating a transition event sequence with a timestamp; Performing timestamp alignment on the conversion event sequence and the multimodal cross-validation result dataset to obtain an aligned validation event set; Mining association rules of state transitions based on the verification event set to generate a rule confidence matrix; Performing three-dimensional visualization on the rule confidence matrix to obtain a thermal map that dynamically renders the psychological state transition path; According to the spatial mapping relationship between the thermal map and the real-time fluctuation curve, the interactive viewing angle parameters are adjusted and a three-dimensional interactive report is output; the three-dimensional interactive report includes a spatial overlay map of EEG active areas and gaze trajectories, a temporal correlation map of physiological data and scene events, and a color-coded continuous distribution map of psychological states.

7. The system according to claim 1, wherein: The status report generating module further includes: The three-dimensional interactive report is screened according to a preset key event detection model to obtain time points that meet a dynamic threshold, thereby obtaining a target time point set; Intercepting the three-dimensional scene rendering frame data corresponding to the target time point to generate a scene screenshot sequence with a time stamp; extracting an emotion feature vector matching the target time point from the mental state continuous distribution map; Performing multimodal alignment on the visual feature vector of the scene screenshot sequence and the emotion feature vector to obtain fused state association data; A preset intervention rule library is queried based on the state association data. If a feature combination that continuously triggers intervention conditions exists in the state association data, an intervention strategy list with priority tags is generated. The intervention rule library contains a mapping relationship between scene features and psychological states. The intervention strategy list is matched with the user's historical feedback data for similarity, and a personalized intervention suggestion set is output.

8. An immersive VR psychological detection method based on multimodal AI, characterized by: The method comprises: Acquire a multimodal dataset of a user in a virtual reality scene based on a unified clock synchronization; the multimodal dataset includes EEG signal data, pupil data, gaze trajectory data, and operation behavior data; perform spatiotemporal alignment and noise reduction standardization on the multimodal dataset to extract key biometric features; Based on the response time difference of the key biometric features in the scene, a three-dimensional spatial attention model is constructed by implementing spatiotemporal correlation mapping through a spatial transformation network; the key biometric features are input into the trained three-dimensional spatial attention model to generate a real-time fluctuation curve; the real-time fluctuation curve includes a stress index curve and a concentration index curve; The real-time fluctuation curve is identified using a time series analysis model, and a three-dimensional interactive report is generated by performing a retrospective analysis based on the psychological state transition nodes and multimodal cross-validation results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the system according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the system according to any one of claims 1 to 7 are implemented.

Images