A virtual reality-based astronaut weightlessness training evaluation method and system

Abstract

The present invention relates to the field of virtual reality technology, and discloses a virtual reality-based astronaut weightlessness training evaluation method and system. The virtual reality-based astronaut weightlessness training evaluation method comprises: collecting eye movement data and vestibular response data of astronauts in a virtual weightlessness environment; calculating an eye-vestibular coupling index based on the eye movement data and vestibular response data; utilizing a self-attention mechanism to achieve multimodal fusion of the eye movement data and vestibular response data; constructing an individualized adaptation model to predict astronauts' resistance to illusions under different weightlessness conditions; comprehensively evaluating astronauts' performance in the virtual weightlessness environment to predict their adaptability in actual space missions; and constructing a comprehensive evaluation method for astronaut weightlessness training through multimodal data fusion and deep learning technology. Compared with traditional single-indicator evaluation methods, this method improves the evaluation accuracy of astronauts' adaptability in a weightlessness environment.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and more particularly to a virtual reality-based astronaut weightlessness training evaluation method and system. Background Art

[0002] During space missions, astronauts face visual-vestibular system conflicts caused by the weightless environment, which impacts their spatial orientation, balance control, and operational capabilities. This conflict can lead to spatial disorientation, unstable attitude control, and decreased operational accuracy, seriously impacting mission efficiency and safety.

[0003] Existing ground-based training methods rely primarily on devices like rotating chairs and tilting platforms to simulate a partial sense of weightlessness. However, these methods struggle to fully simulate the complex visual-vestibular conflict conditions in space. While virtual reality technology offers new possibilities for simulating weightlessness on the ground, existing VR weightlessness training and assessment systems have the following technical issues:

[0004] The existing assessment system mainly relies on the monitoring of a single physiological indicator, such as eye movement data or posture stability indicators, which cannot accurately quantify astronauts' adaptability and illusion resistance in a visual-vestibular conflict environment; the existing system has difficulty identifying and quantifying astronauts' attention resource allocation patterns when dealing with visual-vestibular conflicts, and lacks an accurate assessment of the coordinated adaptability of the vestibular and visual systems; the existing system cannot provide accurate assessments and personalized training plans based on individual differences among astronauts, resulting in low training efficiency and difficulty in improving astronauts' adaptability in a targeted manner.

[0005] Therefore, a method and system are needed to comprehensively evaluate astronauts' adaptability in a visual-vestibular conflict environment in order to improve the accuracy and effectiveness of astronauts' weightlessness training. Summary of the Invention

[0006] The present invention provides a virtual reality-based astronaut weightlessness training evaluation method and system, which solves the technical problem in related technologies that astronauts' adaptability in a visual vestibular conflict environment cannot be accurately quantified.

[0007] The present invention provides an astronaut weightlessness training evaluation method based on virtual reality, comprising:

[0008] Collect eye movement data and vestibular response data of astronauts in a virtual weightless environment;

[0009] The oculovestibular coupling index is calculated based on eye movement data and vestibular response data to quantify the adaptation rate and stability of astronauts in a visual-vestibular conflict environment.

[0010] Use the self-attention mechanism to achieve multimodal fusion of eye movement data and vestibular response data to analyze the astronauts' attention resource allocation pattern;

[0011] Based on eye movement data and vestibular responses, the eye-vestibular coupling index, and the classification and evaluation indicators of attention resource allocation patterns, an individualized adaptation model was constructed to predict astronauts' resistance to illusions under different weightlessness conditions.

[0012] Based on eye movement data and vestibular responses, eye-vestibular coupling index, attention resource allocation pattern classification and evaluation indicators, and individualized adaptation model prediction results, the astronauts' performance in the virtual weightless environment is comprehensively evaluated and their adaptability in actual space missions is predicted.

[0013] Furthermore, the collection of eye movement data and vestibular response data of astronauts in a virtual weightless environment includes:

[0014] Eye tracking equipment integrated into the virtual reality headset collects eye movement parameters, including eye movement trajectory, gaze point distribution, and pupil size changes;

[0015] The inertial sensors fixed on the head-mounted virtual reality device collect the three-axis linear acceleration data, three-axis angular velocity data and head posture data of the astronaut's head;

[0016] Timestamp alignment technology is used to ensure accurate synchronization of eye movement data and vestibular response data.

[0017] Furthermore, the oculovestibular coupling index is calculated based on the eye movement data and the vestibular response data, including:

[0018] Extract eye movement feature vectors from eye movement data, including saccade velocity, fixation duration, fixation point distribution density, and pupil diameter change rate;

[0019] Extracting vestibular feature vectors from vestibular response data, including head angular velocity, linear acceleration, and posture change rate;

[0020] The oculovestibular coupling index was calculated based on the extracted eigenvectors, and the adaptation ability curve was drawn according to the coupling index values in the continuous time window. The initial adaptation rate, steady-state coupling value, adaptation time and fluctuation amplitude were extracted.

[0021] Furthermore, the multimodal fusion of eye movement data and vestibular response data using the self-attention mechanism includes:

[0022] Convert eye movement data and vestibular response data into feature matrices and extract temporal features, frequency domain features, and spatial features;

[0023] Apply the self-attention mechanism to calculate the attention weight matrix and identify the importance relationship between different features;

[0024] Capturing the temporal dependencies of multimodal data based on bidirectional long short-term memory networks;

[0025] By analyzing the self-attention weight matrix and bidirectional long short-term memory network output, the astronauts' attention resource allocation pattern is identified.

[0026] Furthermore, the attention resource allocation mode includes visual preference type, vestibular preference type, balance integration type and random switching type.

[0027] Furthermore, the construction of the individualized adaptation model includes:

[0028] A deep convolutional neural network is used to extract features from eye movement data and vestibular response data;

[0029] Combining the astronauts' personal information with multimodal features, individual feature vectors are constructed through residual networks;

[0030] Construct an illusion resistance prediction model based on the double Q network algorithm;

[0031] Based on the astronauts' performance data during training, the parameters of the individualized adaptation model are continuously optimized.

[0032] Furthermore, the comprehensive assessment of astronauts' performance in a virtual weightless environment includes:

[0033] Calculate the astronauts' comprehensive adaptability score based on multiple indicators;

[0034] The adaptability of astronauts is divided into five levels from A to E based on the comprehensive adaptability score;

[0035] Use multiple visualization methods to display astronauts' performance and training progress in various dimensions;

[0036] Predicting astronauts' adaptation performance in actual space missions based on support vector regression model;

[0037] Generate targeted, personalized training recommendations based on assessment results and predictive analysis.

[0038] Furthermore, the comprehensive adaptability score is calculated based on the following parameters: the steady-state value of the oculovestibular coupling index, the astronaut's adaptation time, the fluctuation amplitude of the coupling index, the task completion score and the attention resource allocation pattern score.

[0039] Furthermore, the virtual weightlessness environment includes: a rotating scene in which visual information is inconsistent with vestibular perception, a scene simulating a spatial positioning task in a weightless state, and a scene simulating an operation task in a space capsule.

[0040] The present invention provides a virtual reality-based astronaut weightlessness training evaluation system, which is used to implement the above-mentioned virtual reality-based astronaut weightlessness training evaluation method, including:

[0041] Multimodal data acquisition module, used to collect eye movement data and vestibular response data of astronauts in a virtual weightless environment;

[0042] The oculovestibular coupling calculation module is used to calculate the oculovestibular coupling index and quantify the adaptation rate and stability of astronauts in the illusion environment;

[0043] Attention resource allocation analysis module, which uses the self-attention mechanism to achieve multimodal data fusion and analyze the astronauts' attention resource allocation pattern;

[0044] An individualized adaptation model building module is used to implement real-time fusion analysis of visual vestibular data based on deep convolutional neural networks and to build an illusion resistance prediction model using deep reinforcement learning algorithms;

[0045] The training effect evaluation and prediction module is used to evaluate astronauts' performance in a virtual weightless environment, predict their adaptability in actual space missions, and provide personalized training recommendations.

[0046] The beneficial effects of the present invention are:

[0047] By integrating multimodal data with deep learning technology, a comprehensive assessment method for astronauts' weightlessness training has been constructed. Compared with traditional single-index assessment methods, this method improves the accuracy of assessing astronauts' adaptability to weightlessness. The eye-vestibular coupling index and attention resource allocation vector can accurately quantify astronauts' adaptability and attention resource allocation patterns in visual-vestibular conflict environments.

[0048] Through the individualized balance ability adaptation model, it is possible to quantify the astronauts' resistance to illusions and adaptability by analyzing the individual's unique response pattern in the visual vestibular conflict environment, providing a scientific basis for personalized training programs;

[0049] Based on multi-dimensional evaluation indicators and deep reinforcement learning models, it is possible to adjust training parameters and difficulty to maximize training effects, reduce training time, reduce resource consumption, and improve training efficiency.

[0050] Through a comprehensive evaluation index system and a space environment performance prediction model, it is possible to predict astronauts' performance in actual space missions with higher accuracy than traditional evaluation methods, providing a more reliable decision-making basis for astronaut selection and mission assignment.

[0051] By discovering astronauts' adaptive weaknesses in advance and conducting targeted training, the operational risks caused by astronauts' poor adaptation during actual space missions can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a flowchart of an astronaut weightlessness training evaluation method based on virtual reality.

[0053] Figure 2 It is a flow chart of step 1 of the present invention.

[0054] Figure 3 It is a flow chart of step 2 of the present invention.

[0055] Figure 4 It is a flow chart of step 3 of the present invention.

[0056] Figure 5 It is a flow chart of step 4 of the present invention.

[0057] Figure 6 It is a flow chart of step 5 of the present invention. DETAILED DESCRIPTION

[0058] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0059] At least one embodiment of the present invention discloses a method for evaluating astronaut weightlessness training based on virtual reality, such as Figures 1 to 6 As shown, the following steps are included:

[0060] Step 1: Collect eye movement data and vestibular response data of astronauts in a virtual weightless environment;

[0061] In this step, a dual-channel data acquisition system for eye tracking and head micro-movement detection is constructed to synchronously record the astronauts' eye movement trajectories, gaze point distribution, and vestibular system response data in a controllable visual vestibular illusion-inducing scenario.

[0062] Step 1.1, construct a visual vestibular illusion induction scene;

[0063] A series of scenarios specifically designed to induce visual-vestibular conflict were created in a VR environment, including those where visual motion was inconsistent with body perception, those with illusory gravity, and those where rotation alternated between the body and the surroundings. These scenarios simulated the perceptual conflict astronauts might experience in the weightless environment of space by precisely controlling the inconsistency between visual flow and vestibular stimulation.

[0064] Step 1.2, configure the multimodal data acquisition system;

[0065] A high-precision eye tracker is integrated into the VR headset to collect eye movement data including gaze point coordinates, eye movement speed, pupil size, etc.; at the same time, the inertial measurement unit sensor integrated into the VR headset collects micro-motion data such as head position, angular velocity and acceleration to achieve indirect measurement of the vestibular system response.

[0066] Step 1.3, realize data synchronous collection and preprocessing;

[0067] A timestamp synchronization mechanism was used to ensure temporal alignment between eye movement data and head micro-movement data. The sampling frequency was set at 120 Hz for eye movement data and 1000 Hz for head micro-movement data. The collected raw data was subjected to noise reduction processing, including median filtering to remove blink artifacts from the eye movement data and wavelet transform to remove high-frequency noise from the head micro-movement data. The data was then normalized.

[0068] Step 1.4, extract multimodal feature vector;

[0069] Eye movement feature vectors and vestibular response feature vectors are extracted from the preprocessed data. The eye movement feature vectors include features such as the gaze point coordinate sequence, eye movement velocity spectrum, and gaze point dwell time distribution; the vestibular response feature vectors include features such as the head angular velocity spectrum, acceleration change rate, and head posture stability index.

[0070] After timestamp alignment and preprocessing, these multimodal data lay the foundation for subsequent eye-vestibular coupling calculation and can accurately characterize the visual-vestibular coordination state of astronauts in a virtual weightless environment.

[0071] Step 2: Calculate the eye-vestibular coupling index based on the eye movement data and vestibular response data to quantify the astronauts' adaptation rate and stability in the visual-vestibular conflict environment;

[0072] This step uses correlation analysis based on the eye movement eigenvectors and vestibular response eigenvectors obtained in Step 1 to calculate the eye-vestibular coupling index, quantifying the astronauts' adaptation rate and stability in the illusory environment. This step converts the multimodal raw data collected in Step 1 into a quantifiable adaptability indicator, establishing a foundation for evaluating the astronauts' visual-vestibular coordination ability.

[0073] Step 2.1, construct a time window analysis model;

[0074] The collected continuous data stream is divided into windows of fixed time length (e.g., 2 seconds). Each window contains time-synchronized eye movement feature vectors and vestibular response feature vectors. The overlap rate of adjacent windows is set to 50% to ensure the continuity of data analysis.

[0075] Step 2.2, calculate the eye movement vestibular covariance matrix;

[0076] For each time window, calculate the covariance of the eye movement feature vector and vestibular response feature vector , represents the correlation between two eigenvectors. The covariance matrix calculation formula is:

[0077] ;

[0078] in Represents the eye movement feature vector Vestibular response eigenvector The covariance between is the number of sampling points in the time window, and Represent the average values of eye movement characteristics and vestibular response characteristics within the time window, Indicates at a point in time The eye movement feature vector A quantity, Indicates at a point in time The vestibular response eigenvector A quantity, is the index of the sampling point in the time window; Represents the summation symbol.

[0079] Step 2.3, calculate the oculovestibular coupling index;

[0080] Based on the covariance matrix, the eye-vestibular coupling index is calculated, which reflects the degree of coordination between eye movements and vestibular responses. The calculation formula is:

[0081] ;

[0082] in is the oculovestibular coupling index, is the feature dimension, which represents the dimension of the feature vector, and represent the variance of eye movement characteristics and vestibular response characteristics, respectively. represents the covariance between eye movement features and vestibular response features, Indicates the sum of all feature dimensions. Represents the square root operation;

[0083] The value range of is [-1,1]. The closer the value is to 1, the higher the coordination between eye movement and vestibular response is. The closer the value is to -1, the higher the inverse coordination between them is. The closer the value is to 0, the lack of coordination between them is.

[0084] Step 2.4, constructing an adaptation rate and stability evaluation model;

[0085] Based on time series The value changes are used to build an evaluation model for astronaut adaptation rate and stability. pass The speed of change from low to high is calculated, that is:

[0086] ;

[0087] in represents the adaptation rate, express About time The derivative of , that is, the rate of change, The time point when obvious changes begin to appear. Indicates at a point in time Calculate the derivative value at .

[0088] stability pass The volatility of the value in the stable stage is calculated, that is:

[0089] ;

[0090] in represents the stability index, Stable stage The standard deviation of the values, indicating the degree of fluctuation, Stable stage The average value of the represents the coefficient of variation, a measure of relative volatility; Values closer to 1 indicate greater stability.

[0091] After step 2 is completed, the output is: time series of eye-vestibular coupling index , adaptation rate , stability index , adaptation curve; these indicators together constitute a quantitative assessment of the adaptability of the astronauts' visual vestibular system, providing basic data support for subsequent attention resource allocation analysis and the establishment of individualized adaptation models. In particular, the eye-vestibular coupling index As a core quantitative indicator of astronauts' adaptability, it will be widely used in subsequent steps.

[0092] Step 3: Use the self-attention mechanism to achieve multimodal fusion of eye movement data and vestibular response data to analyze the astronauts' attention resource allocation pattern;

[0093] This step is based on the multimodal data obtained in step 1 (eye movement feature vector and vestibular response eigenvector and the oculovestibular coupling index calculated in step 2 Utilizing a self-attention algorithm to fuse multimodal data, combined with a recurrent neural network to capture temporal feature changes, we evaluated astronauts' attentional resource allocation strategies under different weightlessness conditions. This step provides an in-depth analysis of astronauts' cognitive mechanisms for coping with visual-vestibular conflict from the perspective of attentional resource allocation, further expanding upon and deepening the coupling index analysis in Step 2.

[0094] Step 3.1, build a self-attention model;

[0095] The self-attention algorithm is introduced to process multimodal data. The self-attention calculation formula is:

[0096] ;

[0097] in is the self-attention function, 、 、 They represent the query matrix, key matrix, and value matrix mapped from the original input data, respectively. yes Dimensions, Representation matrix With the matrix Multiply the transpose of express The square root of , used to scale the dot product result, is a normalization function that converts the input into a probability distribution, Is a value matrix, multiplied by the attention weight to get the final output;

[0098] In this model, the eye movement feature vector and vestibular response eigenvector As input, it is mapped to 、 、 matrix:

[0099] ;

[0100] ;

[0101] ;

[0102] in 、 、 is the weight matrix to be trained, which is used to generate query, key and value matrices respectively. Indicates that the eye movement feature vector and vestibular response eigenvector Concatenate on the feature dimension to form a joint feature representation.

[0103] Step 3.2, build a multi-head attention model;

[0104] To capture relationships at different levels, a multi-head attention model is constructed to map the input data into different subspaces:

[0105] ;

[0106] ;

[0107] in is the multi-head attention function, 、 、 Respectively represent , 1st and 2nd An attention head, represents the number of attention heads (set to 8 in this example), 、 、 They are The query, key, and value matrices for each attention head, Represents the splicing operation, which splices the outputs of all attention heads in the feature dimension. is the output mapping matrix, which is used to map the concatenated features to the required output dimension;

[0108] The multi-head attention model can capture the relationship between eye movement and vestibular response features from different subspaces, providing richer attention distribution information.

[0109] Step 3.3, integrating bidirectional long short-term memory network;

[0110] The output of the multi-head attention model is fed into a bidirectional long short-term memory network algorithm to capture long-term dependencies in time series data. The Bi-LSTM consists of two LSTM layers, one forward and one backward, which can simultaneously consider past and future contextual information. The core computation of the LSTM unit includes:

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] in 、 、 Respectively represent the forget gate, input gate and output gate, which control the flow of information. Represents cell state, stores long-term memory, represents the candidate cell state, that is, the new information that may be added to the cell state at the current time step, represents the cell state at the previous time step, represents the hidden state, as the output of the current time step, represents the hidden state at the previous time step, represents the input data of the current time step, 、 、 、 Represent the weight matrices of the forget gate, input gate, cell state, and output gate, respectively, corresponding to the parameters of each gate. 、 、 、 Represent the bias terms of forget gate, input gate, cell state and output gate respectively, Indicates that the hidden state of the previous time step With the current input Splicing, Represents the sigmoid activation function, with an output range of [0,1], Represents the hyperbolic tangent activation function, with an output range of [-1,1]; Represents the current time step index.

[0118] The output of the bidirectional LSTM is obtained by concatenating the forward and backward hidden states:

[0119] ;

[0120] in is the hidden state output of Bi-LSTM, Represents the hidden state of the forward LSTM, processing information from the past to the present, Represents the hidden state of the backward LSTM, processing information from the future to the present, Represents the concatenation operation on the feature dimension.

[0121] This allows for capturing more complete temporal context information.

[0122] Step 3.4, construct an attention resource allocation model;

[0123] Based on the output of the Bi-LSTM network, an attention resource allocation model is constructed to quantitatively evaluate the attention allocation strategy of astronauts in a weightless environment:

[0124] ;

[0125] ;

[0126] ;

[0127] in 、 、 represent the proportion of attention resources allocated to visual tasks, vestibular balance tasks, and motor control tasks, respectively. The Softmax activation function ensures that the sum of the three is 1, converting the original score into a probability distribution. 、 、 Represent the weight matrices of vision, vestibular balance and motor control tasks respectively, and are used to calculate the attention allocation score of each task. 、 、 represent the bias terms of vision, vestibular balance and motor control tasks respectively, It is the hidden state output of Bi-LSTM, containing timing information.

[0128] By comparing attention resource allocation patterns across tasks of varying difficulty, at different stages, and across individuals, we can identify strategic differences and adaptation processes in astronauts' responses to visual-vestibular conflict. The attention allocation model also calculates the following evaluation metrics:

[0129] Attention switching frequency:

[0130] ;

[0131] in is the attention switching frequency, Indicates a time point The attention allocation vector, Indicates the total length of the sequence, Represents the absolute difference of attention allocation vectors at adjacent time points, represents the sum of all adjacent time point pairs, Indicates finding the average value.

[0132] Attention Stability Index (ASI):

[0133] ;

[0134] in is the attention stability index, is the average attention distribution vector, represents the deviation of the attention allocation vector from the mean, represents the square of the deviation, represents the sum of all time points, Indicates the average value, represents the square root operation, It represents the ratio of standard deviation to mean, that is, coefficient of variation.

[0135] Task Priority Index (TPI):

[0136] ;

[0137] in is the task priority index, Indicates a time point The task performance indicators, Indicates a time point Assigned to The ratio of attention resources to tasks, represents the weighted product of attention resources and task performance, represents the sum of all time points, Represents the weighted average attention allocation ratio.

[0138] After step 3 is completed, the output is: attention resource allocation vector time series, attention resource allocation pattern classification, attention evaluation index, and attention weight matrix;

[0139] Based on Based on the temporal and distribution characteristics of the astronauts' attention resources allocation patterns, the astronauts' attention resources allocation patterns are divided into four categories:

[0140] Visual preference type: mainly relies on visual information for spatial orientation and balance control;

[0141] Vestibular preference: mainly relies on vestibular sensation for spatial orientation and balance control;

[0142] Balanced integration: effectively integrates visual and vestibular information, and allocates attention resources in a balanced manner;

[0143] Random switching: randomly switching between visual and vestibular information, lacking a stable attention allocation strategy;

[0144] Attention assessment indicators include attention switching frequency, attention stability index and task priority index, which quantify different dimensions of astronauts' attention regulation ability;

[0145] The attention weight matrix shows the importance relationship between different modal features and reveals the feature selection preferences of astronauts when dealing with visual-vestibular conflict.

[0146] These data provide insights into the cognitive mechanisms and strategic choices astronauts make in coping with visual-vestibular conflict. Together with the oculovestibular coupling index from Step 2, they provide a comprehensive assessment of astronauts' adaptive abilities. In particular, the classification of attentional resource allocation patterns provides crucial input for the subsequent construction of individualized adaptation models, enabling precise characterization of the cognitive preferences and adaptive strategies of different astronauts.

[0147] Step 4: Based on eye movement data and vestibular responses, the eye-vestibular coupling index, and the classification and evaluation indicators of attention resource allocation patterns, an individualized adaptation model is constructed to predict astronauts' resistance to illusions under different weightlessness conditions.

[0148] This step integrates the core outputs of the first three steps: the multimodal raw data of step 1 (eye movement feature vector and vestibular response eigenvector ), step 2 oculovestibular coupling index As well as the attention resource allocation pattern classification and evaluation indicators in step 3, a deep convolutional neural network is used to implement real-time fusion analysis of visual vestibular data, and a deep reinforcement learning algorithm is used to build a prediction model for illusion resistance and establish an individualized balance ability adaptation model for astronauts. This step further integrates the analytical results of the previous steps to construct a personalized adaptation model tailored to each astronaut's characteristics, achieving the ability to predict astronaut performance during space missions.

[0149] Step 4.1, build a deep convolutional neural network model;

[0150] A multi-channel deep convolutional neural network (CNN) structure was designed to process eye movement data and vestibular response data separately. The CNN structure consists of the following layers:

[0151] Input layer: receives eye movement feature vectors and vestibular response eigenvector , reconstruct the time series data into a two-dimensional feature map;

[0152] Convolution layer: Use multi-scale convolution kernels (3×3, 5×5, 7×7) to extract feature patterns of different scales. The convolution operation is expressed as:

[0153] ;

[0154] in Indicates the Layer feature maps, which are the outputs of the convolutional layers; Indicates the Layer The feature map and Layer The convolution kernel between the feature maps, that is, the convolution weight; represents the bias term; Represents the activation function (using the ReLU function, i.e. ); Indicates that all feature maps of the previous layer are convolved and summed; Indicates the The number of feature maps of the layer;

[0155] Pooling layer: Use the maximum pooling method to reduce dimension and retain features;

[0156] Batch normalization layer: accelerates network convergence and improves stability;

[0157] Fully connected layer: maps the flattened feature vector to the prediction space.

[0158] Step 4.2, realize visual vestibular data fusion;

[0159] A feature-level fusion strategy is used to fuse the eye movement features and vestibular response features extracted by CNN in the feature space:

[0160] ;

[0161] in represents the fused features, and Respectively represent the eye movement features and vestibular response features extracted by CNN, Represents the feature concatenation operation, connecting the two features in the feature dimension. Represents the fusion function (implemented by the fully connected layer), which maps the concatenated features to a unified feature space.

[0162] Step 4.3, build a deep reinforcement learning model;

[0163] Based on the fused features , build a deep reinforcement learning model to predict astronauts’ illusion resistance under different interference conditions. Using the Double Deep Q-Network (DDQN) structure, including the state space , action space and the reward function :

[0164] State Space :By fusion features It reflects the current visual-vestibular coordination status of the astronauts;

[0165] Action Space : Indicates possible resistance strategies, such as “visual dependence”, “vestibular dependence” and “multimodal fusion”;

[0166] Reward Function : Based on the performance score of astronauts when performing specific tasks, such as attitude stability, mission completion accuracy, etc.

[0167] The Q value update formula is:

[0168] ;

[0169] in Indicates that the status Next action The expected cumulative return value, is the learning rate, which controls the update step size, is a discount factor that determines the importance of future rewards and has a value range of [0,1]; is the immediate reward, which indicates the feedback obtained after performing the action. and Represents the current state and the action taken, respectively. Indicates execution of an action Then transfer to the next state, Indicates the maximum Q value that can be obtained in the next state, that is, in state The maximum expected reward among all possible actions, represents the time series difference error, which is the difference between the actual return and the expected return.

[0170] Step 4.4, establish an individualized balance ability adaptation model;

[0171] Based on the training results of deep reinforcement learning, an individualized balance ability adaptation model is constructed. The model consists of the following components:

[0172] Fitness scoring function:

[0173] ;

[0174] in Indicates the astronaut at the time point The adaptability score, 、 and They represent the weight coefficients of eye-vestibular coupling, visual attention resources and optimal strategy return, is the oculovestibular coupling index, Indicates the proportion of attention resources allocated to the visual channel; It represents the maximum Q value in the current state, reflecting the expected return of the optimal strategy.

[0175] Adaptive curve fitting:

[0176] Use an exponential function to fit the astronaut's adaptation curve:

[0177] ;

[0178] in represents the maximum adaptability score, that is, the upper limit of the score in the long-term stable state, represents the individual adaptation rate constant, which determines the steepness of the adaptation curve. represents the exponential decay term, Indicates training time.

[0179] Quantification of individual differences:

[0180] By comparing different astronauts Value and The value can be used to quantify the differences in adaptability between individuals and provide a basis for personalized training programs.

[0181] Step 4 outputs: an individualized balance adaptation model, an illusion resistance prediction model, and an adaptability scoring function. These accurately characterize the astronaut's individual adaptability characteristics, integrating the analytical results from Steps 1 through 3 into a unified, individualized model, laying the foundation for subsequent training effectiveness evaluation and prediction. In particular, the individualized balance adaptation model not only describes the astronaut's current state of adaptation but also predicts their performance under varying training intensities and space mission scenarios, providing the core basis for the comprehensive assessment and personalized training recommendations in Step 5.

[0182] Step 5: Based on eye movement data and vestibular responses, the eye-vestibular coupling index, the classification and evaluation indicators of attention resource allocation patterns, and the prediction results of the individualized adaptation model, comprehensively evaluate the astronauts' performance in the virtual weightless environment and predict their adaptability in actual space missions;

[0183] This step comprehensively utilizes all the outputs of the previous four steps: the multimodal raw data from step 1, the oculovestibular coupling index and adaptability curve from step 2, the attention resource allocation model and evaluation indicators from step 3, and the individualized adaptation model and illusion resistance prediction results from step 4. This step constructs a comprehensive assessment and prediction system. Based on the astronauts' multidimensional performance data in a virtual weightless environment, it predicts their adaptability during actual space missions and provides personalized training recommendations. This step is the final integration stage of the entire method, transforming all the previous analyses into practical assessment results and training guidance.

[0184] Step 5.1, construct a comprehensive evaluation indicator system;

[0185] Based on the multi-dimensional evaluation indicators obtained in the above steps, a comprehensive evaluation indicator system is constructed, including:

[0186] Vestibular-visual adaptation index: based on the eye-vestibular coupling index The temporal variation characteristics of are used to calculate the vestibular-visual adaptation index:

[0187] ;

[0188] in represents the vestibular-visual adaptation index, is the evaluation cycle, i.e. the total evaluation time, express The integral over the entire evaluation period, i.e. the average coupling degree, represents the normalization factor, To adapt to the rate, is the rate weight factor, which controls the influence of the adaptation rate on the total score. Indicates the weighted adjustment term for the adaptation rate.

[0189] Attention regulation index: based on attention resource allocation vector Dynamic change characteristics of the astronauts' attention control ability:

[0190] ;

[0191] in represents the attention regulation index, for The entropy value is calculated as , represents the uncertainty of attention allocation, The smaller it is, the more targeted the attention allocation is. Indicates concentration, represents the integral of attention concentration during the entire evaluation period, represents the normalization factor.

[0192] Illusion resistance indicator: The effect of illusion resistance strategy predicted by deep reinforcement learning model:

[0193] ;

[0194] in Indicates the illusion resistance index, is the number of test state samples, Indicates the Test status, Status The maximum Q value under this state represents the expected return of the optimal strategy. represents the sum of the maximum Q values of all test states, Indicates finding the average value.

[0195] Step 5.2, training effect evaluation model;

[0196] Based on comprehensive evaluation indicators, a training effect evaluation model is constructed to calculate the astronauts' overall adaptability score. :

[0197] ;

[0198] in represents the overall adaptability score, 、 and The weight coefficients of vestibular-visual adaptability, attention regulation ability, and illusion resistance, respectively, indicate the importance of vestibular-visual adaptability, attention regulation ability, and illusion resistance in the total score, which are determined through statistical analysis of historical data.

[0199] Step 5.3, construct a space environment performance prediction model;

[0200] Based on the comparative analysis of ground-based VR training data and existing astronaut space mission performance data, a performance prediction model is constructed:

[0201] ;

[0202] in is the predicted value of space environment performance, which indicates the expected performance of astronauts in actual space missions. For the overall adaptability score, is the training data feature, including the progress curve and stability during training. For the evaluation cycle, Represents a nonlinear mapping function. The nonlinear mapping function is implemented using the Gradient Boosting Decision Tree (GBDT) model. , the model parameters are obtained through historical data training.

[0203] Step 5.4, generate personalized training suggestions;

[0204] Based on the evaluation results and prediction model, personalized training suggestions are automatically generated according to the characteristics of different astronauts:

[0205] Identifying Adaptive Capacity Weaknesses: By Comparison 、 and Three indicators to identify astronauts' relative weaknesses in visual vestibular adaptation, attention regulation, or illusion resistance;

[0206] Training intensity optimization: based on adaptation curves:

[0207] ;

[0208] in Indicates the astronaut at the time point The adaptability score, represents the maximum fitness score, represents the individual adaptation rate constant, Indicates the time point. Calculate the optimal training intensity increase rate :

[0209] ;

[0210] in represents the optimal training intensity increment rate, that is, the ratio at which the training difficulty should be increased. represents the individual adaptation rate constant, represents the maximum fitness score, Score the current adaptability, i.e. the astronaut’s current adaptability level, is a correction factor used to adjust the rate of increase in training intensity based on individual characteristics;

[0211] Personalized training content: Based on weakness indicators, targeted training content is recommended, such as visual vestibular conflict reinforcement training, attention allocation training, or illusion environment adaptation training, and corresponding training parameters are configured.

[0212] Step 5 Output: Comprehensive adaptability score, adaptability level classification, multi-dimensional performance data, space mission adaptability prediction report and personalized training program recommendations;

[0213] The astronauts' adaptability is divided into five levels according to the TAS score:

[0214] Level A (90-100 points): Excellent adaptability, capable of undertaking the most complex space missions;

[0215] Level B (80-89 points): Excellent adaptability, suitable for most space missions;

[0216] Level C (70-79 points): Good adaptability, needs to strengthen training for weaknesses;

[0217] Level D (60-69 points): Basic adaptability, requiring systematic intensive training;

[0218] Level E (below 60 points): Insufficient adaptability and need to re-evaluate suitability for space missions.

[0219] A virtual reality-based astronaut weightlessness training evaluation system, used to implement the above-mentioned virtual reality-based astronaut weightlessness training evaluation method, comprising:

[0220] Multimodal data acquisition module, used to collect eye movement data and vestibular response data of astronauts in a virtual weightless environment;

[0221] The oculovestibular coupling calculation module is used to calculate the oculovestibular coupling index and quantify the adaptation rate and stability of astronauts in the illusion environment;

[0222] Attention resource allocation analysis module, which uses the self-attention mechanism to achieve multimodal data fusion and analyze the astronauts' attention resource allocation pattern;

[0223] An individualized adaptation model building module is used to implement real-time fusion analysis of visual vestibular data based on deep convolutional neural networks and to build an illusion resistance prediction model using deep reinforcement learning algorithms;

[0224] The training effect evaluation and prediction module is used to evaluate astronauts' performance in a virtual weightless environment, predict their adaptability in actual space missions, and provide personalized training recommendations.

[0225] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

Claims

1. A virtual reality-based astronaut weightlessness training evaluation method, characterized in that: include: Collect eye movement data and vestibular response data of astronauts in a virtual weightless environment; The oculovestibular coupling index is calculated based on the eye movement data and the vestibular response data to quantify the adaptation rate and stability of the astronauts in the visual vestibular conflict environment. The oculovestibular coupling index is calculated based on the eye movement data and the vestibular response data, including: Extract eye movement feature vectors from eye movement data, including saccade velocity, fixation duration, fixation point distribution density, and pupil diameter change rate; Extracting vestibular feature vectors from vestibular response data, including head angular velocity, linear acceleration, and posture change rate; The oculovestibular coupling index was calculated based on the extracted eigenvectors, and the adaptation ability curve was drawn according to the coupling index values in the continuous time window. The initial adaptation rate, steady-state coupling value, adaptation time and fluctuation amplitude were extracted. Use the self-attention mechanism to achieve multimodal fusion of eye movement data and vestibular response data to analyze the astronauts' attention resource allocation pattern; Based on eye movement data and vestibular responses, the eye-vestibular coupling index, and the classification and evaluation indicators of attention resource allocation patterns, a personalized adaptation model is constructed to predict astronauts' resistance to illusions under different weightlessness conditions. The personalized adaptation model includes: A deep convolutional neural network is used to extract features from eye movement data and vestibular response data; Combining the astronauts' personal information with multimodal features, individual feature vectors are constructed through residual networks; Construct an illusion resistance prediction model based on the double Q network algorithm; Based on the astronauts' performance data during training, the parameters of the individualized adaptation model are continuously optimized; Based on eye movement data and vestibular responses, eye-vestibular coupling index, attention resource allocation pattern classification and evaluation indicators, and individualized adaptation model prediction results, the astronauts' performance in the virtual weightless environment is comprehensively evaluated and their adaptability in actual space missions is predicted.

2. The virtual reality-based astronaut weightlessness training evaluation method according to claim 1, characterized in that: The collection of eye movement data and vestibular response data of astronauts in a virtual weightless environment includes: Eye tracking equipment integrated into the virtual reality headset collects eye movement parameters, including eye movement trajectory, gaze point distribution, and pupil size changes; The inertial sensors fixed on the head-mounted virtual reality device collect the three-axis linear acceleration data, three-axis angular velocity data and head posture data of the astronaut's head; Timestamp alignment technology is used to ensure accurate synchronization of eye movement data and vestibular response data.

3. The virtual reality-based astronaut weightlessness training evaluation method according to claim 1, characterized in that: The multimodal fusion of eye movement data and vestibular response data using the self-attention mechanism includes: Convert eye movement data and vestibular response data into feature matrices and extract temporal features, frequency domain features, and spatial features; Apply the self-attention mechanism to calculate the attention weight matrix and identify the importance relationship between different features; Capturing the temporal dependencies of multimodal data based on bidirectional long short-term memory networks; By analyzing the self-attention weight matrix and bidirectional long short-term memory network output, the astronauts' attention resource allocation pattern is identified.

4. The virtual reality-based astronaut weightlessness training evaluation method according to claim 3, characterized in that: The attention resource allocation patterns include visual preference type, vestibular preference type, balance integration type and random switching type.

5. The virtual reality-based astronaut weightlessness training evaluation method according to claim 1, characterized in that: The comprehensive assessment of astronauts' performance in a virtual weightless environment includes: Calculate the astronauts' comprehensive adaptability score based on multiple indicators; The adaptability of astronauts is divided into five levels from A to E based on the comprehensive adaptability score; Use multiple visualization methods to display astronauts' performance and training progress in various dimensions; Predicting astronauts' adaptation performance in actual space missions based on support vector regression model; Generate targeted, personalized training recommendations based on assessment results and predictive analysis.

6. The virtual reality-based astronaut weightlessness training evaluation method according to claim 5, characterized in that: The comprehensive adaptability score is calculated based on the following parameters: the steady-state value of the oculovestibular coupling index, the astronaut's adaptation time, the fluctuation amplitude of the coupling index, the task completion score, and the attention resource allocation pattern score.

7. The virtual reality-based astronaut weightlessness training evaluation method according to claim 1, characterized in that: The virtual weightlessness environment includes: a rotating scene in which visual information is inconsistent with vestibular sensation, a scene for simulating spatial positioning tasks in a weightless state, and a scene for simulating operating tasks in a space capsule.

8. A virtual reality-based astronaut weightlessness training assessment system, characterized in that: A virtual reality-based astronaut weightlessness training assessment method for executing any one of claims 1-7, comprising: Multimodal data acquisition module, used to collect eye movement data and vestibular response data of astronauts in a virtual weightless environment; The oculovestibular coupling calculation module is used to calculate the oculovestibular coupling index and quantify the adaptation rate and stability of astronauts in the illusion environment; Attention resource allocation analysis module, which uses the self-attention mechanism to achieve multimodal data fusion and analyze the astronauts' attention resource allocation pattern; An individualized adaptation model building module is used to implement real-time fusion analysis of visual vestibular data based on deep convolutional neural networks and build an illusion resistance prediction model using deep reinforcement learning algorithms; The training effect evaluation and prediction module is used to evaluate astronauts' performance in a virtual weightless environment, predict their adaptability in actual space missions, and provide personalized training recommendations.

Images