A method and system for astronaut multi-modal emotion monitoring and psychological state evolution evaluation in a long-term space station residence environment based on a Transformer model

Abstract

The application discloses a kind of space station long-term stay environment-based astronaut multi-modal emotion monitoring and psychological state evolution evaluation method and system based on Transformer model, it is related to astronaut mental health monitoring and intelligent control field, to solve the current space station environment astronaut mental health monitoring There is single perception dimension, weak model generalization ability, evaluation feedback lag and other problems.The technical points of the present application: the evolution trend of astronaut emotional state is modeled based on the Transformer model of cross-modal mixed attention mechanism, the intra-modal self-attention vector representation of each modality is obtained through the self-attention mechanism, the self-attention output of one modality is used as the query, the self-attention output of the remaining modalities is used as the key and value for calculation, and the cross-modal feature interaction vector is generated;The cross-modal feature interaction vector and the corresponding intra-modal self-attention vector representation are weighted and summed to obtain a multi-modal unified vector representation;And the deep correlation mining of features is realized by combining double-scale position coding.

Description

Technical Field

[0001] This invention relates to the field of astronaut mental health monitoring and intelligent control technology. Specifically, it relates to a method, system, and storage medium for astronaut multimodal emotion monitoring and psychological state evolution assessment in a long-term space station stay environment based on the Transformer model under microgravity conditions. Background Technology

[0002] Currently, life support systems on space stations primarily focus on the physiological health of astronauts and the stable regulation of intravehicular environmental parameters. However, during long-term missions, astronauts are more prone to psychological problems such as anxiety, depression, and cognitive decline, which can severely impact mission execution and team safety. Existing psychological monitoring methods, such as questionnaires, self-assessments, or periodic interviews, suffer from problems such as delayed intervention, strong subjectivity, and poor continuity, making it difficult to meet the demands of future deep space missions for high-frequency, low-interference, and automated psychological monitoring systems. Therefore, there is an urgent need to establish an intelligent sensing system based on multi-source data to identify changes in astronauts' psychological states in real time and provide early warnings.

[0003] Existing space control systems primarily focus on the stable regulation of environmental parameters (including temperature, humidity, air pressure, and oxygen concentration) inside the space station module, as well as the collection of astronauts' basic vital signs (including heart rate and blood oxygen saturation). However, the monitoring and regulation of psychological aspects still rely on offline questionnaire assessments, self-reporting, or periodic interviews, which suffer from poor timeliness, strong subjectivity, and the inability to continuously track data. Furthermore, traditional algorithms often rely on shallow feature extraction of a single modality (including voice or facial expressions), making it difficult to cope with the complex and dynamic multimodal interactive environment of the space station.

[0004] To address the current issues in monitoring astronauts' mental health in the space station environment, such as limited perception dimensions, weak model generalization ability, and delayed assessment feedback, it is urgent to construct a Transformer emotion recognition and state assessment system that integrates multimodal data. This system would be able to acquire unstructured behavioral information of astronauts in real time and perform deep fusion modeling of emotional characteristics. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] Current monitoring of astronauts' mental health in the space station environment suffers from problems such as limited perception dimensions, weak model generalization ability, and delayed assessment feedback.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0008] This invention provides a method for monitoring multimodal emotions and assessing the evolution of psychological states of astronauts in a long-term space station stay environment based on the Transformer model, comprising the following steps:

[0009] Step 1: Real-time synchronous acquisition of high-dimensional multimodal data on various physiological and behavioral aspects of astronauts, including video images, voice, physiological signals, and text data, and processing of the data;

[0010] Step 2: Extract emotion-related feature vectors from each modality of data, and concatenate the features of different modalities according to the time window to obtain a multimodal feature vector;

[0011] Step 3: The Transformer model based on a cross-modal hybrid attention mechanism is used to model the evolution trend of astronauts' emotional states. A cascaded structure of "self-attention - mutual attention" is adopted. The self-attention mechanism obtains the intra-modal self-attention vector representations of each modality. Using the mutual attention mechanism, the self-attention output of one modality is used as the query, and the self-attention outputs of the other modalities are used as keys and values ​​for calculation, generating a cross-modal feature interaction vector. The cross-modal feature interaction vector is then weighted and summed with the corresponding intra-modal self-attention vector representation to obtain a unified multi-modal vector representation. Finally, dual-scale positional encoding is combined to achieve deep feature correlation mining.

[0012] Step 4: Input the high-level feature representation output by the Transformer model into the classifier for automatic identification and classification of the astronauts' emotional states.

[0013] Furthermore, the Transformer model introduces a priori constraint function for mental states:

[0014]

[0015] Represents the prior constraint loss function. Let be the mental state vector at time step t. Let be the mental state vector at the previous time step t−1. The mental state vector at time step t for representational prediction. and These are the regularization hyperparameters, This represents the squared Euclidean distance between vectors.

[0016] Furthermore, the high-dimensional multimodal data mentioned in step one includes: electroencephalogram (EEG) signal data, electrical conductance analysis (EDA) data, task load text data, behavior log text data, subjective satisfaction test result text data, and human-computer interaction voice data.

[0017] Furthermore, the data processing described in step one includes denoising and standardizing the data, as well as dimensionality reduction.

[0018] Furthermore, the data denoising and standardization process includes: performing bandpass filtering on the EEG signal data to remove power frequency interference and high-frequency noise components; further performing artifact correction, including the identification and removal of electrooculogram and electromyogram artifacts; using independent component analysis or wavelet transform to achieve effective separation of noise components; and finally, standardizing the data.

[0019] Smoothing filters are applied to the electrical conductance response (EDA) data to eliminate transient spikes and high-frequency noise, while baseline drift is corrected. Then, the EDA values ​​are normalized, and finally, trend separation is performed to extract the slow-varying baseline component and the fast-varying response component to improve the accuracy of signal analysis.

[0020] The task payload text data is cleaned, outliers are identified and removed, and missing data is supplemented; the data is standardized; for time series payload indicators, smoothing or downsampling operations are performed to ensure that their time resolution is consistent with other modal data.

[0021] Converting behavioral log text data into a computable quantitative feature representation specifically includes: counting the number of occurrences and duration of specific behavioral events within a fixed time window, or converting event types into classification codes; performing integrity checks and anomaly detection on the data, and normalizing the data; and mapping behavioral features to a unified time axis based on the timestamp information of the events to ensure the alignment of multimodal data in the time dimension.

[0022] The text data of subjective satisfaction test results are structured and organized, the scale rating results are converted into numerical form and normalized, and a continuous change curve is generated between two adjacent ratings through an interpolation algorithm to construct a subjective rating signal based on time continuity and align it to the overall data time axis.

[0023] Furthermore, the dimensionality reduction processing of the data includes the following steps:

[0024] For continuous high-dimensional time-series data of electroencephalogram (EEG) signal maps and electrical conductance of skin (EDA) maps, principal component analysis is used for preliminary dimensionality reduction.

[0025] Principal components are obtained by performing singular value decomposition (SVD) on the feature matrix;

[0026] The first k principal components are used to construct a projection matrix to obtain the dimensionality-reduced data.

[0027] Based on the classification or discrete characteristics of task payload text data, behavior log text data, and subjective satisfaction test result text data, numerical encoding and standardization are first performed, followed by dimensionality reduction using Linear Discriminant Analysis (LDA). The objective function for LDA dimensionality reduction is:

[0028]

[0029] in, Represents the inter-class scatter matrix. Represents the intra-class scatter matrix. It is the optimal projection matrix.

[0030] Furthermore, the step of splicing features from different modalities according to a time window includes:

[0031] The feature sequences after dimensionality reduction of each modality are concatenated to obtain the multimodal feature representation vector at each time step;

[0032] Embedding mapping is performed on each feature vector:

[0033]

[0034] in, For a trainable embedding mapping matrix, Here, is the input token dimension required by the Transformer model, and D is the total dimension of the features after dimensionality reduction for each modality. Let be the multimodal feature representation vector at time step t, where T is the length of the time series and ϕ is the mapping function;

[0035] Position coding vectors are generated using dual-scale position coding, specifically by combining long-period scale coding. (t) and short-period scale encoding (t) Weighted fusion is performed to generate the final location encoding vector. :

[0036] (t) (t)

[0037] in, For fusion weighting coefficients.

[0038] The position encoding function is:

[0039] ;

[0040] Where t represents the time step index and i represents the vector dimension index;

[0041] The final input token sequence for the Transformer model is:

[0042] .

[0043] Furthermore, step four also includes: generating multimodal physiological response feature sequences for different emotional states through a data simulation environment, evaluating the performance of the trained Transformer model based on the simulation feature sequences, and further adaptively optimizing and online correcting the model.

[0044] This invention provides a system for monitoring and assessing the multimodal emotions and psychological state evolution of astronauts in a long-term space station stay environment based on the Transformer model. The system has program modules corresponding to the steps of any of the above-described technical solutions, and executes the steps of the above-described method for monitoring and assessing the multimodal emotions and psychological state evolution of astronauts in a long-term space station stay environment based on the Transformer model when running.

[0045] This invention provides a computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps in the method for monitoring and assessing the multimodal emotions and psychological state evolution of astronauts in a long-term space station stay environment based on the Transformer model, as described in any of the above technical solutions.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] This invention introduces a cross-modal hybrid attention mechanism for feature vectors from various modalities, including video, speech, and physiological signals. It achieves information interaction and deep correlation mining between speech, physiology, and behavior through a cascaded "self-attention-mutual attention" approach. This enhances the sensitivity of feature detection for anxiety and depression, reduces long-term prediction errors, and enables deep fusion and semantic alignment between modalities. It can more comprehensively capture subtle emotional changes in individuals under complex situations, effectively reducing false positive and false negative rates. Furthermore, by combining dual-scale positional coding, it achieves deep correlation mining of speech, physiological, and behavioral features, resulting in more robust and reliable emotion assessment results. This makes it particularly suitable for monitoring psychological states in high-reliability aerospace mission environments.

[0048] The method of this invention has strong real-time performance, enabling rapid response to changes in astronauts' emotions. It can complete the assessment of emotional state in milliseconds, realize real-time monitoring and early warning of astronauts' emotional state, and help to take immediate intervention measures when emotional fluctuations occur, thereby improving the psychological safety of space missions.

[0049] This invention is easy to deploy and designed for the limited computing resources and autonomous operation requirements of the space station. Through model optimization using the Transformer model, it reduces system resource consumption while ensuring performance. It can run directly on the space station terminal equipment without relying on ground communication, realizing edge computing deployment and providing reliable emotion monitoring services for space missions. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method for monitoring multimodal emotions and assessing the evolution of psychological state of astronauts in a long-term space station stay environment based on the Transformer model, as described in this embodiment of the invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] Combination Figure 1 As shown, this invention provides a method for monitoring astronauts' multimodal emotions and assessing their psychological state evolution in a long-term space station stay environment based on the Transformer model, comprising the following steps:

[0054] Step 1: Real-time synchronous acquisition of high-dimensional multimodal data on various physiological and behavioral aspects of astronauts, including video images, voice, and text data, and processing of the data;

[0055] Step 2: Extract emotion-related feature vectors from each modality of data, and concatenate the features of different modalities according to the time window to obtain a multimodal feature vector;

[0056] Step 3: The Transformer model based on the cross-modal hybrid attention mechanism is used to model the evolution trend of astronauts' emotional state. The "self-attention-mutual attention" cascade method is adopted. The dynamic interaction relationship between different modal features is captured through the mutual attention mechanism. A cross-modal feature enhancement vector representation is established, which is fused with the self-attention vectorization representation within the modality to obtain a multi-modal enhanced unified vector representation; and combined with dual-scale position encoding.

[0057] Step 4: Input the high-level feature representation output by the Transformer model into the classifier for automatic identification and classification of the astronauts' emotional states.

[0058] This invention introduces cross-modal hybrid attention into the Transformer backbone network, achieving information interaction and deep correlation mining between speech, physiology, and behavior through a cascaded "self-attention-mutual attention" approach. This enhances the sensitivity of feature detection, such as anxiety / depression, and reduces long-term prediction errors. It outputs individual emotional states and psychological evolution curves, forming a precise assessment of astronauts' psychological states. Simultaneously, a multi-head attention mechanism and feedforward neural network are designed to improve the accuracy of emotion recognition; furthermore, multi-level feature fusion and information interaction reduce long-term prediction errors.

[0059] This invention introduces an attention mechanism to dynamically weight the importance of different modalities and combines it with supervised learning based on psychological state labels, thereby achieving accurate identification and trend prediction of psychological states such as anxiety, fatigue, and depression. It features end-to-end adaptive control capabilities, supports operation in computationally limited cabin environments, and can continuously optimize recognition performance by combining historical behavioral data and real-time feedback.

[0060] Specific Implementation Scheme Two: The Transformer model introduces a priori constraint function for mental states:

[0061]

[0062] This represents the prior constraint loss function, used to guide the model to incorporate prior knowledge in emotion recognition or mental state assessment tasks, thereby enhancing the stability and rationality of responses to changes in mental state. Let be the mental state vector at time step t. Let be the mental state vector at the previous time step t−1. The mental state vector at time step t for representational prediction. and These are regularization hyperparameters, used to control the weight of each constraint term. This represents the squared Euclidean distance between vectors, commonly used to measure the similarity between vectors. This implementation scheme is otherwise the same as specific implementation scheme one.

[0063] Specific implementation plan three: In order to accurately simulate the emotional reactions that astronauts gradually accumulate in the high-pressure working environment of the space station, this invention adopts a multimodal data acquisition process.

[0064] Step 1) Task Scenario Setup and Activation

[0065] First, a scenario is constructed for space station missions. Using a mission management platform, an actual space station mission is simulated, including setting mission complexity, determining mission operation duration, and defining corresponding resource constraints. Researchers initiate the scenario through the mission management platform. The system clearly defines the mission phases, including emergency response, collaborative operations, and long-duration inspections, ensuring the expected stress and emotional triggering requirements for mission load, duration, and resource consumption levels. This method effectively constructs high-load simulation scenarios similar to actual space station missions to assess astronauts' psychological and physiological states and coping abilities under high-pressure environments.

[0066] Step 2) Preparation and Calibration of Physiological Signal Sensors

[0067] Before the formal execution of the experimental mission, the astronauts were equipped with and activated a multi-channel EEG (electroencephalography) headset and an electroreactivity (EDA) wristband physiological signal acquisition device. Subsequently, all devices underwent real-time signal quality checks and calibrations to ensure clear, interference-free signals and that the data sampling frequency remained above 256 Hz. Simultaneously with the acquisition of physiological signals, a clock synchronization mechanism was set up to facilitate subsequent data synchronization and alignment.

[0068] Step 3) Behavioral data recording and task log preparation

[0069] During mission execution, the astronauts' specific behaviors are continuously recorded through an automated mission operating system and human-machine interface; including:

[0070] Step 3.1) Record the astronauts' operational command inputs in real time through the space station mission control terminal interface, including the specific timestamp and command content of each operation. This content covers system mode switching, equipment startup or shutdown, parameter setting adjustments, navigation command input, and communication interactions. The recording uses millisecond-level time resolution to form a complete and accurate sequence of operational behaviors.

[0071] Step 3.2) During the astronauts' mission, the system automatically monitors and records various error events that occur during operations, including inputting incorrect values, selecting error modes, equipment malfunction alarms, and redundant operations. In addition to recording the time and type of error events, the system also statistically analyzes the frequency and severity of errors, generating event records categorized as minor, moderate, and severe for subsequent psychological and emotional analysis.

[0072] Step 3.3) The system continuously monitors the astronaut's actions in calling auxiliary functions and system support, including requesting mission prompts, initiating emergency handling procedures, and calling operation instructions. Each call is recorded with the time of occurrence, the specific function type called, and the astronaut's subsequent behavior after the call.

[0073] Step 4) Interpersonal relationship data collection and subjective satisfaction data entry

[0074] Simultaneously, the intravehicular communication system was activated to extract real-time features from the astronauts' voice interactions. Specific measures included real-time extraction and analysis of emotional features in interpersonal dialogues, including tone, speech rate, and volume, and recording the frequency, duration, and participants in the dialogues between astronauts.

[0075] Upon completion of each mission or key mission phase, astronauts immediately conduct self-assessments using electronic questionnaires. Subjective data collection methods include the NASA Mission Load Index (NASA-TLX) for evaluating mission load and stress; the SAM Self-Affect Scale for measuring current subjective feelings of pleasure and arousal; and the POMS scale for measuring psychological state. The questionnaires automatically record the completion time to ensure strict time matching between subjective and objective data.

[0076] Step 5) Data Synchronization and Real-time Storage

[0077] Multimodal data collected by astronauts during the mission, including EEG data, electrical conductance analysis (EDA), mission payload indicators, behavioral logs, interpersonal interaction characteristics, and subjective satisfaction evaluation data, are synchronized in real time via a local area network. These data are assigned a uniform, high-precision timeline during acquisition to ensure strict temporal consistency between data from different sources.

[0078] After unifying the time series, these data with synchronized timestamps are transmitted to the master data server deployed in the space station or ground simulation environment. The server automatically executes the data integration and standardization process, storing and archiving the data of different modalities in a unified data format, forming a multimodal data set with a clear structure and strict time series alignment.

[0079] Data storage not only facilitates subsequent data retrieval and analysis but also provides a high-quality, structured, and unified input for the Transformer emotion recognition model. This enables the model to effectively capture and analyze the complex psychological and emotional state changes of astronauts, improving the accuracy and generalization ability of emotion recognition and prediction. Other aspects of this implementation scheme are the same as in Implementation Scheme Two.

[0080] Specific Implementation Plan Four: The data processing described in Step One includes denoising and standardization, as well as dimensionality reduction. All other aspects of this implementation plan are the same as in Specific Implementation Plan Three.

[0081] Specific Implementation Scheme 5: The data denoising and standardization process includes: performing bandpass filtering on the EEG signal data to remove power frequency interference and high-frequency noise components, with a frequency range of 0.5–45Hz; further performing artifact correction, including the identification and removal of electrooculography and electromyography artifacts, using independent component analysis or wavelet transform to achieve effective separation of noise components; and finally, standardizing the data by using Z-score normalization to adjust the signal amplitude to ensure comparability between different subjects and different channels.

[0082] A smoothing filtering strategy was applied to the electrical conductance analysis (EDA) data, using moving average or low-pass filtering to eliminate transient spikes and high-frequency noise, while correcting for baseline drift. Then, the EDA values ​​were normalized and scaled to a uniform range to eliminate the influence of individual differences in baseline conductivity. Finally, trend separation was performed to extract slowly varying baseline components and rapidly varying response components to improve signal analysis accuracy.

[0083] The task payload text data is cleaned, outliers are identified and removed, and missing data is supplemented. Based on the physical dimensional characteristics of each indicator, the data is scaled to a similar numerical range using Min-Max normalization or standardization methods to avoid imbalance of feature weights due to differences in dimensionality. For time series load indicators, smoothing or downsampling operations are performed to ensure that their time resolution is consistent with that of other modal data.

[0084] Converting behavioral log text data into a computable quantitative feature representation specifically includes: counting the number of occurrences and duration of specific behavioral events within a fixed time window, or converting event types into classification codes; performing integrity checks and anomaly detection on the data, and normalizing the data; and mapping behavioral features to a unified time axis based on the timestamp information of the events to ensure the alignment of multimodal data in the time dimension.

[0085] The textual data of subjective satisfaction test results is structured and organized, converting the scale ratings into numerical form and normalizing them to limit the rating range to a uniform interval. Considering the low frequency of scale data collection, an interpolation algorithm is used to generate a continuous change curve between two adjacent ratings, constructing a subjective rating signal based on time continuity and aligning it to the overall data time axis to ensure that the model can obtain the latest subjective evaluation information as auxiliary input at each time step. Other aspects of this implementation scheme are the same as in specific implementation scheme four.

[0086] Specific implementation plan six: The dimensionality reduction processing of the data includes the following steps:

[0087] For continuous high-dimensional time-series data of electroencephalogram (EEG) signal mapping and electrical conductance analysis (EDA) mapping, principal component analysis is used for preliminary dimensionality reduction; let the original feature matrix of a mode be represented as:

[0088]

[0089] Where T is the time series length and N is the feature dimension.

[0090] Principal components are obtained by performing singular value decomposition (SVD) on the feature matrix:

[0091]

[0092] The first k principal components are used to construct the projection matrix. Then the data after dimensionality reduction is:

[0093]

[0094] To address the classification or discrete characteristics of task payload text data, behavior log text data, and subjective satisfaction test result text data, numerical encoding and standardization are first performed. Then, linear discriminant analysis (LDA) is used for further dimensionality reduction to maximize the ability to distinguish between different emotional states. The LDA dimensionality reduction objective function is:

[0095]

[0096] in, Represents the inter-class scatter matrix. Represents the intra-class scatter matrix. This is the optimal projection matrix, obtained by solving the above objective function, used to project high-dimensional features into a low-dimensional space. Other aspects of this implementation scheme are the same as in specific implementation scheme five.

[0097] Specific implementation plan seven: The step of splicing features of different modalities according to a time window includes:

[0098] The feature sequences after dimensionality reduction from each modality are concatenated to obtain the multimodal feature representation vector at each time step:

[0099]

[0100] Where D represents the total feature dimension after dimensionality reduction for each mode, i.e.:

[0101]

[0102] This multimodal feature vector sequence is denoted as:

[0103]

[0104] To enable the data to be processed by the Transformer model, each feature vector needs to be further embedded. Let the embedding mapping function be ϕ, which is defined as:

[0105]

[0106] in, For a trainable embedding mapping matrix, The required input token dimension for the Transformer model;

[0107] Since Transformer does not have a natural ability to perceive the order of sequences, positional encoding must be added to the embedded vector.

[0108] A dual-scale positional encoding method is adopted, combined with long-period (global) scale encoding. (t) and short-period (local) scale coding (t), generating the final position encoding vector. .

[0109] (t) is used to capture global temporal information of long sequences, using a cardinality. Calculation of the sine function;

[0110] (t) is used to capture local time dependencies, using cardinality. Calculation of the cosine function,

[0111]

[0112]

[0113] in, Represents the time step index; Index representing the dimension of the vector; Indicates the model dimension; Values or ;

[0114] Final position encoding vector It is achieved by encoding long-scale data. (t) and short-scale coding (t) is weighted and summed to obtain:

[0115] (t) (t)

[0116] in, The fusion weight coefficients are parameters obtained through adaptive learning during model training, used to dynamically determine the fusion weights for information at both long and short time scales.

[0117] The position encoding function is:

[0118] ;

[0119] Where t represents the time step index and i represents the vector dimension index;

[0120] The embedding vector and the positional encoding are added together to obtain the final input token sequence of the Transformer model:

[0121] .

[0122] This implementation plan is otherwise the same as Specific Implementation Plan Six.

[0123] Specific implementation plan eight: The Transformer model includes a feature embedding layer, a position encoding layer, a multi-head self-attention layer, a feedforward neural network layer, and an output classification layer.

[0124] The input sequence is first mapped to the model dimension `model_dim` through a linear embedding layer. Subsequently, a position encoding layer adds positional information to the input feature sequence, enabling the transformer to capture the temporal information of the sequence.

[0125] ,

[0126] The input, after positional encoding, enters the Transformer encoder, where the multi-head self-attention mechanism is expressed as follows:

[0127]

[0128] The model employs an encoder structure consisting of multiple stacked standard Transformer encoder layers. Each encoder layer includes a self-attention mechanism and a feedforward network, supplemented by residual connections and layer normalization operations. The encoder output is the hidden state representation for all time steps. .

[0129] The calculation method for each attention point is as follows:

[0130]

[0131] Where Q, K, and V represent the query, key, and value input to the model, respectively. For trainable parameter matrix

[0132] The final state vector (final_state) of the Transformer encoder output sequence is fed into a fully connected layer classifier to predict the astronaut's current mood category (including calm, anxiety, and fatigue). This implementation scheme is otherwise identical to specific implementation scheme seven.

[0133] Specific implementation plan nine: Step four also includes: generating multimodal physiological response feature sequences for different emotional states through a data simulation environment, evaluating the performance of the trained Transformer model based on the simulation feature sequences, and further adaptively optimizing and online correcting the model; specifically:

[0134] First, a data simulation environment for astronaut emotional changes is constructed. This simulation environment is based on actual mission scenarios and combines typical physiological response characteristics (EEG brainwave patterns, EDA skin conductance response patterns) and behavioral indicators (mission load stress, abnormal behavior) of different emotional states to design a series of virtual mission scenarios and generate corresponding simulation data sequences.

[0135] Optimization was performed for the space station environment, as astronauts' facial expressions and physiological signals may differ from those on Earth under microgravity conditions. By employing mean centering and standard deviation scaling to preprocess the modal characteristics, the differences in physiological baselines between individual astronauts and dimensional differences between modalities under microgravity were effectively eliminated, ensuring the stability of the multimodal evaluation model in microgravity.

[0136] Using simulation models to generate emotion-related multimodal feature sequences. , is represented as:

[0137] ,

[0138] Simulation feature sequence The input is fed into a trained Transformer emotion recognition model for prediction, and the resulting sequence of emotion states is obtained from the model's output. Its form is as follows:

[0139]

[0140] Where C is the number of emotion state categories, and the model output is the probability distribution of each emotion state.

[0141] The simulation process compares the model's predictions with the simulated preset real emotion labels. The accuracy and error rate are calculated to quantitatively evaluate the generalization performance and stability of the Transformer model in different task scenarios. Simulation evaluation identifies the model's prediction bias in specific situations or under certain emotional states.

[0142] After identifying model prediction errors or biases during the data simulation phase, a dynamic correction process is further implemented. Specifically, the dynamic correction process aims to adaptively optimize and correct the Transformer emotion recognition model in real time to continuously improve its performance in actual astronaut emotion detection tasks. An online adaptive update mechanism is employed to correct prediction biases. The online adaptive optimization process monitors the deviation between the Transformer model's prediction results in the actual mission environment and subjective satisfaction feedback (i.e., the astronaut's actual self-reported emotional state) in real time. Simultaneously, the dynamic correction process also considers the unique impact of individual astronaut differences and the microgravity environment on physiological responses. An error prediction model is constructed using historical information of accumulated errors to estimate the trend of emotion prediction errors in real time, further improving the accuracy and response speed of dynamic correction. Through the above dynamic correction and online adaptive optimization processes, the Transformer emotion recognition system maintains long-term stable and high-precision emotion state monitoring capabilities in space station mission scenarios.

[0143] The multimodal emotion monitoring and psychological state evolution assessment method (algorithm) for astronauts in a long-term space station stay environment based on the Transformer model proposed in this invention is the underlying technical core of this invention, and various products can be derived based on the algorithm.

[0144] Based on the method proposed in this invention, a system for monitoring astronauts' multimodal emotions and evaluating their psychological state evolution in a long-term space station stay environment based on the Transformer model has been developed using a programming language. The system has program modules corresponding to the steps of the above-mentioned technical solution, and executes the steps in the above-mentioned method for monitoring astronauts' multimodal emotions and evaluating their psychological state evolution in a long-term space station stay environment based on the Transformer model when it is running.

[0145] The developed system (software) computer program is stored on a computer-readable storage medium. This computer program is configured to, when called by a processor, implement the steps of the aforementioned method for monitoring astronauts' multimodal emotions and assessing their psychological state evolution under a long-term stay on a space station based on the Transformer model. In other words, the invention is materialized on a carrier, becoming a computer program product.

[0146] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The computational programs (also referred to as programs, software, software applications, or code) of this invention include machine instructions of a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0148] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A method for monitoring multimodal emotions and assessing the evolution of psychological states of astronauts in a long-term space station stay environment based on the Transformer model, characterized in that: Includes the following steps: Step 1: Real-time synchronous acquisition of high-dimensional multimodal data on various physiological and behavioral aspects of astronauts, including video images, voice and text data, and processing of the data; Step 2: Extract emotion-related feature vectors from each modality of data, and concatenate the features of different modalities according to the time window to obtain a multimodal feature vector; Step 3: The Transformer model based on a cross-modal hybrid attention mechanism is used to model the evolution trend of astronauts' emotional states. A cascaded structure of "self-attention - mutual attention" is adopted. The self-attention mechanism obtains the intra-modal self-attention vector representations of each modality. Using the mutual attention mechanism, the self-attention output of one modality is used as the query, and the self-attention outputs of the other modalities are used as keys and values ​​for calculation, generating a cross-modal feature interaction vector. The cross-modal feature interaction vector is then weighted and summed with the corresponding intra-modal self-attention vector representation to obtain a unified multi-modal vector representation. Finally, dual-scale positional encoding is combined to achieve deep feature correlation mining. Step 4: Input the high-level feature representation output by the Transformer model into the classifier for automatic identification and classification of the astronauts' emotional states.

2. The method according to claim 1, characterized in that: The Transformer model introduces a priori constraint function for mental states: Represents the prior constraint loss function. Let be the mental state vector at time step t. Let be the mental state vector at the previous time step t−1. The mental state vector at time step t for representational prediction. and These are the regularization hyperparameters, This represents the squared Euclidean distance between vectors.

3. The method according to claim 2, characterized in that: The high-dimensional multimodal data mentioned in step one includes: electroencephalogram (EEG) signal data, electrical conductance analysis (EDA) data, task load text data, behavior log text data, subjective satisfaction test result text data, and human-computer interaction voice data.

4. The method according to claim 3, characterized in that: The data processing described in step one includes denoising and standardizing the data, as well as dimensionality reduction.

5. The method according to claim 4, characterized in that: The data denoising and standardization process includes: performing bandpass filtering on the EEG signal data to remove power frequency interference and high-frequency noise components; further performing artifact correction, including the identification and removal of electrooculography and electromyography artifacts; using independent component analysis or wavelet transform to achieve effective separation of noise components; and finally, standardizing the data. Smoothing filters are applied to the electrical conductance response (EDA) data to eliminate transient spikes and high-frequency noise, while baseline drift is corrected. Then, the EDA values ​​are normalized, and finally, trend separation is performed to extract the slow-varying baseline component and the fast-varying response component to improve the accuracy of signal analysis. The task payload text data is cleaned, outliers are identified and removed, and missing data is supplemented; the data is standardized; for time series payload indicators, smoothing or downsampling operations are performed to ensure that their time resolution is consistent with other modal data. Converting behavioral log text data into a computable quantitative feature representation specifically includes: counting the number of occurrences and duration of specific behavioral events within a fixed time window, or converting event types into classification codes; performing integrity checks and anomaly detection on the data, and normalizing the data; and mapping behavioral features to a unified time axis based on the timestamp information of the events to ensure the alignment of multimodal data in the time dimension. The text data of subjective satisfaction test results are structured and organized, the scale rating results are converted into numerical form and normalized, and a continuous change curve is generated between two adjacent ratings through an interpolation algorithm to construct a subjective rating signal based on time continuity and align it to the overall data time axis.

6. The method according to claim 5, characterized in that: The dimensionality reduction process for the data includes the following steps: For continuous high-dimensional time-series data of electroencephalogram (EEG) signal maps and electrical conductance of skin (EDA) maps, principal component analysis is used for preliminary dimensionality reduction. Principal components are obtained by performing singular value decomposition (SVD) on the feature matrix; The first k principal components are used to construct a projection matrix to obtain the dimensionality-reduced data. Based on the classification or discrete characteristics of task payload text data, behavior log text data, and subjective satisfaction test result text data, numerical encoding and standardization are first performed, followed by further dimensionality reduction using Linear Discriminant Analysis (LDA). The objective function for LDA dimensionality reduction is: in, Represents the inter-class scatter matrix. Represents the intra-class scatter matrix. It is the optimal projection matrix.

7. The method according to claim 6, characterized in that: The process of splicing features from different modalities according to a time window includes: The feature sequences after dimensionality reduction of each modality are concatenated to obtain the multimodal feature representation vector at each time step; Embedding mapping is performed on each feature vector: in, For a trainable embedding mapping matrix, Here, is the input token dimension required by the Transformer model, and D is the total dimension of the features after dimensionality reduction for each modality. Let t be the multimodal feature representation vector at time step t, where T is the length of the time series and ϕ is the mapping function; Position coding vectors are generated using dual-scale position coding, specifically by combining long-period scale coding. (t) and short-period scale encoding (t) Weighted fusion is performed to generate the final location encoding vector. : (t) (t) in, For fusion weighting coefficients. The position encoding function is: ； Where t represents the time step index and i represents the vector dimension index; The final input token sequence for the Transformer model is: 。 8. The method according to claim 7, characterized in that: Step four also includes: generating multimodal physiological response feature sequences for different emotional states through a data simulation environment, evaluating the performance of the trained Transformer model based on the simulation feature sequences, and further adaptively optimizing and online correcting the model.

9. A system for monitoring astronauts' multimodal emotions and assessing their psychological state evolution under long-term stay conditions on a space station based on the Transformer model, characterized in that: The system has a program module corresponding to the steps of the method described in any one of claims 1 to 8, and executes the steps in the above-described method for monitoring astronauts' multimodal emotions and assessing their psychological state evolution in a long-term space station stay environment based on the Transformer model when it is run.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the method for monitoring and assessing the multimodal emotions and psychological state evolution of astronauts in a long-term space station stay environment based on the Transformer model, as described in any one of claims 1 to 8.

Images