Training method and device of fine-grained mental workload interpretation model of brain-controlled mechanical arm, and mental workload interpretation method and device

Abstract

The present application relates to the technical field of artificial intelligence, and provides a training method and a brain power load interpretation method and device for a fine-grained brain power load interpretation model of a brain-controlled mechanical arm, which comprises: cross-reconstructing another-dimensional latent feature representation through first and second feature decoders to obtain first and second reconstructed feature views; constructing a cross-feature view cross-reconstruction loss based on the difference between the first feature view and the first reconstructed feature view and the difference between the second feature view and the second reconstructed feature view, so that the model, through cross-dimensional mutual mapping and reconstruction constraints in the training process, excavates and learns shared neural representations commonly contained between different feature views, and then effectively suppresses specific noise interference in a single feature encoding branch by using the correlation of multi-view features, extracts stable neural features with strong discriminability and robustness, and significantly enhances the ability of the brain power load interpretation model to capture weak neural representation differences between adjacent load levels.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, a method for interpreting mental workload, and a device for a fine-grained mental workload interpretation model of a brain-controlled robotic arm. Background Technology

[0002] Mental workload is of great significance in many fields such as transportation, medicine, industry, and aerospace remote operation, and has become a key physiological indicator for measuring human reliability and optimizing human-machine function allocation. With the continuous development of human-computer interaction systems, the use of passive brain-computer interface technology to monitor and analyze the mental workload of operators in real time through neurophysiological signals such as electroencephalogram (EEG) has important application value for ensuring the safety of complex engineering systems and improving operational efficiency.

[0003] Currently, brain-computer interface-based mental workload identification technology has made progress in experimental paradigms and identification algorithms. Existing technologies typically use single-dimensional EEG features and employ conventional machine learning methods or deep neural networks to construct decoding models to classify different levels of mental workload.

[0004] However, existing technologies are limited in their granularity for practical applications. Current classification schemes are mostly confined to coarse-grained distinctions such as high / low or high / medium / low, making it difficult to accurately depict the continuous and subtle changes in mental workload as task difficulty increases, and thus unable to support the needs of higher-precision adaptive interaction. Furthermore, the robustness of decoding performance is poor. Due to the strong non-stationarity and significant individual differences in EEG signals, the differences in neural representations between adjacent workload levels are extremely weak. Traditional recognition methods based on single-feature views lack effective modeling of the correlation and consistency between multi-view features, making them susceptible to interference from noise in specific feature branches. This makes it difficult to extract stable neural representations with strong discriminative and generalizable properties, resulting in the model's recognition accuracy and stability failing to meet the requirements of practical teleoperation tasks when faced with fine-grained workload level classification. Summary of the Invention

[0005] This invention provides a training method, a brain load interpretation method, and a device for a fine-grained brain load interpretation model of a brain-controlled robotic arm. This addresses the challenges of existing brain load recognition technologies, which are limited by coarse-grained classification paradigms, EEG non-stationarity, and large individual differences. Because it is difficult to accurately characterize the continuous subtle fluctuations in load as the task difficulty changes, the decoding robustness and recognition accuracy under fine-grained classification cannot meet the actual teleoperation requirements.

[0006] This invention provides a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm. The fine-grained mental workload interpretation model is obtained by iteratively executing the following steps until a preset iteration termination condition is met: An initial interpretation model is determined, and training sample data and fine-grained mental workload level labels corresponding to the training sample data are obtained. The initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder. The training sample data includes the electroencephalogram (EEG) signals of the operator when performing a teleoperation task. At least two-dimensional feature extraction is performed on the electroencephalogram (EEG) signal to obtain a first feature view and a second feature view; The first feature view and the second feature view are respectively input into the initial interpretation model. The first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation. The first latent feature representation is reconstructed based on the second feature decoder to obtain a second reconstructed feature view, and the second latent feature representation is reconstructed based on the first feature decoder to obtain a first reconstructed feature view; Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, a cross-feature view cross-reconstruction loss is determined. Based on the cross-feature view cross-reconstruction loss, a target loss is determined, and the model parameters of the initial interpretation model are updated based on the target loss.

[0007] According to the training method of a fine-grained mental workload interpretation model for a brain-controlled robotic arm provided by the present invention, the step of determining the target loss based on the cross-feature view cross-reconstruction loss includes: The first latent feature representation and the second latent feature representation are fused to obtain a fused neural representation; The fused neural representation is input into the classifier in the initial interpretation model to obtain the predicted load level result output by the classifier; Based on the fine-grained mental workload level labels and the predicted workload level results, the classification loss is determined; The target loss is determined based on the cross-feature view cross-reconstruction loss and the classification loss.

[0008] According to the present invention, a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm is provided, wherein the initial interpretation model further includes a third feature encoder; The step of fusing the first latent feature representation and the second latent feature representation to obtain a fused neural representation includes: Obtain the spatial distribution information of the electrodes corresponding to the electroencephalogram (EEG) signals, and construct spatial topological features based on the spatial distribution information of the electrodes; The spatial topological features are encoded based on the third feature encoder to obtain a third latent feature representation; The first latent feature representation, the second latent feature representation, and the third latent feature representation are fused to obtain the fused neural representation.

[0009] According to the training method of a fine-grained mental workload interpretation model for a brain-controlled robotic arm provided by the present invention, the step of determining the target loss based on the cross-feature view cross-reconstruction loss and the classification loss includes: Obtain the weight parameters of each network layer in the initial interpretation model, and determine the sparse constraint loss based on the sum of the absolute values ​​of each weight parameter; The target loss is determined based on the sparse constraint loss, the cross-feature view cross-reconstruction loss, and the classification loss.

[0010] According to the present invention, a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm includes, in which, acquiring training sample data, comprises: Obtain raw EEG signal fragments; The original EEG signal segments were segmented using sliding windows of different lengths to obtain multiple sub-segments of signals corresponding to each time scale. The EEG features corresponding to each of the sub-segments are extracted, and the EEG features at the same time scale are aggregated to obtain representative feature samples for each time scale. Based on representative feature samples of each time scale, the original EEG signal segments are expanded to obtain the training sample data.

[0011] According to the present invention, a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm is provided, wherein the teleoperation task includes a continuous motion control task of the spatial robotic arm; the continuous motion control task of the spatial robotic arm is an operation task in which the operator controls the spatial robotic arm to complete continuous motion control through electroencephalogram (EEG) signals, and the continuous motion control task of the spatial robotic arm is used to induce fine-grained mental workload of the operator. The steps for determining the fine-grained mental workload level label include: The difficulty level of each basic task is determined based on at least one of the following: the spatial structural complexity of the target object, visual perception conditions, the number of operational monitoring parameters, and continuous motion control constraints. Collect the EEG signals of the operator when performing the continuous motion control task of the space robotic arm at each of the basic task difficulty levels; Obtain the results of the subjective evaluation scale filled out by the operators for each of the basic task difficulty levels; Based on the basic task difficulty level and the results of the subjective evaluation scale, the task EEG signals are labeled with fine-grained mental workload level to obtain the fine-grained mental workload level label.

[0012] According to the present invention, a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm is provided, wherein the spatial structural complexity of the target object is determined based on the geometric shape of the target object to be operated in the continuous motion control task of the spatial robotic arm; the geometric shape includes at least one of a sphere, cylinder, triangular prism, cross prism and pentagon. The visual perception conditions are determined based on the number of cameras deployed on the space robotic arm and the visual information dimension; the visual information dimension characterizes the richness of information about the target state of the target object and the state of the space robotic arm simultaneously acquired by the operator. The number of operation monitoring parameters is determined based on the number of robotic arm state parameters that the operator simultaneously monitors when performing the continuous motion control task of the space robotic arm; the robotic arm state parameters include at least one of the following: joint position, joint speed, robotic arm posture, end effector position, end effector pose, position of the target object, and relative distance between the target object and the space robotic arm; The continuous motion control constraints are based on at least one of the following in the continuous motion control task of the space manipulator: the allowable operational error of the task, the task completion time, the target docking accuracy between the space manipulator and the target object, the motion trajectory complexity of the space manipulator, and the motion range.

[0013] This invention also provides a fine-grained method for interpreting mental workload, comprising: Acquire real-time EEG signals when a user performs a teleoperation task; The real-time EEG signal is input into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model. The fine-grained mental workload interpretation model is obtained by executing the training method of the fine-grained mental workload interpretation model of the brain-controlled robotic arm.

[0014] The present invention also provides a training device for a fine-grained mental workload interpretation model of a brain-controlled robotic arm, comprising the following units: A determining unit is used to determine an initial interpretation model, acquire training sample data and fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder and a second feature decoder; the training sample data includes electroencephalogram (EEG) signals of an operator performing a teleoperation task. The feature extraction unit is used to extract features from the EEG signal in at least two dimensions to obtain a first feature view and a second feature view. The encoding unit is used to input the first feature view and the second feature view into the initial interpretation model respectively, and the first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation; The reconstruction unit is configured to reconstruct the first latent feature representation based on the second feature decoder to obtain a second reconstructed feature view, and to reconstruct the second latent feature representation based on the first feature decoder to obtain a first reconstructed feature view. The update unit is configured to determine a cross-feature view cross-reconstruction loss based on the difference between the first feature view and the first reconstructed feature view, and the difference between the second feature view and the second reconstructed feature view, determine a target loss based on the cross-feature view cross-reconstruction loss, and update the model parameters of the initial interpretation model based on the target loss.

[0015] The present invention also provides a fine-grained mental workload interpretation device, comprising the following units: The acquisition unit is used to acquire real-time EEG signals when the user performs teleoperation tasks. The prediction unit is used to input the real-time EEG signal into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model. The fine-grained mental workload interpretation model is obtained by executing the training method of the fine-grained mental workload interpretation model of the brain-controlled robotic arm.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm as described above, or implements the fine-grained mental workload interpretation method.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm as described above, or implements the fine-grained mental workload interpretation method.

[0018] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm as described above, or implements the fine-grained mental workload interpretation method.

[0019] The present invention provides a training method, a brain load interpretation method, and a device for a fine-grained brain load interpretation model for a brain-controlled robotic arm. Based on a first feature view and a second feature view extracted from EEG signals, the method performs cross-reconstruction of the potential feature representation of another dimension through a first and a second feature decoder to obtain a first reconstructed feature view and a second reconstructed feature view. The method then utilizes the differences between the first feature view and the first reconstructed feature view, as well as between the second feature view and the second reconstructed feature view, to construct a cross-feature view cross-reconstruction loss. This allows the model to mine and learn shared neural representations common to different feature views during training through cross-dimensional mutual mapping and reconstruction constraints. Furthermore, the method leverages the correlation of multi-view features to effectively suppress specific noise interference in a single feature encoding branch, extracting stable neural features with strong discriminativeness and robustness, and significantly enhancing the brain load interpretation model's ability to capture subtle differences in neural representations between adjacent load levels. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the present invention.

[0022] Figure 2 This is a flowchart illustrating the fine-grained mental workload interpretation method provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the sample expansion and construction process provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the joint architecture of multi-scale sample augmentation and cross-modal cross-reconstruction provided by the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the training device for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the present invention.

[0026] Figure 6 This is a schematic diagram of the fine-grained mental workload interpretation device provided by the present invention.

[0027] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0030] This invention provides a training method for a fine-grained mental workload interpretation model for a brain-controlled robotic arm. The fine-grained mental workload interpretation model is obtained by iteratively executing the following steps until a preset iteration termination condition is met. Specifically, the fine-grained mental workload interpretation model refers to a machine learning model used to identify the fine-grained levels of mental workload generated by an operator during teleoperation tasks. Unlike models that can only perform coarse-grained identification such as high / low or high / medium / low, it can perform more refined classification and interpretation of continuously changing mental workload states. For example, it can divide mental workload into multiple consecutive levels such as level one, level two, level three, level four, and level five, thereby characterizing the subtle differences in neural representations between adjacent workload levels. Fine-grainedness reflects the level of precision in mental workload classification; the total number of levels can be set to three or more according to actual application needs. Those skilled in the art will understand that the more levels there are, the finer the granularity of the interpretation.

[0031] The preset iteration termination condition refers to the criterion for determining whether the fine-grained mental workload interpretation model training has converged. In one implementation, the preset iteration termination condition can be that the number of iterations reaches a preset upper limit; in another implementation, the preset iteration termination condition can be that the target loss value converges, for example, the change in the target loss after multiple consecutive iterations is less than a preset threshold; in yet another implementation, the preset iteration termination condition can also be the model's performance indicators on the validation set, such as the recognition accuracy no longer improving after multiple consecutive rounds, or the total training time reaching a preset time. When any one or more of the above preset iteration termination conditions are met, the iteration stops, and the initial interpretation model corresponding to the model parameters obtained in the current iteration is taken as the final fine-grained mental workload interpretation model. Figure 1 This is a flowchart illustrating the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by this invention. Figure 1 As shown, the method includes the following: Step 110: Determine the initial interpretation model, acquire training sample data and the fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder and a second feature decoder; the training sample data includes the EEG signals of the operator when performing a teleoperation task.

[0032] Specifically, firstly, an initial interpretation model can be determined, and training sample data and corresponding fine-grained mental workload level labels can be obtained. The initial interpretation model refers to the interpretation model whose parameters have not yet been optimized using training sample data before training begins. The initial interpretation model can be a model whose parameters have been randomly or pre-initialized, or it can be a model used as its initial state after pre-training. After iterative training in this embodiment, the initial interpretation model yields the fine-grained mental workload interpretation model.

[0033] The initial interpretation model may include a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder. The first feature encoder and the first feature decoder form a pair, each corresponding to a first feature view; the second feature encoder and the second feature decoder form a pair, each corresponding to a second feature view. The model parameters of the first feature encoder, the second feature encoder, the first feature decoder, and the second feature decoder are independent of each other.

[0034] In one embodiment, each of the above-mentioned feature encoders and feature decoders can be implemented using a multi-layer feedforward neural network composed of stacked fully connected layers and nonlinear activation layers. The feature encoder is used to map the input feature view to a latent feature representation with a dimension lower than the input dimension, and the feature decoder is used to reverse map the latent feature representation back to a reconstructed feature view with the same dimension as the feature view. In another embodiment, each of the above-mentioned feature encoders and feature decoders can also be implemented using other forms of network structures such as convolutional neural networks and graph neural networks. Those skilled in the art can choose according to the data form of the feature view.

[0035] Here, the training sample data includes EEG signals from operators performing teleoperation tasks. The operator refers to the entity performing the teleoperation task, such as the operator of a space station robotic arm or ground-based teleoperation control personnel. A teleoperation task refers to a task in which an operator remotely controls a controlled object through an input device, such as remotely controlling a controlled object like a space robotic arm through a brain-computer interface (BCI) input device or other human-computer interaction (HCI) devices.

[0036] Among them, electroencephalogram (EEG) signals refer to the neurophysiological activity signals generated by the brain during the operation of a teleoperation task, such as scalp EEG signals. In one embodiment, to provide richer spatial sampling information to characterize brain region activity related to changes in mental workload, a high-density scalp EEG acquisition device can be used to acquire EEG signals. The high-density scalp EEG acquisition device includes multiple EEG acquisition electrodes arranged according to preset scalp spatial positions, thereby obtaining EEG signals with a high spatial sampling density.

[0037] In one implementation, an EEG acquisition device is used to collect EEG signals from an operator during a teleoperation task. The collected EEG signals are then preprocessed to obtain training sample data. The preprocessing may include rereference, bandpass filtering, power line interference suppression, artifact removal, downsampling, and segmentation based on a sliding time window. Specifically, the multi-channel EEG signals are first rereferenced, for example, by using the average value of bilateral mastoid electrodes as a reference signal to reduce the influence of the common reference signal on the EEG signals of different electrodes. Then, the rereferenced EEG signals are bandpass filtered, for example, using a bandpass filtering range of 1 Hz to 45 Hz, to reduce the influence of low-frequency drift and high-frequency noise. Further, power frequency interference suppression is applied to the EEG signals to reduce power frequency interference in the experimental environment. Next, independent component analysis is used to perform blind source separation on the multi-channel EEG signals, identifying and removing blinking, eye movement, and other non-neural artifacts based on the temporal, spectral, and spatial distribution characteristics of the independent components. After artifact removal, the EEG data is downsampled to a preset sampling rate, such as 250 Hz, to reduce the computational load in subsequent feature calculations and decoding. Finally, according to the timeline of the teleoperation task, a sliding time window is used to segment the preprocessed continuous EEG signals to obtain multiple EEG analysis samples, which are then used as training sample data. In addition, to establish a temporal correspondence between EEG signals and the teleoperation task process, the EEG signals, the task status information of the teleoperation task, and the operation behavior data can be synchronized in time using a unified timestamp while the EEG signals are being collected.

[0038] The fine-grained mental workload level label reflects the actual fine-grained mental workload level of the operators corresponding to the training sample data, and serves as supervisory information for model training. In one implementation, the training sample data can be labeled with a fine-grained mental workload level based on one or more of the following: the task difficulty level, subjective mental workload evaluation results (such as NASA-TLX scale evaluation results), task completion time, operational error, task success rate, and operational behavior data, thereby obtaining the fine-grained mental workload level label.

[0039] Step 120: Extract features from the EEG signal in at least two dimensions to obtain a first feature view and a second feature view.

[0040] Specifically, features are extracted from the EEG signal in at least two dimensions to obtain a first feature view and a second feature view.

[0041] Considering that different EEG features can describe neural activity from different perspectives, and that single-dimensional EEG features are easily affected by individual differences, noise, and feature redundancy, this embodiment performs feature extraction on EEG signals in at least two dimensions to obtain a first feature view and a second feature view with different information attributes.

[0042] Here, a feature view refers to the feature representation of an EEG signal in a specific dimension. Feature views of different dimensions depict the mental workload-related information contained in the EEG signal from different perspectives and are complementary to each other. "At least two dimensions" means that features are extracted from the EEG signal from two or more different feature dimensions, such as selecting at least two dimensions from frequency domain energy distribution, statistical distribution, spatial distribution, etc.

[0043] In one implementation, the extracted features include at least two of the following: power spectral density features, differential entropy features, and spatial topological features, thereby forming two corresponding feature views.

[0044] The power spectral density feature is used to reflect the energy distribution information of the EEG signal within different frequency ranges. When extracting the power spectral density feature, for each EEG channel, such as the k-th EEG channel, the preprocessed EEG signal undergoes a frequency domain transformation to calculate the power spectral density within the corresponding frequency range, and features are aggregated according to a preset frequency band. The preset frequency band may include one or more of the δ, θ, α, β, and γ bands, and further, several frequency bands can be selected to form the power spectral density feature vector based on the requirements of mental workload recognition.

[0045] Among them, differential entropy features are used to reflect the statistical distribution information of EEG signals within a given time window. When extracting differential entropy features, for each EEG channel, the differential entropy of the EEG signal is calculated within a preset frequency band, and the differential entropies corresponding to different channels and different frequency bands are combined to form a differential entropy feature vector. Under the condition that the EEG signal approximately follows a Gaussian distribution, the differential entropy... It can be represented as: ; in, This represents the variance of the corresponding EEG signal.

[0046] Spatial topological features are used to reflect the distribution of neural activity on the scalp in relation to the EEG electrodes. When extracting spatial topological features, the EEG features of the corresponding electrodes are arranged and mapped according to their spatial positions on the scalp, based on the spatial locations of multiple EEG electrodes, thus forming an EEG feature representation with spatial structural information.

[0047] This allows for the acquisition of at least two of the following: a power spectral density feature view, a differential entropy feature view, and a spatial topological feature view. The first feature view and the second feature view are any two of the aforementioned at least two feature views. For example, in one embodiment, the first feature view is a power spectral density feature view, and the second feature view is a differential entropy feature view; in another embodiment, the first feature view is a power spectral density feature view, and the second feature view is a spatial topological feature view. For ease of explanation, this embodiment uses a power spectral density feature view as the first feature view and a differential entropy feature view as the second feature view, and the first feature view is denoted as... Let the second feature view be denoted as .

[0048] Furthermore, in order to make different feature views have relatively consistent numerical scales, feature standardization processing can be performed on the first feature view and the second feature view respectively, and the standardized first feature view and the second feature view can be used as input to the initial interpretation model.

[0049] Step 130: Input the first feature view and the second feature view into the initial interpretation model respectively. The first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation.

[0050] Specifically, after obtaining the first feature view and the second feature view, the first feature view and the second feature view can be input into the initial interpretation model respectively. The first feature encoder encodes the first feature view to obtain the first latent feature representation; the second feature encoder encodes the second feature view to obtain the second latent feature representation.

[0051] Here, latent feature representation refers to the low-dimensional feature representation in the latent space (latent space) after the feature view is encoded by the feature encoder. Latent feature representation has a higher level of abstraction and a more compact data form compared to the original feature view. The above encoding process can be represented as: ; ; in, This represents the first feature encoder, i.e., the power spectral density feature encoder; This represents the second feature encoder, namely the differential entropy feature encoder; This represents the first latent feature representation; This represents the second latent feature representation.

[0052] Step 140: Reconstruct the first latent feature representation based on the second feature decoder to obtain a second reconstructed feature view; reconstruct the second latent feature representation based on the first feature decoder to obtain a first reconstructed feature view.

[0053] Specifically, to enable latent feature representations to not only retain information from a single feature view but also learn stable neural information shared across different feature views, this embodiment employs a cross-feature view reconstruction approach to establish mutual constraints between different feature views. Cross-reconstruction refers to using the latent feature representation corresponding to one feature view to reconstruct another feature view, rather than reconstructing its own corresponding feature view.

[0054] Here, the first latent feature representation can be reconstructed based on the second feature decoder to obtain the second reconstructed feature view. That is, the first latent feature representation is input into the second feature decoder, and the second feature view is reconstructed to obtain: ; The second latent feature representation is reconstructed based on the first feature decoder to obtain the first reconstructed feature view. Specifically, the second latent feature representation is input into the first feature decoder to reconstruct the first feature view, resulting in: ; in, This represents the first feature decoder, i.e., the power spectral density feature decoder; This represents the second feature decoder, namely the differential entropy feature decoder; This represents the second reconstructed feature view obtained by reconstructing based on the first latent feature representation; This represents the first reconstructed feature view obtained by reconstructing based on the second potential feature representation.

[0055] Therefore, the first feature view and the second feature view no longer participate in subsequent processing only through their respective independent encoding paths, but establish mutual constraints through cross-feature view reconstruction, so that the components that can effectively reconstruct the information of the other feature view are retained in the latent feature representation.

[0056] Step 150: Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, determine the cross-feature view cross-reconstruction loss, determine the target loss based on the cross-feature view cross-reconstruction loss, and update the model parameters of the initial interpretation model based on the target loss.

[0057] Specifically, after obtaining the first reconstructed feature view and the second reconstructed feature view, the cross-feature view cross-reconstruction loss can be determined based on the differences between the first feature view and the first reconstructed feature view, as well as the differences between the second feature view and the second reconstructed feature view.

[0058] Here, the cross-feature view cross-reconstruction loss is used to reflect the degree of error in the cross-reconstruction between different feature views. The smaller the value of the cross-feature view cross-reconstruction loss, the better the latent feature representation corresponding to each feature view can explain the other feature view. In one implementation, the difference between the first feature view and the first reconstructed feature view, and the difference between the second feature view and the second reconstructed feature view can be calculated separately, for example, using mean squared error to measure the above differences: ; ; Furthermore, the cross-feature view cross-reconstruction loss can be expressed as the sum of the two differences mentioned above: ; in, Indicates the cross-feature view cross-reconstruction loss; This represents the second reconstructed feature view obtained by reconstructing based on the first latent feature representation; This represents the first reconstructed feature view obtained by reconstructing based on the second latent feature representation; Indicates the first feature view. Indicates the second feature view; This indicates the difference between the first feature view and the first reconstructed feature view; This indicates the difference between the second feature view and the second reconstructed feature view.

[0059] It should be noted that the method of measuring the above differences is not limited to mean square error. Other difference measurement methods such as mean absolute error (L1 norm) and cosine distance can also be used. This embodiment of the invention does not make specific limitations on this.

[0060] Furthermore, the target loss can be determined based on the cross-feature view reconstruction loss. The target loss reflects the overall optimization objective of the initial interpretation model in the current iteration and serves as the basis for updating the model parameters.

[0061] In one implementation, the backpropagation algorithm can be used to calculate the gradient of the target loss relative to each trainable parameter in the initial interpretation model, such as the trainable parameters of the first feature encoder, the second feature encoder, the first feature decoder, and the second feature decoder. Then, a gradient descent optimizer, such as stochastic gradient descent, Adam, etc., is used to update the trainable parameters based on the gradient, thereby gradually reducing the target loss.

[0062] The above steps are executed iteratively, that is, samples are continuously obtained from the training sample data and the above process of feature extraction, encoding, cross reconstruction, loss calculation and parameter update is repeated. Each iteration improves the performance of the initial interpretation model slightly until the preset iteration termination condition is met, that is, the fine-grained mental workload interpretation model is obtained after training.

[0063] The method provided in this invention, based on a first feature view and a second feature view extracted from EEG signals, performs cross-reconstruction of the latent feature representation of another dimension through a first and a second feature decoder to obtain a first reconstructed feature view and a second reconstructed feature view. It then utilizes the differences between the first feature view and the first reconstructed feature view, as well as between the second feature view and the second reconstructed feature view, to construct a cross-feature view cross-reconstruction loss. On the one hand, through cross-dimensional mutual mapping and reconstruction constraints, it forces the model to mine and learn shared neural representations commonly contained between different feature views, thereby significantly enhancing the model's ability to capture subtle differences in neural representations between adjacent workload levels, achieving the ability to capture the continuous changes in mental workload as task difficulty increases. The system accurately depicts subtle changes, effectively solving the problem of limited granularity and difficulty in accurately depicting continuous changes in existing technologies. On the other hand, the cross-feature view cross-reconstruction loss guides the model to continuously optimize during training, ensuring that the reconstruction result of any view depends on the effective information of the other view. This effectively suppresses specific noise interference in single feature coding branches, extracts stable neural features with strong discriminativeness and robustness, overcomes the technical defects of poor decoding robustness caused by individual differences in EEG signals and limitations of single feature view modeling, and significantly improves the accuracy and stability of fine-grained mental workload recognition. It can support high-precision adaptive interaction and early warning, meeting the actual needs of teleoperation tasks such as space robotic arms.

[0064] Based on the above embodiments, step 150, which involves determining the target loss based on the cross-feature view cross-reconstruction loss, includes: Step 1501: Fuse the first latent feature representation and the second latent feature representation to obtain a fused neural representation; Step 1502: Input the fused neural representation into the classifier in the initial interpretation model to obtain the predicted load level result output by the classifier; Step 1503: Based on the fine-grained mental workload level labels and the predicted workload level results, determine the classification loss; Step 1504: Determine the target loss based on the cross-feature view cross-reconstruction loss and the classification loss.

[0065] Specifically, firstly, the first latent feature representation and the second latent feature representation can be fused to obtain a fused neural representation. This fused neural representation reflects a unified and more complete neural representation obtained by integrating the latent feature representations corresponding to different feature views. Compared to the latent feature representation corresponding to any single feature view, the fused neural representation simultaneously contains information related to mental workload from multiple feature views.

[0066] In one implementation, the first latent feature representation and the second latent feature representation can be fused using a feature concatenation method, namely: ; in, Indicates fusion neural representation; This represents the first latent feature representation; This represents the second latent feature representation; This indicates a feature concatenation operation. It should be noted that the method for fusing the first and second latent feature representations is not limited to feature concatenation; it can also be achieved using weighted summation, element-wise addition, or other fusion networks. This embodiment of the invention does not specifically limit this method.

[0067] The fused neural representation is input into the classifier in the initial interpretation model to obtain the predicted mental workload level output by the classifier. Here, the classifier refers to the network module in the initial interpretation model used to classify and predict the operator's mental workload level based on the fused neural representation.

[0068] Here, the predicted workload level refers to the fine-grained mental workload level currently assigned to the operator, predicted by the classifier based on the fused neural representation. The predicted workload level can be the predicted probability corresponding to each fine-grained mental workload level, or it can be a specific level determined based on the predicted probability. In one implementation, the classifier includes a fully connected layer, a nonlinear activation layer, and a Softmax classification layer. Specifically, firstly, the fused neural representation is feature-mapped through a fully connected layer; then, a nonlinear activation function is used to enhance the model's ability to express nonlinear differences between different mental workload levels; finally, the Softmax classification layer outputs the predicted probability corresponding to different mental workload levels. Let the total number of mental workload levels be C, then the classification probability is expressed as: ; in, The output of the classifier represents the first... The category score corresponding to each level of mental workload; This indicates that the operator belongs to the first Predicted probability of each level of mental workload; This represents the total number of mental workload levels. Furthermore, the current mental workload level can be determined based on the category with the highest predicted probability. ; in, This indicates the predicted level of mental workload.

[0069] Furthermore, the classification loss can be determined based on the fine-grained mental workload level labels and the predicted workload level results. The classification loss reflects the degree of deviation between the predicted workload level results and the fine-grained mental workload level labels; the smaller the classification loss value, the closer the classifier's prediction result is to the true label. In one implementation, the cross-entropy loss function can be used to determine the classification loss: ; in, Indicates classification loss; The labels represent the true, fine-grained levels of mental workload, for example, using one-hot encoding, with a value of 1 at the position corresponding to the true level and a value of 0 at the other positions; This indicates that the model predicts the operator belongs to the first... The probability of each level of mental workload; This indicates the total number of mental workload levels.

[0070] Finally, the target loss can be determined based on the cross-feature view cross-reconstruction loss and the classification loss. In one implementation, the target loss can be obtained by weighted summing of the cross-feature view cross-reconstruction loss and the classification loss.

[0071] The method provided in this invention further introduces classification supervision on the basis of cross-feature view cross-reconstruction constraints, so that the fine-grained mental workload interpretation model can learn the shared neural representations between different feature views and have the ability to distinguish the fine-grained mental workload level, thereby improving the representation robustness and the accuracy of fine-grained mental workload interpretation.

[0072] Based on the above embodiments, the initial interpretation model further includes a third feature encoder; Step 1501 includes: Step 1501-1: Obtain the spatial distribution information of the electrodes corresponding to the EEG signal, and construct spatial topological features based on the spatial distribution information of the electrodes; Step 1501-2: Encode the spatial topological features based on the third feature encoder to obtain the third latent feature representation; Step 1501-3: Fuse the first latent feature representation, the second latent feature representation, and the third latent feature representation to obtain the fused neural representation.

[0073] Specifically, the initial interpretation model also includes a third feature encoder. The third feature encoder is used to encode the feature view in the third dimension. Its model parameters are independent of the first and second feature encoders. Its network structure can be the same as or different from the first and second feature encoders. For example, it can also use a multi-layer feedforward neural network composed of fully connected layers and non-linear activation layers, or a convolutional neural network or graph neural network suitable for processing spatial structure information.

[0074] First, the spatial distribution information of the electrodes corresponding to the EEG signals can be obtained, and spatial topological features can be constructed based on the electrode spatial distribution information. The electrode spatial distribution information reflects the distribution relationship of each EEG acquisition electrode on the scalp, such as the spatial coordinates of each EEG acquisition electrode in the scalp coordinate system or the relative positional relationship between each EEG acquisition electrode.

[0075] Here, spatial topological features are used to reflect the distribution of neural activity of EEG acquisition electrodes in the spatial location of the scalp. In one embodiment, the spatial topological features are constructed by arranging and mapping the EEG features of the corresponding electrodes according to their spatial locations on the scalp, based on the electrode spatial distribution information of multiple EEG acquisition electrodes, thereby forming an EEG feature representation with spatial structural information.

[0076] Then, the spatial topological features can be encoded based on the third feature encoder to obtain the third latent feature representation. The third latent feature representation refers to the low-dimensional feature representation of the spatial topological features in the latent space after encoding by the third feature encoder.

[0077] Finally, the first, second, and third latent feature representations can be fused to obtain a fused neural representation. In one implementation, feature splicing can be used to fuse the three: ; in, Indicates fusion neural representation; This represents the first latent feature representation; This represents the second latent feature representation; This represents the third latent feature representation, that is, the latent representation corresponding to the spatial topological features.

[0078] It should be noted that when introducing spatial topological feature views, the cross-reconstruction relationship between spatial topological features and the first and second feature views can be established in the same way as the aforementioned cross-feature view cross-reconstruction, so that the three feature views form mutual constraints. Spatial topological features can participate in cross-reconstruction as independent feature views, or they can participate in shared representation learning after being fused with the first or second feature view.

[0079] The method provided in this invention introduces spatial topological features that reflect the spatial distribution relationship of electrodes, in addition to power spectral density features and differential entropy features. This allows the fused neural representation to simultaneously include neural information from multiple aspects, such as frequency domain energy distribution, statistical distribution, and spatial topological distribution. This makes fuller use of the spatial information provided by high spatial sampling density EEG signals and further improves the accuracy and stability of fine-grained mental workload interpretation.

[0080] Based on the above embodiments, step 1504 includes: Step 1504-1: Obtain the weight parameters of each network layer in the initial interpretation model, and determine the sparse constraint loss based on the sum of the absolute values ​​of each weight parameter; Step 1504-2: Determine the target loss based on the sparse constraint loss, the cross-feature view cross-reconstruction loss, and the classification loss.

[0081] Specifically, firstly, the weight parameters of each network layer in the initial interpretation model can be obtained, and the sparse constraint loss can be determined based on the sum of the absolute values ​​of each weight parameter.

[0082] The weight parameters of each network layer refer to the trainable weights of each network layer in the initial interpretation model. The weight parameters of each network layer can be the weight parameters of at least some of the network layers, including the first feature encoder, the second feature encoder, the first feature decoder, the second feature decoder, and the classifier.

[0083] Here, the sparse constraint loss reflects the sparsity of the weight parameters of each network layer under which sparse constraints are applied. A smaller sparse constraint loss indicates a weaker dependence of the fine-grained mental workload interpretation model on a large number of redundant features and random noise features. In one implementation, L1 regularization constraints are applied to at least some parameters in the feature encoder, feature decoder, and classifier. The sparse constraint loss is expressed as: ; in, This represents the sparse constraint loss; This represents the weight parameter matrix of the network layer participating in the sparse constraint; Represents the set of parameters participating in the sparse constraint; Represents the weight parameter matrix The sum of the absolute values ​​of all elements in the equation.

[0084] Finally, the target loss can be determined based on the sparse constraint loss, the cross-feature view cross-reconstruction loss, and the classification loss. In one implementation, the classification loss, the cross-feature view cross-reconstruction loss, and the sparse constraint loss are jointly optimized to construct the overall training objective of the model: ; in, Indicates target loss; Indicates classification loss; Indicates the cross-feature view cross-reconstruction loss; This represents the sparse constraint loss; The weight coefficients represent the cross-feature view cross-reconstruction loss. The weight coefficients represent the sparse constraint loss. By jointly optimizing the above objective loss, the model can achieve fine-grained mental workload discrimination ability while enabling different feature views to form mutual constraints in the latent space, and reducing the impact of redundant features and specific noise on the fine-grained mental workload interpretation model.

[0085] The method provided in this invention utilizes sparse constraints to limit the dependence of fine-grained mental workload interpretation models on a large number of redundant features and random noise features, making the neural representations shared across feature views more focused on stable neural information related to changes in mental workload. This suppresses redundant information in high-dimensional EEG features and specific noise in each feature view, thereby improving the compactness and discriminativeness of shared neural representations.

[0086] Based on the above embodiments, obtaining training sample data in step 110 includes: Step 1101: Obtain raw EEG signal fragments; Step 1102: The original EEG signal segments are segmented using sliding windows of different lengths to obtain multiple sub-segments corresponding to each time scale; Step 1103: Extract the EEG features corresponding to each of the sub-segments, and aggregate the EEG features at the same time scale to obtain representative feature samples for each time scale. Step 1104: Based on representative feature samples of each time scale, the original EEG signal segments are expanded to obtain the training sample data.

[0087] Specifically, firstly, raw EEG signal segments can be acquired. These raw EEG signal segments refer to EEG signal segments of a certain duration obtained from the operator's teleoperation task and after preprocessing. In one embodiment, the duration of a raw EEG signal segment is 10 seconds.

[0088] Secondly, the original EEG signal segments were segmented using sliding windows of different lengths to obtain multiple sub-segments corresponding to each time scale. Here, the sliding window refers to the time window used to segment the original EEG signal segments along the time axis, with different lengths of sliding windows corresponding to different time scales.

[0089] Here, the time scale reflects the length of the sliding window, with different time scales corresponding to different time resolutions. The multiple sub-segments corresponding to each time scale refer to the multiple EEG signal segments obtained after segmenting the original EEG signal fragment using a sliding window of a certain length. In one implementation, 10 time scales can be set, each corresponding to a different time window length; for the t-th time scale, the original EEG signal fragment is divided into... Each sub-segment is denoted as... .

[0090] Furthermore, the EEG features corresponding to each sub-segment signal can be extracted separately, and the EEG features at the same time scale can be aggregated to obtain representative feature samples at each time scale. Here, the EEG features corresponding to each sub-segment signal refer to the EEG feature vectors extracted from each sub-segment signal. The EEG features corresponding to each sub-segment signal can be one or more of the following: power spectral density features, differential entropy features, and spatial topological features.

[0091] Here, a representative feature sample refers to a feature representation that aggregates the EEG features of multiple sub-segments of signals at a certain time scale and can represent the overall information of that time scale. In one implementation, the EEG features corresponding to multiple sub-segments of signals at the same time scale are aggregated by mean. For the t-th time scale, its representative feature sample... The calculation is as follows: ; in, Indicates the first Representative feature samples corresponding to each time scale; Indicates the first The first time scale The EEG feature vectors corresponding to each sub-segment signal; Indicates the first The number of sub-segments of signal at each time scale. It should be noted that the method of aggregating EEG features at the same time scale is not limited to mean aggregation; other statistical aggregation methods such as weighted average and maximum value aggregation can also be used.

[0092] Finally, the original EEG signal segments can be augmented based on representative feature samples from different time scales to obtain training sample data. In one implementation, representative feature samples obtained at different time scales are... These samples serve as independent feature samples during the model training phase, enabling a single original EEG signal segment to generate multiple feature representations with different temporal resolutions across multiple time scales. This increases the number of effective training samples without requiring additional EEG data collection. Furthermore, during model testing and online prediction, to reduce the impact of feature fluctuations at different time scales on prediction results, representative feature samples from multiple time scales corresponding to the same original EEG signal segment can be aggregated to form a stable EEG feature representation for final decoding.

[0093] The method provided in this invention utilizes multi-scale time windows to perform sliding window segmentation, feature extraction, and statistical aggregation on raw EEG signal segments, thereby expanding the number of training samples without increasing additional experimental acquisition costs, and thus improving the stability of model training under limited sample conditions.

[0094] Based on the above embodiments, the teleoperation task includes a continuous motion control task for a space robotic arm; the continuous motion control task for a space robotic arm is an operation task in which the operator controls the space robotic arm to complete continuous motion control through electroencephalogram (EEG) signals, and the continuous motion control task for a space robotic arm is used to induce fine-grained mental load on the operator. The steps for determining the fine-grained mental workload level label include: Step 21: Determine the difficulty level of each basic task based on at least one of the following: spatial structure complexity of the target object, visual perception conditions, number of operation monitoring parameters, and continuous motion control constraints. Step 22: Collect the EEG signals of the operator when performing the continuous motion control task of the space robotic arm at each of the basic task difficulty levels; Step 23: Obtain the results of the subjective evaluation scale filled out by the operators for each of the basic task difficulty levels; Step 24: Based on the basic task difficulty level and the subjective evaluation scale results, perform fine-grained mental workload level labeling on the task EEG signals to obtain the fine-grained mental workload level label.

[0095] Specifically, teleoperation tasks include continuous motion control tasks for space robotic arms. These tasks involve operators using electroencephalogram (EEG) signals to control the space robotic arm and perform continuous motion control maneuvers, thus inducing fine-grained mental workload for the operator.

[0096] In one implementation, a seven-degree-of-freedom robotic arm model can be used to construct a space robotic arm teleoperation simulation environment. This simulates the continuous motion control tasks performed by the space station robotic arm, including target search, target approach, motion direction adjustment, attitude control, and target docking. Operators remotely control the brain-controlled robotic arm via a brain-computer interface, enabling the end effector to approach and dock with the target object according to preset task requirements. During task execution, operators need to continuously acquire visual information about the robotic arm and the target object, determine the spatial position, attitude, and structural features of the target object, and continuously adjust the robotic arm's motion direction, speed, and end effector attitude based on the current motion state. Unlike tasks involving only single operations or discrete decisions, continuous motion control tasks for space robotic arms are characterized by continuous perception, continuous decision-making, and continuous motion control. Operators need to continuously monitor multiple state variables throughout the task and constantly correct their operational behavior based on the relative state between the robotic arm and the target object, thus creating a mentally demanding state with continuously changing characteristics.

[0097] The steps for determining fine-grained mental workload level labels here include the following: First, the difficulty level of each basic task can be determined based on at least one of the following: the spatial structural complexity of the target object, visual perception conditions, the number of operational monitoring parameters, and continuous motion control constraints. Among these, the spatial structural complexity of the target object, visual perception conditions, the number of operational monitoring parameters, and continuous motion control constraints are the task difficulty factors for jointly adjusting the continuous motion control task of the space robotic arm.

[0098] Here, the basic task difficulty level refers to the task difficulty level with different cognitive processing requirements formed by combining at least one of the above-mentioned task difficulty factors according to a preset combination method. In one embodiment, the above-mentioned task difficulty factors are combined according to a preset combination method to form multiple basic task difficulty levels. By changing the combination method of different task difficulty factors, different degrees of cognitive processing requirements are formed between adjacent basic task difficulty levels, thereby inducing multiple fine-grained mental load states.

[0099] Secondly, it can collect task-related EEG signals from operators performing continuous motion control tasks of the spatial robotic arm at various basic task difficulty levels. Specifically, task-related EEG signals refer to the EEG signals collected from operators during the execution of continuous motion control tasks of the spatial robotic arm at a specific basic task difficulty level.

[0100] In one implementation, a high-density scalp EEG acquisition device is used to collect whole-brain EEG signals during the operation of a spatial robotic arm's continuous motion control task. The high-density scalp EEG acquisition device includes multiple EEG acquisition electrodes arranged according to preset scalp spatial positions to obtain EEG signals with high spatial sampling density. Simultaneously, it can also record task status and behavioral data during the robotic arm's continuous motion control process, including one or more of the following: robotic arm joint position, joint velocity, robotic arm end effector position, robotic arm end effector posture, target object position, target object posture, robotic arm motion trajectory, task completion time, operational error, and task success rate. A unified timestamp is used to synchronize the EEG signals, task status information, and operational behavior data, thereby establishing a temporal correspondence between the EEG signals and the continuous motion control task process.

[0101] Furthermore, the results of subjective evaluation scales completed by operators for each basic task difficulty level can be obtained. These subjective evaluation scale results refer to the operators' subjective assessment of their perceived mental workload at a specific basic task difficulty level. In one implementation, the subjective evaluation scale can be the NASA-TLX scale. After completing a task at a specific basic task difficulty level, operators complete the NASA-TLX scale to subjectively evaluate the mental workload of the current task phase.

[0102] Finally, based on the basic task difficulty level and the results of the subjective evaluation scale, the task EEG signals can be labeled with fine-grained mental workload levels to obtain fine-grained mental workload level labels. In one embodiment, the EEG analysis samples corresponding to the task EEG signals can be labeled with fine-grained mental workload levels based on one or more of the following information: basic task difficulty level, subjective evaluation scale results, task completion time, operational error, task success rate, and robotic arm control behavior data.

[0103] In a more specific implementation, two stages can be set up: a difficulty perception calibration stage and a formal mental load induction stage, denoted as block-1 and block-2, respectively. In block-1, multiple preset task difficulties are presented sequentially in ascending order of difficulty, with each task difficulty lasting for a preset time. After the task is completed, the operator fills out a NASA-TLX scale. Based on the operator's NASA-TLX evaluation results at different task difficulties, the task difficulty is individually calibrated. When the difference in subjective mental load evaluation between adjacent task difficulty levels is lower than a preset threshold, at least one task difficulty factor among the target object spatial structure complexity, visual perception conditions, number of operational monitoring parameters, and continuous motion control constraints is adjusted to create a more obvious difference in mental load between adjacent task difficulty levels. After completing block-1, multiple individualized task difficulty levels with significant load differentiation are selected based on the operator's subjective mental load evaluation results, and the corresponding task difficulty levels are used as fine-grained mental load induction conditions for the formal experiment. In block-2, the selected task difficulty levels are presented in a random order. Each difficulty level task is executed for a preset time. After the task is completed, the operator fills out a NASA-TLX questionnaire and then proceeds to the next difficulty level task after a preset rest period.

[0104] In one implementation, each task difficulty level in block-1 is executed for 3 minutes, followed by a 1-minute assessment and a 1-minute rest. In block-2, each task difficulty level is executed for 5 minutes, followed by a 1-minute assessment and a 1-minute rest. When the operator completes the current target docking task ahead of schedule within the specified time, the current task is regenerated or restarted, allowing the operator to continuously perform the continuous motor control operations corresponding to the current task difficulty level until the preset task time ends, ensuring that the EEG data has a relatively consistent acquisition length under different task difficulty conditions.

[0105] The method provided in this invention uses the continuous motion control task of a spatial robotic arm as a carrier for inducing mental workload. It forms multiple basic task difficulty levels through the joint adjustment of multiple task difficulty factors, and combines the basic task difficulty levels with the results of subjective evaluation scales to perform fine-grained mental workload level labeling on the task EEG signals. This achieves stable induction and accurate labeling of fine-grained mental workload, providing high-quality supervision information for the training of fine-grained mental workload interpretation models.

[0106] Based on the above embodiments, the spatial structural complexity of the target object is determined based on the geometric shape of the target object to be operated in the continuous motion control task of the spatial robotic arm; the geometric shape includes at least one of a sphere, cylinder, triangular prism, cross prism and pentagon; The visual perception conditions are determined based on the number of cameras deployed on the space robotic arm and the visual information dimension; the visual information dimension characterizes the richness of information about the target state of the target object and the state of the space robotic arm simultaneously acquired by the operator. The number of operation monitoring parameters is determined based on the number of robotic arm state parameters that the operator simultaneously monitors when performing the continuous motion control task of the space robotic arm; the robotic arm state parameters include at least one of the following: joint position, joint speed, robotic arm posture, end effector position, end effector pose, position of the target object, and relative distance between the target object and the space robotic arm; The continuous motion control constraints are based on at least one of the following in the continuous motion control task of the space manipulator: the allowable operational error of the task, the task completion time, the target docking accuracy between the space manipulator and the target object, the motion trajectory complexity of the space manipulator, and the motion range.

[0107] Specifically, the spatial structural complexity of the target object is determined based on the geometric shape of the target object to be manipulated in the continuous motion control task of the space robotic arm. The spatial structural complexity of the target object reflects the degree of complexity in its geometric shape.

[0108] The geometric shape includes at least one of a sphere, cylinder, triangular prism, cross, and pentagon. In one embodiment, the spatial structure complexity of the target object is adjusted by changing its geometric shape and spatial structure. As the spatial structure complexity of the target object increases, the difficulty for the operator to judge the target object's posture, spatial boundaries, and the trajectory of the robotic arm's end effector increases.

[0109] Here, the visual perception conditions are determined based on the number of cameras deployed on the space robotic arm and the dimensions of the visual information. Specifically, the visual perception conditions reflect the complexity of the visual information that the operator can acquire during task execution.

[0110] Among them, the number of cameras refers to the number of cameras deployed on the space robotic arm; the visual information dimension represents the richness of information obtained by the operator at the same time as the target state of the target object and the state of the space robotic arm. The target state of the target object refers to the spatial position, posture, structure and other state information of the target object, while the state of the space robotic arm refers to the motion state, posture and other state information of the space robotic arm.

[0111] In one implementation, the visual perception conditions are adjusted by changing the number of cameras, the camera angles, and the target state information that the operator can simultaneously acquire: when the visual perception conditions are simple, the operator only needs to complete the target positioning and robotic arm control based on a single or small amount of perspective information; when the visual perception conditions are complex, the operator needs to process visual information from multiple perspectives simultaneously and integrate the different visual information.

[0112] Here, the number of operational monitoring parameters is determined based on the number of robotic arm status parameters that the operator simultaneously monitors while performing continuous motion control tasks on the space robotic arm. Specifically, the number of operational monitoring parameters refers to the number of robotic arm status parameters that the operator needs to monitor simultaneously while performing the task.

[0113] The robotic arm state parameters include at least one of the following: joint position, joint velocity, robotic arm posture, end effector position, end effector pose, target object position, and relative distance between the target object and the robotic arm.

[0114] Joint position refers to the spatial coordinates or angle values ​​of each rotational or translational joint of the space robot arm at the current moment, used to characterize the configuration state of the robot arm. Joint velocity refers to the rate of change of position of each joint of the space robot arm per unit time, reflecting the speed and dynamic characteristics of joint movement.

[0115] Here, the robotic arm posture refers to the orientation and tilt of the entire robotic arm or its end effector in three-dimensional space, typically characterized by Euler angles, quaternions, or rotation matrices. The end effector position refers to the coordinate position in three-dimensional space of the functional component at the very end of the robotic arm used to perform operations, such as a gripper or docking mechanism.

[0116] The end effector pose refers to the six degrees of freedom state information of the end effector, encompassing both position and orientation, namely its coordinates (x, y, z) in three-dimensional space and its rotation angles around the three axes. The target object position refers to the three-dimensional spatial coordinates of the target object to be manipulated, such as the experimental payload to be grasped, in the task coordinate system or the world coordinate system.

[0117] Here, the relative distance between the target object and the space robot arm refers to the Euclidean distance between a specific reference point on the target object, such as the docking center or center of mass, and the end effector or specific joint of the robot arm, which is used to characterize the degree of spatial proximity and operational margin between the two.

[0118] In one implementation, the number of operational monitoring parameters is adjusted by changing the number of robotic arm status parameters that the operator needs to monitor simultaneously. The more robotic arm status parameters the operator needs to monitor simultaneously, the greater the number of operational monitoring parameters and the higher the corresponding cognitive processing requirements.

[0119] Among them, the continuous motion control constraints are determined based on at least one of the following in the continuous motion control task of the space manipulator: the allowable operation error of the task, the task completion time, the target docking accuracy between the space manipulator and the target object, the motion trajectory complexity of the space manipulator, and the motion range.

[0120] Here, continuous motion control constraints are used to reflect the strictness of the constraints imposed on the operation process by the continuous motion control task of the space robot. Among them, the allowable operation error refers to the range of operation deviations allowed by the task, the task completion time refers to the required completion time of the task, the target docking accuracy refers to the required accuracy of docking between the space robot and the target object, the motion trajectory complexity refers to the complexity of the motion trajectory of the space robot, and the motion range refers to the spatial range covered by the motion of the space robot.

[0121] In one implementation, continuous motion control constraints can be adjusted by adjusting one or more of the following: allowable operational error of the task, target docking accuracy, task completion time, complexity of the robotic arm's motion trajectory, posture adjustment requirements, and the robotic arm's range of motion.

[0122] The method provided in this invention refines and adjusts the difficulty of continuous motion control tasks of a space robotic arm in multiple dimensions, including the geometric shape of the target object, visual perception, state parameter monitoring, and motion control constraints. This creates different levels of cognitive processing requirements between adjacent task difficulty levels, thereby achieving stable induction of multi-level fine-grained mental workload.

[0123] Considering that after training to obtain a fine-grained mental workload interpretation model, it can be applied to perform fine-grained mental workload interpretation.

[0124] Based on any of the above embodiments, the present invention provides a fine-grained method for interpreting mental workload. Figure 2 This is a flowchart illustrating the fine-grained mental workload interpretation method provided by the present invention, as shown below. Figure 2 As shown, the method includes: Step 210: Acquire real-time EEG signals when the user performs a teleoperation task; Step 220: Input the real-time EEG signal into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model; The fine-grained mental workload interpretation model is obtained by executing the training method of the fine-grained mental workload interpretation model of the brain-controlled robotic arm.

[0125] Specifically, firstly, real-time EEG signals can be acquired when the user performs a teleoperation task. Real-time EEG signals refer to EEG signals collected in real time during the actual execution of the teleoperation task.

[0126] The user here refers to the object that actually performs the teleoperation task and whose mental workload needs to be interpreted, such as the operator of a space robotic arm.

[0127] In one implementation, during online operation, the EEG acquisition device continuously acquires scalp EEG signals when the user performs continuous motion control tasks of the spatial robotic arm, and sends the EEG data to the signal processing module in real time through the data transmission interface. The signal processing module performs rereference, bandpass filtering, power frequency interference suppression, artifact removal and downsampling processing on the EEG signals in the same manner as in the training phase, and uses a sliding time window to extract the current EEG analysis sample from the continuous EEG signals as the real-time EEG signal.

[0128] Secondly, real-time EEG signals can be input into a fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the model.

[0129] Here, the fine-grained mental workload prediction level refers to the fine-grained mental workload level that the user currently belongs to, as predicted by the fine-grained mental workload interpretation model based on real-time EEG signals.

[0130] The fine-grained mental workload interpretation model is obtained by training the aforementioned fine-grained mental workload interpretation model of the brain-controlled robotic arm. In one embodiment, for the current real-time EEG signal, features are extracted at multiple preset time scales, and at least two feature views among power spectral density features, differential entropy features, and spatial topological features are constructed. The at least two feature views are input into the pre-trained fine-grained mental workload interpretation model, which encodes and fuses the different feature views to obtain a shared neural representation. Then, a classifier outputs the predicted probabilities corresponding to multiple mental workload levels, and the user's current fine-grained mental workload prediction level is determined based on the category with the highest predicted probability.

[0131] Furthermore, to reduce the impact of occasional EEG fluctuations in a single time window on online prediction results, time smoothing can be applied to the fine-grained mental workload prediction results corresponding to multiple consecutive time windows. Time smoothing can be achieved by one or more of the following methods: moving average, exponential weighted average, median filtering, or probability accumulation. This allows for the acquisition of a dynamic curve showing the change of the user's mental workload state over time based on the prediction results corresponding to consecutive time windows.

[0132] In one implementation, mental workload levels can be divided into multiple consecutive levels, such as level one, level two, level three, level four, and level five, where level one represents a low mental workload state and level five represents a high mental workload state. The system determines the current mental workload prediction level based on the prediction probability output by the classifier and displays the current level and its state curve over time in real time on a display terminal. Furthermore, when the fine-grained mental workload prediction levels corresponding to multiple consecutive real-time EEG signals reach a preset risk level and the duration exceeds a preset time threshold, a warning message can be output. The warning message can be visually prompted through the display terminal, or auditorily prompted through a speaker, or a combination of visual and auditory prompts can be used.

[0133] The method provided in this invention applies a fine-grained mental workload interpretation model trained based on cross-feature view cross-reconstruction shared representation learning to real-time scenarios of teleoperation tasks. It can perform fine-grained and stable online reading, dynamic decoding, and risk warning of the mental workload status of users during the execution of teleoperation tasks.

[0134] Based on any of the above embodiments Figure 3 This is a schematic diagram of the sample expansion and construction process provided by the present invention, as shown below. Figure 3 As shown, firstly, the EEG trials containing lead and time dimensions are processed, i.e., the raw EEG signal segments are input. These raw EEG signal segments undergo multi-scale sliding window processing, segmenting them using windows of different lengths. Then, the feature extraction stage begins, and sub-window feature averaging is performed on the obtained sub-segment features to construct a multi-scale feature vector composed of combinations of different colored blocks. In the sample expansion / construction (multi-scale) stage, for the training scenario, representative features at each time scale are stored as independent samples in the sample library; for the testing scenario, features at different scales are stacked to ultimately generate test samples for input to the fine-grained mental workload interpretation model.

[0135] Figure 4 This is a schematic diagram of the joint architecture for multi-scale sample augmentation and cross-modal cross-reconstruction provided by the present invention, as shown below. Figure 4 As shown, the EEG test signals were processed through the PSD (Power Spectral Density) branch of the yellow path and the DE (Differential Entropy) branch of the green path, respectively, and then processed according to... Figure 3 The process shown generates multi-scale feature input vectors. and In the model architecture, the input vectors are mapped to latent feature representations through their respective encoders. and Then, cross-reconstruction is performed using the decoder, that is, by Reconstructed Corresponding reconstruction loss ,Depend on Reconstructed Corresponding reconstruction loss Fusion Node F Will and After fusion, the data is input into a classifier, which then outputs prediction results for low, medium, or high load.

[0136] The training device for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the present invention will be described below. The training device for the fine-grained mental workload interpretation model of the brain-controlled robotic arm described below can be referred to in correspondence with the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm described above.

[0137] Based on any of the above embodiments, the present invention provides a training device for a fine-grained mental workload interpretation model of a brain-controlled robotic arm. Figure 5 This is a schematic diagram of the training device for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the present invention, as shown below. Figure 5 As shown, the device includes: The determining unit 510 is used to determine an initial interpretation model, acquire training sample data and fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder and a second feature decoder; the training sample data includes EEG signals of the operator when performing a teleoperation task. The feature extraction unit 520 is used to extract features from the electroencephalogram signal in at least two dimensions to obtain a first feature view and a second feature view. The encoding unit 530 is used to input the first feature view and the second feature view into the initial interpretation model respectively, and the first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation; The reconstruction unit 540 is used to reconstruct the first latent feature representation based on the second feature decoder to obtain a second reconstructed feature view, and to reconstruct the second latent feature representation based on the first feature decoder to obtain a first reconstructed feature view; The update unit 550 is used to determine the cross-feature view cross-reconstruction loss based on the difference between the first feature view and the first reconstructed feature view, and the difference between the second feature view and the second reconstructed feature view, determine the target loss based on the cross-feature view cross-reconstruction loss, and update the model parameters of the initial interpretation model based on the target loss.

[0138] The apparatus provided in this invention, based on a first feature view and a second feature view extracted from electroencephalogram (EEG) signals, performs cross-reconstruction of the latent feature representations of another dimension through a first and a second feature decoder, obtaining a first reconstructed feature view and a second reconstructed feature view. It then utilizes the differences between the first feature view and the first reconstructed feature view, as well as between the second feature view and the second reconstructed feature view, to construct a cross-feature view cross-reconstruction loss. On the one hand, through cross-dimensional mutual mapping and reconstruction constraints, it forces the model to mine and learn shared neural representations commonly contained between different feature views, thereby significantly enhancing the model's ability to capture subtle differences in neural representations between adjacent workload levels, achieving the ability to capture the continuous changes in mental workload as task difficulty increases. The system accurately depicts subtle changes, effectively solving the problem of limited granularity and difficulty in accurately depicting continuous changes in existing technologies. On the other hand, the cross-feature view cross-reconstruction loss guides the model to continuously optimize during training, ensuring that the reconstruction result of any view depends on the effective information of the other view. This effectively suppresses specific noise interference in single feature coding branches, extracts stable neural features with strong discriminativeness and robustness, overcomes the technical defects of poor decoding robustness caused by individual differences in EEG signals and limitations of single feature view modeling, and significantly improves the accuracy and stability of fine-grained mental workload recognition. It can support high-precision adaptive interaction and early warning, meeting the actual needs of teleoperation tasks such as space robotic arms.

[0139] Based on any of the above embodiments, the updating unit 550 specifically includes: A fusion unit is used to fuse the first latent feature representation and the second latent feature representation to obtain a fused neural representation; An input unit is used to input the fused neural representation into the classifier in the initial interpretation model to obtain the predicted load level result output by the classifier; A classification loss unit is determined to determine the classification loss based on the fine-grained mental workload level labels and the predicted workload level results; A target loss unit is determined to determine the target loss based on the cross-feature view cross-reconstruction loss and the classification loss.

[0140] Based on any of the above embodiments, the initial interpretation model further includes a third feature encoder; The fusion unit is specifically used for: Obtain the spatial distribution information of the electrodes corresponding to the electroencephalogram (EEG) signals, and construct spatial topological features based on the spatial distribution information of the electrodes; The spatial topological features are encoded based on the third feature encoder to obtain a third latent feature representation; The first latent feature representation, the second latent feature representation, and the third latent feature representation are fused to obtain the fused neural representation.

[0141] Based on any of the above embodiments, the target loss determination unit is specifically used for: Obtain the weight parameters of each network layer in the initial interpretation model, and determine the sparse constraint loss based on the sum of the absolute values ​​of each weight parameter; The target loss is determined based on the sparse constraint loss, the cross-feature view cross-reconstruction loss, and the classification loss.

[0142] Based on any of the above embodiments, the determining unit 510 is specifically used for: Obtain raw EEG signal fragments; The original EEG signal segments were segmented using sliding windows of different lengths to obtain multiple sub-segments of signals corresponding to each time scale. The EEG features corresponding to each of the sub-segments are extracted, and the EEG features at the same time scale are aggregated to obtain representative feature samples for each time scale. Based on representative feature samples of each time scale, the original EEG signal segments are expanded to obtain the training sample data.

[0143] Based on any of the above embodiments, the teleoperation task includes a continuous motion control task for a space robotic arm; the continuous motion control task for a space robotic arm is an operation task in which an operator controls the space robotic arm to complete continuous motion control through electroencephalogram (EEG) signals; the continuous motion control task for a space robotic arm is used to induce fine-grained mental workload for the operator. It also includes a label determination unit, which is specifically used for: The difficulty level of each basic task is determined based on at least one of the following: the spatial structural complexity of the target object, visual perception conditions, the number of operational monitoring parameters, and continuous motion control constraints. Collect the EEG signals of the operator when performing the continuous motion control task of the space robotic arm at each of the basic task difficulty levels; Obtain the results of the subjective evaluation scale filled out by the operators for each of the basic task difficulty levels; Based on the basic task difficulty level and the results of the subjective evaluation scale, the task EEG signals are labeled with fine-grained mental workload level to obtain the fine-grained mental workload level label.

[0144] Based on any of the above embodiments, the spatial structural complexity of the target object is determined based on the geometric shape of the target object to be operated in the continuous motion control task of the spatial robotic arm; the geometric shape includes at least one of a sphere, cylinder, triangular prism, cross prism and pentagon; The visual perception conditions are determined based on the number of cameras deployed on the space robotic arm and the visual information dimension; the visual information dimension characterizes the richness of information about the target state of the target object and the state of the space robotic arm simultaneously acquired by the operator. The number of operation monitoring parameters is determined based on the number of robotic arm state parameters that the operator simultaneously monitors when performing the continuous motion control task of the space robotic arm; the robotic arm state parameters include at least one of the following: joint position, joint speed, robotic arm posture, end effector position, end effector pose, position of the target object, and relative distance between the target object and the space robotic arm; The continuous motion control constraints are based on at least one of the following in the continuous motion control task of the space manipulator: the allowable operational error of the task, the task completion time, the target docking accuracy between the space manipulator and the target object, the motion trajectory complexity of the space manipulator, and the motion range.

[0145] The fine-grained mental workload interpretation device provided by the present invention will be described below. The fine-grained mental workload interpretation device described below can be referred to in correspondence with the fine-grained mental workload interpretation method described above.

[0146] Based on any of the above embodiments, the present invention provides a fine-grained mental workload interpretation device. Figure 6 This is a schematic diagram of the fine-grained mental workload interpretation device provided by the present invention, as shown below. Figure 6 As shown, the device includes: The acquisition unit 610 is used to acquire real-time EEG signals when the user performs a teleoperation task. The prediction unit 620 is used to input the real-time EEG signal into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model. The fine-grained mental workload interpretation model is obtained by executing the training method of the fine-grained mental workload interpretation model of the aforementioned brain-controlled robotic arm.

[0147] The device provided in this invention applies a fine-grained mental workload interpretation model trained based on cross-feature view cross-reconstruction shared representation learning to real-time scenarios of teleoperation tasks. It can perform fine-grained and stable online reading, dynamic decoding, and risk warning of the mental workload status of users during the execution of teleoperation tasks.

[0148] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm. This method includes: determining an initial interpretation model; acquiring training sample data and fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder; the training sample data includes EEG signals from an operator performing a teleoperation task; performing feature extraction on the EEG signals in at least two dimensions to obtain a first feature view and a second feature view; inputting the first feature view and the second feature view into the initial interpretation model, and having the first feature encoder interpret the EEG signals... The first feature view is encoded to obtain a first latent feature representation. The second feature encoder encodes the second feature view to obtain a second latent feature representation. The first latent feature representation is reconstructed based on the second feature decoder to obtain a second reconstructed feature view. The second latent feature representation is then reconstructed based on the first feature decoder to obtain a first reconstructed feature view. Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, a cross-feature view cross-reconstruction loss is determined. A target loss is determined based on the cross-feature view cross-reconstruction loss, and the model parameters of the initial interpretation model are updated based on the target loss.

[0149] The processor 710 can call logic instructions in the memory 730 to execute a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm. The method includes: acquiring real-time EEG signals when a user performs a teleoperation task; inputting the real-time EEG signals into the fine-grained mental workload interpretation model to obtain a fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model; wherein the fine-grained mental workload interpretation model is obtained based on the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm described above.

[0150] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the above methods. The method includes: determining an initial interpretation model; acquiring training sample data and fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder; the training sample data includes electroencephalogram (EEG) signals of an operator performing a teleoperation task; performing feature extraction on the EEG signals in at least two dimensions to obtain a first feature view and a second feature view; and combining the first feature view with the... The second feature views are input into the initial interpretation model. The first feature encoder encodes the first feature view to obtain a first latent feature representation, and the second feature encoder encodes the second feature view to obtain a second latent feature representation. Based on the second feature decoder, the first latent feature representation is reconstructed to obtain a second reconstructed feature view. Based on the first feature decoder, the second latent feature representation is reconstructed to obtain a first reconstructed feature view. Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, a cross-feature view cross-reconstruction loss is determined. Based on the cross-feature view cross-reconstruction loss, a target loss is determined, and the model parameters of the initial interpretation model are updated based on the target loss.

[0152] A computer can execute the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm provided by the above methods. The method includes: acquiring real-time EEG signals when a user performs a teleoperation task; inputting the real-time EEG signals into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model; wherein the fine-grained mental workload interpretation model is obtained based on the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm described above.

[0153] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm provided by the methods described above. This method includes: determining an initial interpretation model; acquiring training sample data and fine-grained mental workload level labels corresponding to the training sample data; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder; the training sample data includes electroencephalogram (EEG) signals of an operator performing a teleoperation task; performing feature extraction on the EEG signals in at least two dimensions to obtain a first feature view and a second feature view; and inputting the first feature view and the second feature view into the initial... In the interpretation model, the first feature encoder encodes the first feature view to obtain a first latent feature representation, and the second feature encoder encodes the second feature view to obtain a second latent feature representation. Based on the second feature decoder, the first latent feature representation is reconstructed to obtain a second reconstructed feature view, and based on the first feature decoder, the second latent feature representation is reconstructed to obtain a first reconstructed feature view. Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, a cross-feature view cross-reconstruction loss is determined. Based on the cross-feature view cross-reconstruction loss, a target loss is determined, and the model parameters of the initial interpretation model are updated based on the target loss.

[0154] When executed by a processor, the computer program implements a training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm provided by the methods described above. This method includes: acquiring real-time EEG signals when a user performs a teleoperation task; inputting the real-time EEG signals into the fine-grained mental workload interpretation model to obtain a fine-grained mental workload prediction level output by the model; wherein the fine-grained mental workload interpretation model is obtained based on the training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm described above.

[0155] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0156] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a fine-grained mental workload interpretation model of a brain-controlled robotic arm, characterized in that, The fine-grained mental workload interpretation model is obtained by iteratively executing the following steps until a preset iteration termination condition is met: An initial interpretation model is determined, and training sample data and fine-grained mental workload level labels corresponding to the training sample data are obtained; the initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder. The training sample data includes electroencephalogram (EEG) signals of operators performing teleoperation tasks. At least two-dimensional feature extraction is performed on the electroencephalogram (EEG) signal to obtain a first feature view and a second feature view; The first feature view and the second feature view are respectively input into the initial interpretation model. The first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation. The first latent feature representation is reconstructed based on the second feature decoder to obtain a second reconstructed feature view, and the second latent feature representation is reconstructed based on the first feature decoder to obtain a first reconstructed feature view; Based on the differences between the first feature view and the first reconstructed feature view, and the differences between the second feature view and the second reconstructed feature view, a cross-feature view cross-reconstruction loss is determined. Based on the cross-feature view cross-reconstruction loss, a target loss is determined, and the model parameters of the initial interpretation model are updated based on the target loss.

2. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to claim 1, characterized in that, The determination of the target loss based on the cross-feature view cross-reconstruction loss includes: The first latent feature representation and the second latent feature representation are fused to obtain a fused neural representation; The fused neural representation is input into the classifier in the initial interpretation model to obtain the predicted load level result output by the classifier; Based on the fine-grained mental workload level labels and the predicted workload level results, the classification loss is determined; The target loss is determined based on the cross-feature view cross-reconstruction loss and the classification loss.

3. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to claim 2, characterized in that, The initial interpretation model also includes a third feature encoder; The step of fusing the first latent feature representation and the second latent feature representation to obtain a fused neural representation includes: Obtain the spatial distribution information of the electrodes corresponding to the electroencephalogram (EEG) signals, and construct spatial topological features based on the spatial distribution information of the electrodes; The spatial topological features are encoded based on the third feature encoder to obtain a third latent feature representation; The first latent feature representation, the second latent feature representation, and the third latent feature representation are fused to obtain the fused neural representation.

4. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to claim 2, characterized in that, The determination of the target loss based on the cross-feature view cross-reconstruction loss and the classification loss includes: Obtain the weight parameters of each network layer in the initial interpretation model, and determine the sparse constraint loss based on the sum of the absolute values ​​of each weight parameter; The target loss is determined based on the sparse constraint loss, the cross-feature view cross-reconstruction loss, and the classification loss.

5. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to any one of claims 1 to 4, characterized in that, The acquisition of training sample data includes: Obtain raw EEG signal fragments; The original EEG signal segments were segmented using sliding windows of different lengths to obtain multiple sub-segments of signals corresponding to each time scale. The EEG features corresponding to each of the sub-segments are extracted, and the EEG features at the same time scale are aggregated to obtain representative feature samples for each time scale. Based on representative feature samples of each time scale, the original EEG signal segments are expanded to obtain the training sample data.

6. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to any one of claims 1 to 4, characterized in that, The teleoperation task includes a continuous motion control task for a space robotic arm; the continuous motion control task for a space robotic arm is an operation task in which the operator controls the space robotic arm to complete continuous motion control through electroencephalogram (EEG) signals, and the continuous motion control task for a space robotic arm is used to induce fine-grained mental load on the operator. The steps for determining the fine-grained mental workload level label include: The difficulty level of each basic task is determined based on at least one of the following: the spatial structural complexity of the target object, visual perception conditions, the number of operational monitoring parameters, and continuous motion control constraints. Collect the EEG signals of the operator when performing the continuous motion control task of the space robotic arm at each of the basic task difficulty levels; Obtain the results of the subjective evaluation scale filled out by the operators for each of the basic task difficulty levels; Based on the basic task difficulty level and the results of the subjective evaluation scale, the task EEG signals are labeled with fine-grained mental workload level to obtain the fine-grained mental workload level label.

7. The training method for the fine-grained mental workload interpretation model of the brain-controlled robotic arm according to claim 6, characterized in that, The spatial structural complexity of the target object is determined based on the geometric shape of the target object to be operated in the continuous motion control task of the spatial robotic arm; the geometric shape includes at least one of a sphere, cylinder, triangular prism, cross, and pentagon. The visual perception conditions are determined based on the number of cameras deployed on the space robotic arm and the visual information dimension; the visual information dimension characterizes the richness of information about the target state of the target object and the state of the space robotic arm simultaneously acquired by the operator. The number of operation monitoring parameters is determined based on the number of robotic arm state parameters that the operator simultaneously monitors when performing the continuous motion control task of the space robotic arm; the robotic arm state parameters include at least one of the following: joint position, joint speed, robotic arm posture, end effector position, end effector pose, position of the target object, and relative distance between the target object and the space robotic arm; The continuous motion control constraints are based on at least one of the following in the continuous motion control task of the space manipulator: the allowable operational error of the task, the task completion time, the target docking accuracy between the space manipulator and the target object, the motion trajectory complexity of the space manipulator, and the motion range.

8. A fine-grained method for interpreting mental workload, characterized in that, include: Acquire real-time EEG signals when a user performs a teleoperation task; The real-time EEG signal is input into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model. The fine-grained mental workload interpretation model is obtained by training the fine-grained mental workload interpretation model of the brain-controlled robotic arm as described in any one of claims 1 to 7.

9. A training device for a fine-grained mental workload interpretation model of a brain-controlled robotic arm, characterized in that, include: The determining unit is used to determine the initial interpretation model, acquire training sample data and the fine-grained mental workload level labels corresponding to the training sample data; The initial interpretation model includes a first feature encoder, a second feature encoder, a first feature decoder, and a second feature decoder; The training sample data includes electroencephalogram (EEG) signals of operators performing teleoperation tasks. The feature extraction unit is used to extract features from the EEG signal in at least two dimensions to obtain a first feature view and a second feature view. The encoding unit is used to input the first feature view and the second feature view into the initial interpretation model respectively, and the first feature encoder encodes the first feature view to obtain the first latent feature representation, and the second feature encoder encodes the second feature view to obtain the second latent feature representation; The reconstruction unit is configured to reconstruct the first latent feature representation based on the second feature decoder to obtain a second reconstructed feature view, and to reconstruct the second latent feature representation based on the first feature decoder to obtain a first reconstructed feature view. The update unit is configured to determine a cross-feature view cross-reconstruction loss based on the difference between the first feature view and the first reconstructed feature view, and the difference between the second feature view and the second reconstructed feature view, determine a target loss based on the cross-feature view cross-reconstruction loss, and update the model parameters of the initial interpretation model based on the target loss.

10. A fine-grained mental workload interpretation device, characterized in that, include: The acquisition unit is used to acquire real-time EEG signals when the user performs teleoperation tasks. The prediction unit is used to input the real-time EEG signal into the fine-grained mental workload interpretation model to obtain the fine-grained mental workload prediction level output by the fine-grained mental workload interpretation model. The fine-grained mental workload interpretation model is obtained by training the fine-grained mental workload interpretation model of the brain-controlled robotic arm as described in any one of claims 1 to 7.

Images