Human-robot tightly coupled multi-modal missing robust cognitive load evaluation method

Abstract

The application provides a multi-modal missing robust cognitive load evaluation method for human-machine close coupling, and relates to the technical field of human-machine state perception interaction. The method comprises the following steps: acquiring multi-modal data and preprocessing, and generating an availability mask according to quality scores. A model containing an expert branch, a shared representation and a gating network is input, and the branch weight is regulated according to the mask; when missing, the query is compensated by the shared branch to update the representation. After the gating network outputs the original weight, the fusion weight is obtained by constraining and normalizing according to the mask; according to the fusion weight, the output of each branch and the global representation are obtained, and the cognitive load evaluation result and the reliability index are obtained through the output head; if the full mode is unavailable, the evaluation failure information is output. Through quality scores, mask determination, missing compensation and weight renormalization, the application realizes robust online evaluation under the condition of missing or degradation of modes, and is suitable for close-coupled human-machine task scenarios such as automatic driving takeover, human-machine cooperation and flight control.

Description

Technical Field

[0001] This invention relates to the field of human-machine state perception and interaction technology, specifically to a robust cognitive load assessment method for multimodal missing conditions in tightly coupled human-machine systems. Background Technology

[0002] Tightly coupled human-machine tasks refer to mission scenarios where operators form a continuous closed loop of perception, decision-making, operation, and feedback with complex human-machine interaction systems such as vehicles, intelligent cockpits, aircraft, special equipment, collaborative robots, and unmanned systems. Typical applications include autonomous driving takeover, complex monitoring, flight control, teleoperation, and digital surgery. In these tasks, real-time and accurate assessment of operator cognitive load is crucial for ensuring mission safety and improving human-machine collaboration efficiency. Since single-modal information cannot fully reflect the operator's cognitive state, multimodal data integrating various types of signals from multiple aspects of the human and machine systems for cognitive load assessment has become the mainstream technical approach in this field.

[0003] Existing multimodal cognitive workload assessments typically employ feature-level or decision-level fusion methods. For example, in driving workload calculation methods, multidimensional workload information is simultaneously acquired from multiple sensors such as EEG, eye-tracking, ECG, respiration, skin conductance, and GPS, and then comprehensively calculated by combining vehicle status, driving behavior, and subjective evaluation data. Other methods include preprocessing multimodal human factors data and inputting it into a state recognition model to output driving status and generate feedback interventions. Additionally, based on multimodal physiological and behavioral data such as EEG, ECG, skin conductance, and eye-tracking, a mental workload classification model is established through feature extraction and pattern recognition. The general implementation of these methods is as follows: multimodal data acquisition → time alignment and preprocessing → feature extraction and stitching → fusion classification / regression → output assessment results.

[0004] The aforementioned methods are highly dependent on the integrity of input modalities and the stability of signal quality. The fusion inference mechanism lacks the ability to adaptively perceive and dynamically adjust the availability of modalities: it cannot explicitly distinguish between available, deweighted, and unavailable states of modalities at the time window level, nor can it adaptively close or reduce the fusion contribution of untrusted branches or compensate for missing information in a lightweight manner when modalities are missing or their quality degrades. Furthermore, it lacks explicit handling strategies for extreme cases such as all modalities being unavailable. When sensors experience modal loss or quality degradation due to occlusion, poor contact, vibration, or link anomalies, the model still uses a fixed input structure and fusion weights, leading to decreased evaluation accuracy, unstable output, and even unreliable evaluation results. This makes it difficult to meet the robustness and reliability requirements of online cognitive load assessment for tightly coupled human-machine tasks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a robust cognitive load assessment method for multimodal missing features in tightly coupled human-machine tasks. This method solves the problems of existing cognitive load assessment methods in tightly coupled human-machine tasks being highly dependent on the integrity of input modalities and the stability of signal quality, resulting in decreased assessment accuracy and unstable output when modalities are missing, quality degrades, or time is out of sync. Furthermore, the inconsistency between the full-modal conditions during the training phase and the missing-modal conditions during the deployment phase leads to output deviations, and there is a lack of explicit handling strategies for extreme cases such as when all modalities are unavailable.

[0006] To achieve the above objectives, this invention provides a robust cognitive load assessment method for multimodal loss in tightly coupled human-machine interactions, comprising: Acquire multiple modal data of operators within the current time window, and perform time alignment and preprocessing on each modal data; Calculate the quality score for each modality and generate a modality availability mask based on the quality score; the modality availability mask is used to indicate whether the corresponding modality is available, downweighted, or unavailable. The preprocessed multimodal data is input into the cognitive load assessment model; the cognitive load assessment model includes multiple modality expert branches, shared representation branches, gating networks, and output heads; the cognitive load assessment model is a model trained by consistency constraints between the full modality input path and the input path with missing or degraded modality. Based on the modal availability mask, for unavailable modes, the corresponding modal expert branches are closed; for deweighted modes, the weights of the corresponding modal expert branches are reduced. When there are unavailable modes or reduced-weight modes that meet the preset compensation triggering conditions, the shared representation branch generates a compensation query and extracts information from the features of the remaining available modes based on the compensation query to update the global representation of the available modes; Based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain the fused weights; The outputs of each modality expert branch, the output of the shared representation branch, and the updated global representation are fused according to the fusion weights to obtain the fused representation; the fused representation is input into the output header to output the cognitive load assessment results and reliability indicators for the current time window; When all modalities are unavailable, the fusion and evaluation of the current time window are not performed, and the status information, reasons and reliability indicators of the unmet evaluation conditions are output.

[0007] In one embodiment of the present invention, the multiple modal data includes at least two of the following: human physiological signals, human behavioral signals, machine state signals, human-machine interaction behavioral signals, and environmental / task context signals.

[0008] In one embodiment of the present invention, the modality expert branch extracts modality features from the corresponding modality data and aggregates the modality features into a global representation of the corresponding modality; the shared representation branch generates a cross-modality shared representation based on the modality features of the currently available modality and generates a compensation query based on the cross-modality shared representation.

[0009] In one embodiment of the present invention, a quality score for each modal data is calculated, and a modal availability mask is generated based on the quality score, including: When the quality score of the modal data is not lower than the preset quality threshold, the corresponding modality is determined to be in a usable state; When the quality score of modal data is lower than the preset quality threshold but higher than the preset unusable threshold, the corresponding modality is determined to be in a downweighted state. When the quality score of the modal data is not higher than the preset unusable threshold, the corresponding modality is determined to be unusable.

[0010] In one embodiment of the present invention, information is extracted from the features of the remaining available modalities based on a compensation query, and the global representation of the available modalities is updated, including: Using the compensation query as the query vector and the sequence features of the remaining available modalities as the key vector and value vector, the compensation term is calculated through cross-attention. The compensation term is added to the global representation of the available modes and then layer normalized to obtain the updated global representation.

[0011] In one embodiment of the present invention, based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain fused weights, including: The original weights of the branches corresponding to the modes that are in an unavailable state are reset to zero; The original weights of the branches corresponding to the modes in the reduced weight state are multiplied by the corresponding reduction coefficient, which takes a value between 0 and 1; The combined weight is obtained by dividing the weights of each constrained branch by the sum of all constrained weights.

[0012] In one embodiment of the present invention, the cognitive load assessment model is trained in the following manner: For the same sample, construct a full-modal input path and a modality-deficient or quality-degraded input path; the missing or quality-degraded input path is obtained by simulation through at least one of the following methods: random modality discarding, masking to zero, adding noise, or occlusion; The inputs of the two paths are fed into the cognitive load assessment model to be trained, and their respective prediction results are obtained. The task loss is calculated based on the prediction results and sample labels, and consistency constraints are imposed on the prediction results or shared representations of the two paths. The consistency constraints include any one or more of the following: distribution consistency constraints, numerical consistency constraints, and representation consistency constraints. The model parameters of the modal expert branch, shared representation branch, gating network, and output head are updated based on task loss and consistency constraints, so that the prediction results or shared representations of the same sample under the full modal input path are consistent or close to those under the input path with missing modalities or degraded quality.

[0013] This invention also provides a robust cognitive load assessment system for multimodal loss in tightly coupled human-machine interactions, comprising: The operator acquisition unit is used to acquire human physiological and behavioral signals of the operator within the current time window. The device / self-entity interface unit is used to access machine-side status signals, human-machine interaction behavior signals, and environmental / task context signals. Edge computing units are used to cache, synchronize, and preprocess data from each modality, calculate the quality score of each modality, and generate a modality availability mask based on the quality score. The modality availability mask is used to indicate whether the corresponding modality is available, downweighted, or unavailable. Collaborative assessment platform, used for: The preprocessed multimodal data is input into the cognitive load assessment model; the cognitive load assessment model includes multiple modality expert branches, shared representation branches, gating networks, and output heads; the cognitive load assessment model is a model trained by consistency constraints between the full modality input path and the input path with missing or degraded modality. Based on the modal availability mask, for unavailable modes, the corresponding modal expert branches are closed; for deweighted modes, the weights of the corresponding modal expert branches are reduced. When there are unavailable modes or reduced-weight modes that meet the preset compensation trigger conditions, the shared representation branch generates a compensation query and extracts information from the features of the remaining available modes based on the compensation query to update the global representation of the available modes. Based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain the fused weights; The outputs of each modality expert branch, the output of the shared representation branch, and the updated global representation are fused according to the fusion weights to obtain the fused representation; the fused representation is input into the output header to output the cognitive load assessment results and reliability indicators for the current time window; When all modalities are unavailable, the fusion and evaluation of the current time window are not performed, and the status information, reasons and reliability indicators of the unmet evaluation conditions are output.

[0014] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0016] This invention provides a robust cognitive load assessment method for multimodal deficiencies in tightly coupled human-machine interactions. Compared with existing technologies, it has the following advantages: By generating an availability mask based on quality scores, unusable modes are explicitly masked and inferior modes are weighted to avoid noise interference. Compensation queries are generated through shared branches, and the global representation is updated using the remaining modes, supplementing discriminative information without reconstructing the original signal, ensuring continuous and accurate evaluation. The fusion weights are dynamically adjusted by combining mask constraints and renormalization to improve the adaptability of multimodal combinations. Errors are proactively reported when all modes fail, avoiding invalid inference. This mechanism significantly enhances the robustness and reliability of evaluation under modal loss or degradation, and is suitable for tightly coupled task scenarios such as autonomous driving, human-machine collaboration, and teleoperation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a robust cognitive load assessment method for multimodal loss oriented towards tight human-machine coupling provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system structure provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the overall process of the method provided in the embodiments of the present invention; Figure 4 This is a schematic diagram of an offline training strategy provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the modal expert branch and shared representation branch fusion model structure provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the bidirectional missing compensation and guided aggregation module provided in an embodiment of the present invention; Figure 7 A simplified diagram of the robust cognitive load assessment mechanism under multimodal integrity and deficiency conditions provided in this embodiment of the invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This application provides a robust cognitive load assessment method for multimodal missing features in tightly coupled human-machine tasks. It addresses the technical problems of existing cognitive load assessment methods in tightly coupled human-machine tasks, which are highly dependent on modal integrity and signal quality, suffer from decreased assessment accuracy and unstable output when modalities are missing, quality degrades, or time is out of sync, and experience output deviations due to inconsistencies between full-modal conditions during training and missing-modal conditions during deployment. The method achieves adaptive expert routing, missing feature compensation, and gated weight renormalization based on modal quality scores and availability masks. It maintains the continuity and reliability of the assessment output without reconstructing the original signals of missing modalities, effectively improving the robustness of online cognitive load assessment in scenarios with missing modalities or quality degradation.

[0021] like Figures 1-7 As shown, the present invention provides a robust cognitive load assessment method for multimodal missing data in a tightly coupled human-machine environment, including an offline training phase and an online assessment phase.

[0022] The offline training phase includes: collecting and synchronizing multimodal training samples, calculating the quality score and modality availability mask for each modality, and constructing full-modal input paths as well as input paths with missing or degraded modalities. These two types of inputs are then fed into the modality expert branch, shared representation branch, missing representation compensation module, and gated fusion module, respectively, to calculate the cognitive load task loss and the consistency loss between the full-modal path and the missing modality path. The model parameters are updated based on the joint loss to obtain the trained cognitive load assessment model.

[0023] During the offline training phase, a trained cognitive load assessment model is obtained through modality random missing enhancement, modality quality degradation enhancement, and consistency training between full modality paths and missing modality paths.

[0024] The online evaluation phase includes: acquiring multimodal data from the human, machine, interaction, and environment / task sides within the current time window and performing time alignment and preprocessing. Calculating quality scores for each modality and generating modality availability masks. Extracting features through modality expert branches and shared representation branches, and closing or downweighting corresponding expert branches based on the modality availability mask. When unavailable modalities exist, generating compensation queries based on shared representations and performing guided aggregation from the remaining available modalities. Masking and renormalizing the gating weights to obtain a fused representation. Outputting the cognitive load level or continuous load value and its reliability index (the reliability index quantifies the reliability of the cognitive load evaluation results within the current time window and is related to the quality status of the input modalities, the number of available modalities, the distribution of gating weights, and the confidence level of model predictions. A higher reliability index value indicates a more reliable evaluation result); when all input modalities are unavailable, stopping the cognitive load evaluation for the current time window and outputting status information indicating that the evaluation conditions are not met.

[0025] The online assessment phase uses a trained model to robustly assess the cognitive load of operators within the current time window.

[0026] like Figure 3 As shown, the method flow of the present invention is divided into an offline training stage on the left and an online evaluation stage on the right.

[0027] Offline training phase: Input multimodal synchronous data (including non-contact visual data: video, eye tracking, AU, etc.; physiological data: EEG, ECG, EDA, etc.). Through training augmentation strategies (training only), perform modal dropout or masking augmentation, randomly selecting m... vis or m phys Set to 0 to simulate modal loss or quality degradation at the deployment end. Input the mask m into the expert-shared-gated encoder, where each expert branch extracts the corresponding modal features, and the shared expert learns the cross-modal shared representation z. shared The gating network outputs weights g based on the input and mask. Consistency / alignment / compensation effectiveness constraints are then applied, including representation consistency constraints, cross-modal alignment constraints, and task loss L. task The joint training objective is L = L task + λ1·L cons +λ2·L aux +λ3·L reg The final output is the trained model parameters θ. * (Including shared experts and gating strategies) for online assessment.

[0028] Online evaluation phase: Input available modal data, and generate a modal availability mask m=[m] after quality evaluation. vis ,m phys ],in ∈[0,1], i can be represented as vis or phys, m i =1 indicates that the corresponding mode is available, 0< <1 indicates that the corresponding mode is weighted. =0 indicates that the corresponding mode is unavailable; binary mask ∈{0,1} is a special case of continuous availability masking. Execute dynamic inference path: close missing branches, renormalize gating weights, and only compute available experts, shared experts, and compensation modules. Execute bidirectional missing compensation: generate compensation queries from shared representations, for example... Used to guide the updating of visual representations. Used to guide the updating of physiological representations. The final output is the cognitive load level or score and confidence level, which is used for display, alarm and recording on the subject's end.

[0029] In one embodiment, the method for assessing robust cognitive load due to multimodal loss in tightly coupled human-machine interactions includes the following steps: S100: Obtain multiple modal data of the operator within the current time window, and perform time alignment and preprocessing on each modal data. Step S100 also includes the following:

[0030] S100-1: Acquisition of multimodal data.

[0031] In one embodiment, the multiple modal data includes at least two of the following: human physiological signals, human behavioral signals, machine state signals, human-machine interaction behavioral signals, and environmental / task context signals.

[0032] The human-side physiological signals preferably include EEG signals; in optional embodiments, they may further include one or more of the following: ECG signals, EDA signals, PPG (photoplethysmography) signals, respiratory signals, or functional near-infrared spectroscopy (fNIRS) signals. Human-side behavioral signals preferably include at least one of the following: operator eye movement sequences, fixation points, pupil diameter, blinking, facial region video, facial key points, posture, voice, key presses, mouse actions, or manipulation behaviors. Aircraft-side status signals include status data of the vehicle, aircraft, smart cockpit, special equipment, unmanned system, or robot, such as equipment availability status, system alarm information, telemetry data, and control status. Human-machine interaction behavioral signals include interaction event logs between the operator and the system, operation response latency, key press / manipulation data continuity, and takeover event validity. Environmental / task context signals include illumination, noise, network latency, link packet loss rate, task stage, and task context integrity.

[0033] In one embodiment, the aforementioned multimodal data is acquired through an operator acquisition unit and a device / autonomous entity interface unit. The operator acquisition unit is deployed in a vehicle cockpit, flight simulator, aircraft training platform, special equipment control console, collaborative robot workstation, complex monitoring platform, or digital surgical environment to collect human-side physiological and behavioral signals from the operator. The device / autonomous entity interface unit is used to access aircraft-side status signals, human-machine interaction behavior signals, and environmental / task context signals.

[0034] like Figure 2 As shown, the overall system architecture of this invention is divided into three main parts: the subject end / airborne end, data transmission and synchronization, and the test subject end / ground station.

[0035] The subject / airborne end includes: a non-contact visual acquisition module for acquiring RGB / IR / Depth video, eye tracking, facial landmarks, and facial motion unit (AU) data, and calculating the visual modality quality score s_vis; and a contact physiological acquisition module for acquiring physiological signals such as EEG, ECG, and EDA, and calculating physiological modality quality scores s_vis, including lead impedance, artifact rate, and packet loss rate. phys Synchronization and edge gateways are used for timestamp alignment, data caching, initial denoising and normalization, and to generate a modal availability mask m=[m vis , m phys Local recording and fault tolerance module for local caching and breakpoint resumption; optional airborne edge inference module for performing modality-deficient adaptive inference and outputting airborne alarms or cognitive load scores.

[0036] The data transmission and synchronization component includes: an interface and transmission link, supporting wired or wireless methods, with compression and bandwidth adaptive capabilities; a protocol and security module for data packetization, verification, and retransmission, with optional encryption and access control; a real-time and fault-tolerant module for monitoring transmission latency and packet loss rate, and reporting missing modal or poor quality information when the link is abnormal; a data access service for streaming access and API interfaces, supporting data storage and message queues; and an optional deployment module for distributing inference tasks to airborne edge devices or ground stations.

[0037] The test site / ground station includes: a receiving and data management module for data storage, annotation, playback, version management, and auditing; and a preprocessing and quality assessment module for data alignment, noise reduction, segmentation, and generating a mask m based on the quality score. i =1(s i ≥τ i ); Modality-deficient adaptive inference engine θ *It adopts an expert-shared-gated structure, closes missing branches by mask m, generates compensation vectors from shared representations, and outputs cognitive load levels or continuous scores; the human-computer interaction interface / decision support module is used for real-time curve display, dashboard, alarm prompts, event annotation, recording and export; the output and interface module is used for generating reports and interface linkage with other systems.

[0038] It should be noted that the system can perform inference on the airborne terminal or the ground station, and automatically switch to an efficient path when a mode is missing, so as to reduce computing power consumption and latency.

[0039] S100-2: Time alignment.

[0040] Because modal data from different sources have different sampling rates and transmission link delays, in one embodiment, time alignment is performed on the multimodal data. This time alignment can be achieved using any combination of one or more of the following methods: 1) Unified hardware trigger and shared clock: The unified hardware trigger signal controls the synchronous start of each acquisition device, and the shared clock is used to stamp the data of each mode with a unified timestamp.

[0041] 2) Unified timestamp and software alignment: Each acquisition device records its local timestamp, which is then converted to the same time base by the edge computing unit or collaborative evaluation platform, and then interpolated or resampled for alignment according to the timestamp.

[0042] 3) Post-processing alignment based on critical events: Using critical events in the task (such as takeover events, instruction issuance events, operation response events, etc.) as alignment anchors, post-processing alignment of multimodal data before and after the event is performed.

[0043] 4) Time correction based on link delay and jitter estimation: Estimate the transmission delay and jitter of each data link and compensate for the arrival time of each mode of data.

[0044] 5) Resampling or window alignment: Resample modal signals with different sampling rates, or map each modal data to a unified time window sequence ("time window" is a fixed-duration segment for sliding continuous multimodal data, each time window corresponds to an evaluation unit, and the model continuously outputs evaluation results according to the time window sliding method), so that each time window contains data of the time range corresponding to each modality.

[0045] In one embodiment, the time-aligned modal data is segmented according to a preset time window length T to obtain the sequence features of each modality within the time window. The time window length can be set according to the task scenario and model latency requirements, for example, it can be set to 1 second, 2 seconds, 5 seconds or 10 seconds. Each time window corresponds to an evaluation unit, and the model continuously outputs cognitive load evaluation results in a time window sliding manner.

[0046] S100-3: Preprocessing.

[0047] In one embodiment, the time-aligned modal data are preprocessed, and the preprocessing includes one or more of the following methods: 1) Denoising: Use methods such as bandpass filtering, adaptive filtering, or wavelet denoising to remove noise and artifacts from physiological signals. ("Artifacts" refer to interference components in physiological signals caused by non-cognitive load sources, such as signal distortion caused by electrooculography, electromyography, electrode movement, power frequency interference, etc. The quality of physiological signals can be assessed based on the proportion of artifacts.)

[0048] 2) Normalization: Perform z-score standardization (z-score standardization is zero-mean standardization, which maps features to a distribution with a mean of 0 and a standard deviation of 1) or min-max normalization (min-max normalization refers to linearly mapping features to the [0,1] interval, i.e., normalizing each feature dimension according to (xx) min ) / (x max -x min (Scaling) maps features of different dimensions to a uniform scale.

[0049] 3) Resampling: Modal data with different sampling rates are uniformly resampled to the preset target sampling rate.

[0050] 4) Detrending: Removes linear trends or baseline drift from the signal.

[0051] 5) Segmentation and feature extraction: The signal is segmented according to time windows, and time-domain, frequency-domain or time-frequency-domain features are extracted within each time window, or the original waveform or feature sequence is organized into a token sequence for model input.

[0052] It should be noted that the specific preprocessing method can be flexibly selected according to the modality type and task scenario, and there is no limitation on the implementation form of preprocessing.

[0053] S200: Calculate the quality score for each modality data, and generate a modality availability mask based on the quality score; the modality availability mask is used to indicate whether the corresponding modality is available, downweighted, or unavailable. Step S200 also includes the following:

[0054] S200-1: Calculation of quality score.

[0055] In one embodiment, a quality score is calculated for each mode i within each time window. The quality score This is used to measure the completeness, reliability, and usability of the corresponding modal data within that time window. After normalization, the quality score... The value range is [0,1], and the higher the value, the better the quality of the modal data.

[0056] In one embodiment, when modal data is completely missing, let =0.

[0057] In one embodiment, the quality scoring indicators for different modal types can be selected according to the scenario: 1) Human physiological signals: Judgment can be made based on lead impedance, artifact ratio, frequency band energy abnormality, signal integrity, or noise ratio. For example, for EEG signals, indicators such as the proportion of artifact segments within the time window, whether the energy of each frequency band is abnormal, and whether the signal amplitude exceeds the limit can be calculated.

[0058] 2) Human-side behavioral signals: These can be determined based on eye movement loss rate, gaze point confidence, facial key point confidence, posture recognition confidence, occlusion rate, or exposure.

[0059] 3) Machine-side status signals: These can be determined based on equipment availability, system alarms, telemetry integrity, or control status stability.

[0060] 4) Human-computer interaction behavior signals: can be determined based on the completeness of event logs, operation response delay, continuity of key / operation data, or the validity of takeover events.

[0061] 5) Environment / task context signals: These can be determined based on network latency, link packet loss rate, illumination, noise, task phase integrity, or context reliability.

[0062] S200-2: Example of human-side behavioral signal quality scoring.

[0063] In one embodiment, taking eye-tracking modality as an example, within a time window containing N frames of eye-tracking data, let d t This indicates whether a valid pupil or fixation point detection result was obtained in frame t. A valid result is indicated by d. t =1, otherwise d t =0; let c =0; t ∈[0,1] represents the pupil detection confidence or fixation confidence of the corresponding frame; let b t This indicates whether the frame is considered a blink frame.

[0064] The eye-tracking loss rate, the average confidence of effective frames, and the proportion of blinking frames can be expressed as follows: in, For eye movement loss rate, The average confidence score of the effective frames is given by max(·, 1), which means taking the larger of the value in parentheses and 1 to avoid the denominator being zero. The percentage of blink frames represents the proportion of blink frames. These three metrics are used to measure the degree of missing eye movement data, detection quality, and the degree of blink interference, respectively.

[0065] An eye-tracking modal quality score can be represented as follows: Here, `clip[x, 0, 1]` restricts the value of `x` to between 0 and 1: 0 for `x < 0`, 1 for `x > 1`, and `x` itself otherwise. α, β, and γ are non-negative weights satisfying α + β + γ = 1. This can be... A binary availability mask is generated by comparing the mask with a preset quality threshold, or the normalized mask can be directly used. This serves as a continuous availability mask. The weights and thresholds can be determined based on device performance, acquisition environment, or verification data; the present invention does not limit their specific values ​​or combinations.

[0066] S200-3: Generation of modal availability mask.

[0067] In one embodiment, a quality score is calculated for each mode i within each time window. and will Normalize to the [0,1] interval; when modal data is completely missing, let =0.

[0068] Modal availability mask Used to indicate the available state of a corresponding mode. In one embodiment, a binary mask may be used: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Compared with the preset quality threshold Compare and generate a binary availability mask: in This is an indicator function. When =1 indicates that mode i is available / trusted. =0 indicates that mode i is missing or of insufficient quality and is therefore unavailable.

[0069] In another embodiment, a continuous value mask can be used: directly using the normalized quality score. As a mask value, i.e. = , ∈[0,1]. Where: =1 indicates that the corresponding mode is fully available; 0< <1 indicates that the corresponding mode has quality degradation, and its expert branches participate in the fusion according to Reduce the weight; =0 indicates that the corresponding mode is missing or of insufficient quality and is therefore unusable, thus disabling the corresponding expert branch. Optionally, the continuous mask can be discretized into a three-state mask: =1 corresponds to an available state, 0< <1 corresponds to a reduced weighting state and the reduced weighting coefficient is , =0 corresponds to an unusable state. The tri-state mask is merely an intuitive representation of the continuous mask and does not change the actual calculation method of the continuous mask.

[0070] A tri-state mask is a discretized representation of a continuous-value mask, used to intuitively describe modal states: =1 corresponds to an available state, 0< <1 corresponds to a deweighted state. =0 corresponds to an unavailable state. In subsequent gating weight calculations, the actual continuous mask values ​​are still used in the calculations; the tri-state mask is only used for state determination and control logic.

[0071] In embodiments employing continuous value masks, a compensation trigger threshold can be set. ∈(0,1). By default, when Missing compensation is triggered when =0; when set At that time, if < If so, the corresponding modality will be treated as unavailable and missing compensation will be triggered; if ≤ If the value is less than 1, then the corresponding modal expert branch will be downweighted only according to the continuous availability mask, and missing compensation will not be triggered.

[0072] It should be noted that the above quality thresholds Compensation trigger threshold The settings can be configured separately for different modalities or uniformly. Specific values ​​can be determined based on device performance, acquisition environment, or verification data; this invention does not limit specific values. When using a binary mask, a modality only has two states: available and unavailable. When using a continuous value mask (including tri-state representation), a modality has three states: available, deweighted, and unavailable. "Available state" means the corresponding modality data is complete and meets quality requirements, and its expert branches participate in fusion normally. "Deweighted state" means the corresponding modality data is still usable but has quality degradation; its expert branches contribute less when participating in fusion according to the deweighting coefficient. "Unavailable state" means the corresponding modality data is missing or severely lacking in quality; the corresponding expert branches are closed and do not participate in the fusion of the current time window.

[0073] S300: Input the preprocessed multimodal data into the cognitive load assessment model; the cognitive load assessment model includes multiple modality expert branches, a shared representation branch, a gating network, and an output head; wherein, the cognitive load assessment model is a model trained by consistency constraints between the full modality input path and the input path with missing or degraded modalities. Step S300 also includes the following:

[0074] S300-1: Overall structure of the model.

[0075] like Figure 5 As shown, the input on the left is the visual modal input x. vis / tokens (which may include RGB / IR / Depth video, eye tracking, facial AU, etc.) and physiological modality input x phys / tokens (which may include EEG, ECG, EDA, etc.). Input data first passes through a quality evaluator, based on scores. i Generate modal availability mask m=[m vis , m phys ], where m i ∈[0,1];m i =1 indicates that the corresponding mode is available, 0 <m i <1 indicates that the corresponding modal quality has degraded and is calculated according to m i Participating in the integration with reduced weight, m i =0 indicates that the corresponding mode is missing or of insufficient quality and is therefore unusable.

[0076] The data then enters the encoder, which includes a visual expert branch, a physiological expert branch, and a shared expert branch. The visual expert branch outputs the visual modality representation z. vis Physiology expert branch outputs physiological modality representation z phys When m phys When =0, this branch is disabled; the shared expert branch outputs a cross-modal shared representation z. shared , as a source of information for public representation and loss compensation.

[0077] Each expert's output is fed into a gating network, which calculates the original weights g = softmax(·) and performs masking constraints and renormalization: g' = renorm(g⊙m). Then, weighted fusion is performed to calculate the fused representation. Finally, the input and output headers will be merged, and the output will be the cognitive load level or score and confidence level.

[0078] In one embodiment, the cognitive load assessment model includes multiple modal expert branches, a shared representation branch, a gating network, and an output head.

[0079] The input to the cognitive load assessment model includes: time-aligned and preprocessed modal feature sequences. Quality score vector and modal availability mask vector , where M is the total number of modes.

[0080] S300-2: Modal Expert Branch.

[0081] In one embodiment, each modality corresponds to a modality expert branch (i.e., expert sub-network), which is used to encode the feature sequence of the corresponding modality, extract modality-specific features, and aggregate the modality features into a global representation of the corresponding modality.

[0082] For the i-th mode, its expert branch can be represented as: in, The i-th mode is the feature sequence after time alignment and preprocessing. For the expert subnetwork corresponding to the i-th mode, This represents the modal representation output by the i-th modal expert branch.

[0083] The specific structure of the expert subnetwork is not limited, and any neural network structure capable of sequence feature extraction, such as Transformer encoder, convolutional neural network, recurrent neural network, long short-term memory network, or temporal convolutional network, can be selected.

[0084] In one embodiment, human-side behavioral / visual modality and human-side physiological modality are used as examples: The input for the visual expert branch is ∈ The output is In this application, Represents the set of real numbers; Describe the space of real matrices with T rows and d columns. Let T represent a d-dimensional real vector space. T is the sequence length within the time window, and d is the feature dimension. In a preferred embodiment, the visual expert branch employs a Transformer encoder and adds a [CLS] marker before the input sequence. The output vector at the position corresponding to the [CLS] marker serves as the global representation of that modality. .

[0085] The input for the Physiology Expert branch is ∈ The output is In a preferred embodiment, the physiological expert branch can also employ a Transformer encoder, and the global representation of the physiological modality can be obtained by converging through [CLS] tags. .

[0086] It should be noted that, in this application, "modal features" refers to the intermediate features extracted by the modal expert branch from the input sequence; "modal representation" refers to the feature representation output by the modal expert branch; and "global representation" refers to the window-level vector obtained after the modal representations are aggregated. For example, in an embodiment using a Transformer encoder and adding a [CLS] marker before the input sequence, [CLS] is a special marker appended to the sequence encoder input, used to aggregate global information of the entire sequence; in a preferred embodiment, the output vector corresponding to [CLS] serves as the global representation of the modality. The output vector at the position corresponding to the [CLS] marker is the global representation of the modality; in an embodiment using pooling, the vector obtained after performing average pooling or max pooling on the modal representation is also the global representation of the modality. In this document, without causing ambiguity, "modal representation" and "global representation" are collectively referred to as modal branch output.

[0087] S300-3: Shared representation branch.

[0088] The shared representation branch is used to learn common semantic representations across modalities, serving as a source of information for cross-modal alignment anchors and missing information compensation.

[0089] In one embodiment, the input to the shared representation branch is the feature sequence of the currently available modality or its global representation, and the output is the cross-modal shared representation. .

[0090] In a preferred embodiment, the shared representation branch takes the global representation of all currently available modalities as input and is generated through a self-attention mechanism (referring to an attention calculation method where the query vector Q, key vector K, and value vector V all come from the same sequence, used for information aggregation within the same sequence) or a feedforward mapping network. .

[0091] In another embodiment, the shared representation branch runs in parallel with each modality expert branch, taking the original feature sequences of each modality as input, and extracting cross-modality common representations through a parameter-shared encoder.

[0092] The shared representation Its functions include: (1) participating in gating fusion as one of the fusion branches; (2) generating compensation queries for missing or low-quality modalities; and (3) serving as an alignment object for full modal paths and missing modal paths in consistency training.

[0093] S300-4: Gated network.

[0094] In this application, the weight vector directly output by the gating network is called the "original weight" (or "original gating weight"); the weight vector obtained after modal availability mask constraint and renormalization is called the "fused weight" (or "gating weight after mask constraint"). Both are collectively referred to as "gating weight" in this application, and their specific meaning is determined according to the context.

[0095] The gated network is used to output the original weights of each fusion branch.

[0096] In one embodiment, the input feature h of the gating network can be expressed as: That is, the input of the gating network is the output of each modality expert branch. Cross-modal shared representation The quality score vector s and the modal availability mask vector m together constitute the output. The gating network outputs the original weight vector g of each fusion branch.

[0097] Gated networks are implemented using single-layer or multi-layer fully connected networks with softmax (softmax function, normalized exponential function): Concat(·) represents the vector concatenation operation, which connects the vectors end to end along the dimensional direction to synthesize a higher-dimensional vector; Indicates the total number of input modes; This represents the input characteristics of the gating network; This represents the weight matrix of the gated network; Represents the bias vector; This represents the original gating weight vector for each fusion branch. If... The total number of branches participating in gating fusion is Then it can be made , Gated network output In settings In the implementation of one modality expert branch and one shared representation branch, .

[0098] The output of the gating network Let Σ represent the original weights of the e-th fusion branch, satisfying Σ =1.

[0099] S300-5: Output head.

[0100] The output head is used to output the cognitive load assessment results based on the fused representation.

[0101] In one embodiment, the output head includes a cognitive load output head and a reliability output head.

[0102] The cognitive load output head is used to output cognitive load levels or continuous load values. When the task is cognitive load classification, the cognitive load output head can use a softmax classifier to output the probability distribution of each cognitive load level; when the task is cognitive load regression, the cognitive load output head can use a linear regression layer or an MLP (Multilayer Perceptron) to output continuous cognitive load scores.

[0103] The reliability output header is used to output a reliability metric corresponding to the cognitive load assessment results. In one embodiment, the reliability output header takes a fused representation as input and outputs a reliability score between 0 and 1 via a sigmoid function (a sigmoid function used to map the input to a value between 0 and 1 to represent probability or confidence) or softmax. The reliability metric is related to the quality state of the input modality, the gating weight distribution, or the model prediction confidence.

[0104] S300-6: Offline training phase.

[0105] In one embodiment, the cognitive load assessment model is obtained through offline training, the training process including: S310. Collect multimodal training samples, and construct a full-modal input path and a modality-degraded or quality-degraded input path for the same sample. The missing or quality-degraded input path is obtained by simulation through at least one of the following methods: random modality discarding, masking to zero, adding noise, or occlusion.

[0106] S320. Input the inputs of the two paths into the cognitive load assessment model to be trained, and obtain their respective prediction results.

[0107] S330. Calculate the task loss based on the prediction results and the true labels, and impose consistency constraints on the prediction results or shared representations of the two paths.

[0108] S340. Update the model parameters of the modal expert branch, shared representation branch, gating network, and output head according to the task loss and consistency constraints, so that the prediction results or shared representations of the same sample under the full modal input path and the input path with missing modalities or degraded quality are consistent or close.

[0109] like Figure 4 As shown, the offline training phase primarily uses complete multimodal input, combined with random missing data augmentation. Specifically, the input is multimodal synchronous data (non-contact visual vis and physiological phys), and through training augmentation strategies (used only for training), modality dropout or masking augmentation is performed on the input, randomly selecting... or Set to 0 to simulate potential modal loss or quality degradation at the deployment end. Input the mask m into the expert. shared In a gated encoder, each expert branch extracts corresponding modal features, and shared expert learning enables cross-modal shared representations. The gating network outputs fusion weights g based on the input and mask.

[0110] Subsequently, consistency constraints, alignment constraints, and task loss are applied to the model. The consistency constraints are used to maintain shared representations across different modal combinations. Stability; cross-modal alignment is used to align available modalities with the shared space; task loss Used to optimize cognitive load classification or regression objectives. The joint training objective is... After training, the model parameters θ are obtained. This parameter includes shared experts and gating strategies, which automatically switch to efficient paths and generate compensation vectors when modalities are missing during online inference.

[0111] In one embodiment, to reduce the output deviation between full-modal input conditions during the training phase and missing-modal input conditions during the deployment phase, this invention performs modality dropping or masking enhancement on the input modalities during the training phase to simulate scenarios where modality loss or insufficient quality may occur at the deployment end. Specifically, two inputs are constructed for the same sample during training: 1) Full-modal input (or high-quality input) is used to generate the reference output; 2) Missing / degraded modal inputs (achieved by randomly discarding modalities, setting the mask to zero, or adding noise / occlusion, etc.) are used to generate deployment-side outputs.

[0112] Furthermore, a consistency constraint is introduced into the loss function to ensure that the prediction results or shared representations of the same sample under full-modal input paths and under input paths with missing or degraded modalities remain consistent or close, thereby reducing output bias under different combinations of input modalities. An example of joint loss is as follows: in: Task loss: Used to optimize cognitive load level / score prediction tasks. Cross-entropy loss (CE) can be used for classification scenarios, and mean squared error loss (MSE) or equivalent forms can be used for regression scenarios.

[0113] This is a consistency constraint: used to ensure that the "full modal output" and the "missing modal output" remain consistent or close on the same sample. For example, it can take any of the following forms: Distribution consistency: the probability distribution of classification outputs and Apply KL (Kullback–Leibler Divergence) divergence constraints: Numerical consistency: for regression output and Apply distance constraints: in, (· ·) denotes the Kullback–Leibler Divergence operator, used to measure the degree of difference between two probability distributions. / Indicates full-modal path output. / This indicates the output of a missing modality path. In this application, the superscript "full" indicates the output of a full modality path, and the superscript "miss" indicates the output of a path with missing modality or degraded quality; "p" represents the classification probability distribution. This indicates continuous regression output.

[0114] Auxiliary constraint terms: These provide auxiliary supervision and regularization for intermediate representations and compensation mechanisms, thereby improving the stability of shared representations, the effectiveness of missing data compensation, or the reliability of output. Auxiliary constraints do not change the definition of the main task and are considered a loss to the main task. Optional enhancements.

[0115] In some optional implementations, It may include, but is not limited to, one or more of the following: 1) Shared representation consistency / alignment constraints: These are used to ensure that the shared representations of the same sample remain consistent or aligned across all modal paths and missing / degraded paths; for example, distance constraints can be used. in This represents the shared representation obtained from the full-modal path. This represents the shared representation obtained from missing / degraded paths.

[0116] 2) Compensation information validity constraints: used to constrain compensation queries / compensation vectors or compensation items It exhibits stability under different masking strategies or aligns with the direction of output change to prevent the compensation mechanism from degenerating into noise injection; its implementation can employ distance constraints, contrast constraints, or lightweight regularization, etc.

[0117] 3) Confidence / Reliability Auxiliary Supervision: When the system output includes confidence or reliability indicators, weak supervision or consistency constraints can be established between confidence and quality scores, gating weights, or historical consistency, so that the confidence output maintains a corresponding relationship with the input quality status and the model fusion process.

[0118] The specific form is not limited, and implementation methods such as distance constraints, contrastive learning constraints, distillation constraints, or lightweight regularization are allowed; and Whether to enable or disable it, the sub-items it contains, and the weighting coefficients can all be selected by the project implementation.

[0119] Regularization terms: Used to constrain gate weights or model parameters, preventing overfitting and improving deployability. For example, they may include: Entropy regularization / sparse regularization of gating weights (encourages more stable or sparser gating). Parameter norm regularization (such as weight decay).

[0120] in These are weighting coefficients used to balance the contributions of each loss term, and can be determined through validation sets or empirical settings.

[0121] S400. Based on the modal availability mask, for modes in an unavailable state, close the corresponding modal expert branch; for modes in a deweighted state, reduce the weight of the corresponding modal expert branch. Step S400 also includes the following:

[0122] S400-1: Mask-driven expert routing.

[0123] 1) For modalities that are in an unavailable state (corresponding to a mask value of 0 or a quality score ≤ unavailable threshold), the corresponding modality expert branch is closed. That is, the modality expert branch does not participate in feature extraction within the current time window, and its output does not participate in subsequent fusion. In one embodiment, the closure operation includes: skipping the forward computation of the branch, setting its output to a zero vector, or resetting its gating weight to zero.

[0124] Here, "unusable threshold" refers to the lowest quality limit for determining whether a corresponding mode is completely unusable. When the quality score s... i When the quality score is not higher than the unavailability threshold, the corresponding modality is determined to be unavailable; when the quality score is higher than the unavailability threshold but lower than the quality threshold τ, the corresponding modality is determined to be unavailable. i When this occurs, the corresponding mode is determined to be in a quality degradation state. In embodiments using continuous value masks, the unusable threshold can be considered as 0, meaning that when the mask value is equal to 0, it is determined to be unusable.

[0125] 2) For modes in a reduced-weight state (corresponding to a mask value of the reduced-weight coefficient w) i 0 <wi <1), this modality expert branch still participates in feature extraction, and its output is calculated normally; the attenuation of its fusion contribution is achieved by multiplying the gate weights by the corresponding weighting coefficient w in subsequent S600 steps. i To achieve this, this step does not perform additional weight reduction processing on the output of this modal branch.

[0126] Among them, w i is the weighting coefficient for mode i, used to reflect the degree of quality degradation of that mode. In one embodiment, w i Take the quality score s directly after modal normalization. i In another embodiment, w i Based on quality ratings i The weighting coefficient is calculated using a preset mapping function. The smaller the value of the weighting coefficient, the more severe the modal quality degradation, and the greater the attenuation of its contribution to the fusion.

[0127] It should be noted that the processing of the reduced weight state is only achieved through the mask constraint and renormalization in step S600. The "reduction" in step S400 describes the expected effect of the final fusion contribution of the modality expert branch, and does not involve two independent weight decays in the feature extraction stage or the output stage.

[0128] S400-2: Efficiency optimization.

[0129] In one embodiment, when the mask value corresponding to a modality is 0, the closing operation further includes skipping the forward computation of that modality branch. Since the expert branches corresponding to unavailable modalities are closed, only available expert branches and shared representation branches are computed within the current time window, reducing invalid computation and lowering inference latency. This processing method has significant effects in online evaluation scenarios, especially under deployment conditions where edge computing resources are limited.

[0130] In one embodiment, for a shared representation branch, when at least one input modality is available, the mask of the shared representation branch is set to 1, ensuring that it always participates in gating fusion. When all input modalities are unavailable, gating fusion and cognitive load assessment for the current time window are not performed, and the process proceeds to the subsequent S800 step.

[0131] The following are some exemplary scenarios for illustration.

[0132] Example 1: In an autonomous driving takeover scenario, if the camera is occluded or eye-tracking data is lost, the human-side behavior / visual modality is determined to be unavailable. The system disables the visual expert branch, and subsequent inference is completed by expert branches based on human-side physiological modality, machine-side state signals, and human-machine interaction behavior signals, as well as a shared representation branch.

[0133] Example 2: In complex monitoring scenarios, the detachment of physiological electrodes leads to a decrease in the quality of human physiological signals, but they are not completely lost. The human physiological modality is judged to be in a deweighted state. Instead of shutting down the physiological expert branch, the system multiplies its weight by a deweighting coefficient during subsequent gating fusion to reduce its contribution to the fusion result.

[0134] Example 3: Multimodal inputs are complete, but some signals have low quality, such as a high proportion of physiological artifacts, incomplete interface event logs, and telemetry data packet loss. The system determines the corresponding modality to be downweighted or unavailable based on the quality score, and performs corresponding shutdown or downweighting processing based on the mask.

[0135] S500: When there are unavailable modes or downweighted modes that meet preset compensation trigger conditions, the shared representation branch generates a compensation query, and extracts information from the features of the remaining available modes based on the compensation query to update the global representation of the available modes. Step S500 also includes the following:

[0136] S500-1: Triggering condition.

[0137] In one embodiment, when any modality is detected to be in an unavailable state (mask value of 0), or in a deweighted state and the mask value is lower than the compensation trigger threshold, When the mask value is 0, the missing data compensation module is triggered. By default, the compensation module is triggered when the mask value is 0.

[0138] In this application, the "preset compensation trigger condition" refers to: the modality being in an unavailable state, or the modality being in a downweighted state and its availability mask value being lower than a preset compensation trigger threshold. The compensation trigger threshold can be set individually or uniformly according to the modality type, device performance, or application scenario. By default, missing compensation is only triggered when the mask value is 0.

[0139] S500-2: Generation of compensation query.

[0140] In this application, "compensation information" is a general term for all types of intermediate information in the missing modality compensation mechanism; "compensation query" refers to the query vector q generated by the shared representation. comp The term "compensation term" refers to the update amount Δ obtained through guided aggregation, which is used to update the global representation of available modalities.

[0141] When missing compensation is triggered, for the k-th missing or low-quality modality, a compensation query for that modality is generated by the shared representation branch: Where k represents the type of missing or low-quality modality, which can correspond to human physiological, human behavior, machine state, human-computer interaction behavior or environment / task context modality; Indicates cross-modal shared representation; This represents a trainable mapping network configured based on the k-th missing modality. This represents a compensation query generated for the k-th missing modality.

[0142] It is important to emphasize that the compensation query is compensation information / compensation vector generated from shared representations, used to guide the aggregation of available modal features. It is not equivalent to the missing modal data collected by real sensors, nor is it intended to reconstruct the original signal of the missing modality. The information source of the compensation query is the shared semantics of the acquired available modal data, not the estimated missing modal data itself.

[0143] In one embodiment, human-side physiological modality and human-side behavioral / visual modality are used as examples: When a physiological modality is missing, a physiological missing compensation query is generated from the shared representation: When human-side behavioral / visual modalities are missing, a visual missingness compensation query is generated from the shared representation: in, This is a physiological deficiency compensation mapping network. This is a visual missing modality compensation mapping network. Different missing modality types may correspond to different or shared mapping network parameters. The compensation query is used to guide the aggregation of remaining available modality sequences and is not equivalent to the reconstruction of the original signal of the missing modality.

[0144] S500-3: Guided aggregation based on cross-attention.

[0145] like Figure 6 As shown, in the missing data compensation module, the left-hand input includes: available visual tokens and their global representations. Physiological tokens and their global representation can be used. Shared representation (From the shared expert branch); and modal availability mask. Shared representations are used to generate compensating queries for visual modalities. Compensation queries oriented towards physiological modalities The middle section is a guided cross-attention update mechanism: A) When physiological modalities are missing At that time, with For the query vector Q, with visual sequence features Given key vector K and value vector V, a compensation term is calculated using cross-attention to update the visual global representation: .

[0146] B) When visual modality is missing ( When =0), Let Q be the query vector, with physiological sequence features. Given a key vector K and a value vector V, a compensation term is calculated using cross-attention to update the physiological global representation: Then, fusion and aggregation are performed to obtain the fused representation. When a single modality is available, it degenerates into a fusion approach of "available modality representation + shared representation". Finally, the fused representation is input to the output header, and the output is the cognitive load level or score and confidence level.

[0147] It should be noted that during online inference, when m i When the value is 0, the missing expert branch is closed, only available experts and shared experts are calculated, and the gating weights are renormalized, thereby reducing computing power and latency.

[0148] It should be noted that this step is only executed when the compensation triggering condition described in step S500-1 is met; when the missing compensation is not triggered, no compensation query is generated, nor is the global representation of each available modality updated. The original global representation output by each modality expert branch remains unchanged and is directly used for subsequent fusion.

[0149] In one embodiment, information related to the missing modality is extracted from the sequence features of the remaining available modalities through a cross-attention mechanism (which refers to an attention calculation method in which the query vector Q comes from one modality or branch, while the key vector K and value vector V come from another modality or branch, used to achieve cross-modal information interaction and guided aggregation), to generate compensation terms and update the global representation of the available modalities.

[0150] Taking the absence of human physiological modalities and the availability of human behavioral / visual modalities as an example: Search for physiological loss compensation For the query vector Q, with visual sequence features Given a key vector K and a value vector V, the visual-side compensation term is calculated using cross-attention. : Here, Δ represents the compensation term, which is the latent information related to the missing modality extracted from the features of the available modality sequence through guided aggregation. The subscripts vis and phys indicate that the compensation term is used to update the global representation of the visual modality and the global representation of the physiological modality, respectively.

[0151] Its attention operator can be expressed as: in, The feature dimension of the key vector is given when the input is... ∈ hour, That is, equal to d; when the input is X phys ∈ The same principle applies at other times.

[0152] Then, the compensation term is added to the global representation of the visual modality, vis_cls, and layer normalization is performed to obtain the updated global representation: LN (Layer Normalization) is used to normalize feature vectors within a layer to stabilize the training and inference process.

[0153] Taking the absence of human-side behavioral / visual modalities and the availability of human-side physiological modalities as an example: Visual loss compensation query Let Q be the query vector, with physiological sequence features. Given a key vector K and a value vector V, the physiological side compensation term is calculated using cross-attention. : CrossAttn(Q, K, V) represents the cross-attention operator, which is calculated in the same way as the attention operator Attn(Q, K, V) below. The difference is that in cross-attention, the query vector Q and the key-value pair (K, V) come from different modalities or branches.

[0154] Then update the global representation of the physiological modality: It should be noted that the aforementioned cross-attention is only one preferred implementation of guided aggregation. The guided aggregation mechanism can also be implemented using other methods such as relevance matching, alignment mapping, or confidence-based weighted aggregation; this invention does not limit the specific implementation form.

[0155] S500-4: Extension of multimodal missing scenarios.

[0156] In one embodiment, when there are multiple unavailable or downgraded modes, a corresponding compensation query is generated for each missing or low-quality mode, and information is extracted from the remaining available modes to update the global representation of the corresponding available mode.

[0157] In one embodiment, when the missing modality is a machine-side state, human-computer interaction behavior, or environment / task context modality, a corresponding type of compensation query is generated from the shared representation in the same manner, and the remaining available modalities are guided to complete the aggregation update.

[0158] In one embodiment, when a deweighted mode exists and its availability mask value is not lower than the compensation trigger threshold, no missing compensation is triggered for that deweighted mode; instead, deweighting is performed only according to its continuous availability mask value during the gating fusion stage. When an unavailable mode exists, or a deweighted mode exists and its mask value is lower than the compensation trigger threshold, missing compensation is triggered for that unavailable or low-quality mode. By default, i.e., when no compensation trigger threshold is set, missing compensation is triggered only when the mask value is 0.

[0159] By default, τ is not set. comp In this case, missing compensation is triggered only when the mask value is 0; compensation is not triggered in the deweighted mode.

[0160] The following are some exemplary scenarios for illustration.

[0161] Example 1: In an autonomous driving takeover scenario, eye-tracking data is lost (visual modality is unavailable). The shared representation branch generates a visual loss compensation query based on available physiological signals, machine-side state signals, and interaction behavior signals. Using this compensation query as the query vector, cross-attention calculation is performed using physiological sequence features as keys and values. Potential discriminative information related to the visual modality loss is extracted from the physiological sequence, and the global representation of the physiological modality is updated.

[0162] Example 2: In an industrial human-machine collaboration scenario, electrode detachment leads to the unavailability of physiological modalities. A physiological deficiency compensation query is generated by the shared representation branch. Cross-attention calculation is performed using this query as Q and visual sequence features as K and V. Information related to cognitive load discrimination is extracted from visual, machine-side state, and interaction behavior sequences to update the global representation of available modalities.

[0163] Example 3: In teleoperation scenarios, the environment / task context signal is incomplete. The shared representation branch generates an environment / task context missing compensation query, which guides the aggregation and update of the remaining available modal sequences to compensate for the lack of information caused by the missing environment context.

[0164] S600: Based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain the fused weights. Step S600 also includes the following:

[0165] S600-1: The original weights output by the gated network.

[0166] In one embodiment, the input feature h of the gating network is: Here, Concat(·) represents the vector concatenation operation, which connects the vectors end-to-end along the dimensional direction to create a higher-dimensional vector; z iLet i be the output representation of the i-th modal expert branch. Indicates the total number of input modes. For the output of the shared representation branch, s is the quality score vector and m is the modal availability mask vector.

[0167] The gating network outputs the original weight vector g of each fusion branch: in, This represents the input characteristics of the gating network; This represents the weight matrix of the gated network; Represents the bias vector; This represents the original gating weight vector for each fusion branch. If... The total number of branches participating in gating fusion is Then it can be made , Gated network output In settings In the implementation of one modality expert branch and one shared representation branch, .

[0168] S600-2: Mask Constraints and Renormalization In one embodiment, the original weights are constrained and renormalized based on a modal availability mask. For the e-th fusion branch: 1) When the corresponding mode is in an unavailable state (mask value) When the weight of the branch is 0, reset the original weight of the branch to zero. 2) When the corresponding mode is in a reduced-weight state (mask value) = ,0< When <1), multiply the original weight of the branch by the corresponding weight reduction coefficient. ; 3) When the corresponding mode is in an available state (mask value) When =1), the original weights remain unchanged.

[0169] Then, the weights of each constrained branch are divided by the sum of all constrained weights to complete the renormalization: in, Indicates the index of the current fusion branch. This represents the branch index during the renormalization summation process; and They represent the first The original gating weights and masked gating weights of each branch; and These represent the availability masks for the corresponding fusion branches, and E is the total number of branches participating in gating fusion.

[0170] When the above unified processing is applied to all modal expert branches and shared representation branches, The condition >0 must be met. When all input modalities are unavailable, gating fusion and cognitive load assessment for the current time window are not performed, and the process proceeds to the subsequent S800 steps.

[0171] In one embodiment, for modality expert branches, the branch mask is taken as the availability mask of the corresponding modality; for shared representation branches, when at least one input modality is available, let This allows shared representation branches to participate in fusion; when all input modalities are unavailable, let It does not perform gating fusion and cognitive load assessment for the current time window, and outputs state information that does not meet the assessment conditions.

[0172] When using a binary availability mask When using a continuous availability mask, Modalities that are degraded in quality but still usable participate in fusion according to their availability.

[0173] In one embodiment, when all modes are fully available and all branch masks are 1, = The gating weights remain unchanged.

[0174] The gating network closes (sets) unavailable branches to zero, reduces the weight of deweighted branches, and reallocates weights within the set of currently available branches.

[0175] In one embodiment, the model includes three fusion branches: a visual expert branch, a physiological expert branch, and a shared representation branch. The original weights output by the gating network are g=[0.5, 0.3, 0.2] (corresponding to visual, physiological, and shared representations, respectively).

[0176] If visual modality is unavailable (m vis =0), physiological modalities are available (m phys =1), shared representation branches are available (m shared If the sum is 1), then the constrained weights are [0, 0.3, 0.2], and after renormalization, the fusion weights g'=[0, 0.6, 0.4].

[0177] If visual modality is reduced in weight (m) vis =0.3), physiological modalities are available (m phys =1), shared representation branches are available (m shared=1), then the constrained weights are [0.15, 0.3, 0.2], and after renormalization, the fusion weights g'=[0.23, 0.46,0.31].

[0178] S700. The outputs of each modality expert branch, the output of the shared representation branch, and the updated global representation are fused according to the fusion weights to obtain a fused representation. The fused representation is input into the output head to output the cognitive load assessment result and reliability index of the current time window.

[0179] In one embodiment, during the computation of the fused representation, the modal expert branches corresponding to each modality expert branch... Determine as follows: 1) When the missing compensation in step S500 is triggered in this time window, for the available modalities that have been compensated and updated, Take the global representation of the modality after compensation and update (e.g., vis) cls *、phys cls *); For modalities that have not been compensated and updated, Take the original global representation of the output of this modal branch.

[0180] 2) When missing compensation is not triggered in this time window, Directly retrieve the original global representation of each modal branch output (e.g., vis) cls phys cls ).

[0181] In both of the above scenarios, the shared representation branch corresponds to All .

[0182] In one embodiment, after missing compensation in step S500 is performed when unavailable modalities exist, the global representation of each available modality expert branch has been updated. The fused representation is as follows: Where e is the fusion branch index, with a value range from 1 to E; E is the total number of branches participating in gating fusion, which is equal to the sum of the number of modal expert branches M and the number of shared representation branches 1. For fusion representation, Let be the fusion weight of the e-th fusion branch after masking and renormalization. Let be the output representation of the e-th fusion branch. For the i-th modal expert branch, This can be the global representation of the modal branch, including the compensated and updated global representation; if compensation is not triggered, the original global representation is used; for shared representation branches, .

[0183] In one embodiment, when missing compensation is not triggered, The original global representation of this modal branch (e.g., vis) cls phys cls (etc.) and .

[0184] In one embodiment, the fusion representation After the input / output headers, the cognitive load output header outputs either the cognitive load level or a continuous load value. For classification tasks, the cognitive load output header outputs the probability distribution of each cognitive load category, taking the category with the highest probability as the cognitive load level; for regression tasks, it outputs continuous cognitive load scores.

[0185] Simultaneously, the reliability output head outputs a reliability index corresponding to the cognitive load assessment result based on the fused representation. In one embodiment, the reliability index is a value between 0 and 1, representing the reliability of the current assessment result. The reliability index can be calculated based on factors such as the distribution entropy of the gating weights, the number of available modes, the mean of the quality score, or the confidence level of the model prediction.

[0186] In one embodiment, the cognitive load assessment results and reliability indicators are displayed on a collaborative assessment platform. The operator acquisition unit, device / autonomous entity interface unit, edge computing unit, and collaborative assessment platform interact with each other via a standard communication protocol. The collaborative assessment platform provides real-time curves, dashboards, alarm prompts, and event annotation functions, and supports process data recording and export.

[0187] In one embodiment, the cognitive load assessment results output during the online assessment phase can be applied to human-machine tightly coupled task scenarios such as autonomous driving takeover, collaborative robot assembly, complex monitoring, flight control, teleoperation, and digital surgery, providing decision-making basis for operator status monitoring, task allocation, human-machine collaboration adaptation, and system alarms.

[0188] In one embodiment, the quality score, availability mask, gating weight, result reliability index, and modal contribution weight can be recorded as process analysis information for subsequent data playback, offline analysis, and system optimization.

[0189] S800: When all modes are in an unavailable state, do not perform fusion and evaluation for the current time window, and output the state information, reasons and reliability indicators for not meeting the evaluation conditions.

[0190] In one embodiment, when the quality scores of all input modalities are not higher than the corresponding unavailability threshold, i.e., the availability mask of all modalities is 0, the input conditions for performing cognitive load assessment are not met. In this case, gating fusion and cognitive load assessment for the current time window are not performed.

[0191] In one embodiment, the system outputs status information, reasons, and reliability indicators for situations where evaluation conditions are not met. The status information includes an identifier indicating that the evaluation is unavailable; the reasons include which specific modalities are unavailable, the quality score of each modality, and the corresponding reasons for unavailability; the reliability indicators are used to indicate the degree to which the current evaluation result is unavailable or unreliable, for example, setting the reliability indicator to 0, or outputting a preset "unavailable" identifier and its corresponding confidence value. For example, when the human physiological modality is unavailable due to electrode detachment, the human behavioral modality is unavailable due to camera obstruction, or the machine-side status signal is unavailable due to telemetry link interruption, the system outputs "The current time window does not meet the evaluation conditions. Reasons: physiological signal electrode detachment, visual signal camera obstruction, machine-side telemetry link interruption. Reliability indicator: 0 (unavailable)."

[0192] This can further trigger fault alerts and log entries for data links, sensors, or device interfaces. The system can generate alarm signals to prompt operators or maintenance personnel to check the status of relevant sensors, links, and device interfaces.

[0193] In one embodiment, the system can detect faults through self-test mechanisms such as Power-On Test (BIT), Periodic BIT, or Maintenance BIT, locate faulty sensors or links, and trigger alarm linkage. Key modules can be configured with redundancy / backup and isolation designs to improve the system's reliability and maintainability.

[0194] In one embodiment, when any available modality reappears within a subsequent time window, the system automatically resumes the cognitive load assessment function and continues to output assessment results.

[0195] like Figure 7 As shown, the robust evaluation mechanism of this invention consists of two processes: top row and bottom row.

[0196] The top row shows the standard processing flow under the condition of multimodal integrity: multimodal acquisition (such as EEG, eye tracking, video, etc.) → time synchronization and preprocessing → expert + shared representation gating fusion → cognitive load output (level or score).

[0197] The bottom row illustrates the robust handling process at the deployment end in case of missing modalities: Deployment data (modalities may be missing or degraded) → Quality inspection / mask generation (modal availability mask) ∈[0,1], where =1 indicates availability, 0< <1 indicates a lower weighting. =0 indicates unavailable; m∈{0,1} is a special case) → Missing data compensation module or weighted fusion module → Robust inference outputs cognitive load. This process shares the same model structure as the previous process, but automatically triggers missing data compensation and gating renormalization when a modality is missing, thereby ensuring the continuity and reliability of the output.

[0198] The two rows of processes together constitute the adaptive evaluation mechanism of this invention under multimodal integrity and missing conditions.

[0199] The following are some specific examples to illustrate this.

[0200] Example 1: Autonomous Driving Takeover Scenario In autonomous driving takeover or intelligent cockpit tasks, human-side physiological modalities are available, but human-side behavioral / visual modalities are missing, such as due to camera occlusion, lost eye movements, or decreased confidence in facial key points. The system disables or downweights the corresponding behavioral visual expert branches based on availability masks, generates missing robust compensation queries based on shared representations, guides the update of the global representation using available physiological sequences and machine-side state sequences, and outputs cognitive load levels or continuous values ​​and reliability indicators.

[0201] Example 2: Complex monitoring or industrial human-machine collaboration scenarios In complex monitoring or industrial human-machine collaboration tasks, human-side behavioral modalities, machine-side state modalities, and interactive behavioral modalities are available, but human-side physiological modalities are missing, for example, due to electrode detachment, interruption of physiological acquisition links, or insufficient physiological signal quality. The system disables or downweights physiological expert branches based on availability masks, generates physiological gap compensation queries based on shared representations, guides the updating of global representations of human-side behavioral, machine-side state, and interactive behavioral sequences, and outputs cognitive load assessment results and reliability indicators.

[0202] Example 3: Flight control or remote operation scenario In flight control, teleoperation, or digital surgery tasks, multimodal inputs are complete, but some signals have low quality, such as high proportion of physiological artifacts, incomplete interface event logs, telemetry data packet loss, network latency jitter, or missing task context. The system deweights or disables corresponding modalities based on quality scores and renormalizes gating weights according to availability masks. Simultaneously, it records quality scores, availability masks, gating weights, and reliability metrics for process analysis and collaborative adaptation.

[0203] Example 4: Full-modal unavailability scenario If all input modalities are unavailable, the system will not perform gating fusion and cognitive load assessment for the current time window, and will output the status, cause and reliability indicators of the failure to meet the assessment conditions. At the same time, it will trigger data link, sensor or device interface fault prompts and log recording.

[0204] This invention also provides a robust cognitive load assessment system for multimodal loss in tightly coupled human-machine interactions, comprising: The operator-side data acquisition unit is deployed in vehicle cockpits, flight simulators, aircraft training platforms, special equipment control consoles, collaborative robot workstations, complex monitoring platforms, or digital surgical environments to collect human physiological and behavioral signals from the operator. The subject-side device adopts a modular structure, including a computing unit module (motherboard, CPU / GPU, memory, solid-state drive / hard drive, touchscreen, etc.), a sensor interface module (eye tracker interface, EEG acquisition interface, camera interface, etc.), a power supply and grounding module, and an external communication module (wired / wireless). The subject-side device is equipped with a removable storage medium for data export. "Subject-side" refers to the device deployed on the operator's side, carrying the operator-side data acquisition unit and edge computing unit; "Main-side" refers to the evaluation end carrying the collaborative evaluation platform for monitoring, display, and management.

[0205] Device / Autonomous Interface Unit: Used to access machine-side status signals, human-machine interaction behavior signals, and environment / task context signals.

[0206] Edge computing unit: used for multimodal data caching, time synchronization, quality assessment and preliminary processing.

[0207] Collaborative assessment platform: Used for process data monitoring, cognitive load assessment, reliability output, result display, and optional human-machine collaboration adaptation. The main testing software is implemented as a desktop application, providing functions such as data monitoring, waveform / list display, online assessment, and alarm prompts.

[0208] In one embodiment, the participant converts multi-source data into a unified encapsulation format (including timestamps, modality identifiers, data payloads, and quality fields) and sends it to the host via a standard communication protocol. The host uses multi-threading to receive and update interface elements; high-frequency data is refreshed as waveforms, and low-frequency data is refreshed as lists. The participant can write the raw data and evaluation results to local storage; the host can save process data and event annotations, supporting playback and export. Optionally, sensitive data can be anonymized or encrypted, and access can be restricted through access control.

[0209] In one embodiment, the system sets reliability and maintainability indicators and implements self-testing and alarm linkage at the hardware and software levels, including: power-on BIT, periodic BIT and maintenance BIT; redundancy / backup and isolation design of key modules; fault detection, location and alarm prompts; and meeting electromagnetic compatibility immunity and environmental adaptability requirements.

[0210] An electronic device provided in this application includes a memory and a processor; the memory is a computer-readable storage medium used to store computer programs; the processor is used to implement the above-mentioned human-machine tightly coupled multimodal missing robust cognitive load assessment method when executing the computer program.

[0211] In this embodiment, the beneficial effects of electronic devices and computer-readable storage media are similar to those of the above-described robust cognitive load assessment method for multimodal missing data oriented towards tight human-machine coupling, and will not be repeated here.

[0212] The present invention describes electronic devices that can serve as servers or clients of this application, which are examples of hardware devices applicable to various aspects of this application. Electronic devices are intended to represent various forms of digital electronic computer devices. Electronic devices can also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0213] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0214] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a read-only memory (ROM) or a random access memory (RAM), etc. The integrated units described above can be implemented in hardware or as software functional units.

[0215] In summary, compared with existing technologies, it has the following beneficial effects: 1. By jointly determining modal quality scores and availability masks, the available, deweighted, and unavailable states of each modal data are accurately identified at the time window level. Based on the mask, the expert branches of untrusted modalities are adaptively closed or deweighted, and the gating weights are constrained and renormalized. This effectively reduces the impact of unavailable or low-trust modalities on the fusion representation and improves the stability of fusion inference under modal missing or quality degradation conditions. 2. By generating a missing robust compensation query through cross-modal shared representation and performing guided aggregation, potential information related to cognitive load discrimination is extracted from the remaining available modalities without reconstructing the original signal of the missing modality, and the global representation is updated, thus maintaining the continuity and accuracy of the evaluation output under the conditions of missing modality or quality degradation. 3. By enhancing modal random missing features, enhancing modal quality degradation, and training with consistency constraints between full modal paths and missing modal paths, the output deviation between full modal conditions in the training phase and missing modal conditions in the deployment phase is reduced. At the same time, quality scores, availability masks, gating weights, and reliability indicators can be recorded as process analysis information. Modal expert branches can be enabled, disabled, or downgraded according to actual data collection and configuration. This is suitable for human-machine tightly coupled task scenarios such as autonomous driving takeover, collaborative robot assembly, complex monitoring, flight control, teleoperation, and digital surgery.

[0216] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust cognitive load assessment method for multimodal missing modes oriented towards tight human-machine coupling, characterized in that, include: Acquire multiple modal data of operators within the current time window, and perform time alignment and preprocessing on each modal data; Calculate the quality score for each modality of data, and generate a modality availability mask based on the quality score; the modality availability mask is used to indicate whether the corresponding modality is available, downweighted, or unavailable. The preprocessed multimodal data is input into the cognitive load assessment model; the cognitive load assessment model includes multiple modality expert branches, a shared representation branch, a gating network, and an output head; wherein, the cognitive load assessment model is a model trained by consistency constraints between the full modality input path and the input path with missing modality or quality degradation. Based on the modal availability mask, for unavailable modes, the corresponding modal expert branches are closed; for deweighted modes, the weights of the corresponding modal expert branches are reduced. When there are unavailable modes or reduced-weight modes that meet the preset compensation triggering conditions, the shared representation branch generates a compensation query and extracts information from the features of the remaining available modes based on the compensation query to update the global representation of the available modes; Based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain the fused weights; The outputs of each modality expert branch, the output of the shared representation branch, and the updated global representation are fused according to the fusion weights to obtain a fused representation; the fused representation is input into the output head to output the cognitive load assessment result and reliability index of the current time window; When all modalities are unavailable, the fusion and evaluation of the current time window are not performed, and the status information, reasons and reliability indicators of the unmet evaluation conditions are output.

2. The method according to claim 1, characterized in that, The multiple modal data include at least two of the following: human physiological signals, human behavioral signals, machine state signals, human-machine interaction behavioral signals, and environmental / task context signals.

3. The method according to claim 1, characterized in that, The modality expert branches extract modality features from the corresponding modality data and aggregate the modality features into a global representation of the corresponding modality; the shared representation branch generates a cross-modality shared representation based on the modality features of the currently available modalities and generates the compensation query based on the cross-modality shared representation.

4. The method according to claim 1, characterized in that, The modal availability mask is a continuous value mask with a value range of [0,1]; where: A mask value of 1 indicates that the corresponding mode is available. A mask value between 0 and 1 indicates that the corresponding mode is in a deweighted state, and the degree of deweighting is related to the size of the mask value; A mask value of 0 indicates that the corresponding mode is unavailable.

5. The method according to claim 1, characterized in that, The step of extracting information from the features of the remaining available modalities based on the compensation query and updating the global representation of the available modalities includes: Using the compensation query as the query vector and the sequence features of the remaining available modalities as the key vector and value vector, the compensation term is calculated through cross-attention. The compensation term is added to the global representation of the available modes and then layer normalized to obtain the updated global representation.

6. The method according to claim 1, characterized in that, The process involves constraining and renormalizing the original weights of each branch output from the gated network based on the modal availability mask to obtain fused weights, including: For a mode that is in an unavailable state, set the mask value of its corresponding branch to 0, so that the original weight of the branch is set to zero. For a mode in a reduced-weight state, its mask value is used as the reduction coefficient, and the original weight of the branch is multiplied by the mask value. The fusion weight is obtained by dividing the weight of each constrained branch by the sum of all constrained weights.

7. The method according to claim 1, characterized in that, The cognitive load assessment model was trained in the following manner: For the same sample, construct a full-modal input path and a modality-degraded or quality-degraded input path; the missing or quality-degraded input path is obtained by simulation through at least one of random modality discarding, masking to zero, adding noise, or occlusion; The inputs of the two paths are fed into the cognitive load assessment model to be trained, and their respective prediction results are obtained. The task loss is calculated based on the prediction results and sample labels, and consistency constraints are applied to the prediction results or shared representations of the two paths. The consistency constraints include any one or more of the following: distribution consistency constraints, numerical consistency constraints, and representation consistency constraints. The model parameters of the modal expert branch, shared representation branch, gating network, and output head are updated based on the task loss and the consistency constraint, so that the prediction results or shared representations of the same sample under the full modal input path and the input path with missing modalities or degraded quality are consistent or close.

8. A robust cognitive load assessment system for multimodal missing modes oriented towards tight human-machine coupling, characterized in that, include: The operator acquisition unit is used to acquire human physiological and behavioral signals of the operator within the current time window. The device / self-entity interface unit is used to access machine-side status signals, human-machine interaction behavior signals, and environmental / task context signals. The edge computing unit is used to cache, synchronize, and preprocess data of each modality, calculate the quality score of each modality, and generate a modality availability mask based on the quality score; the modality availability mask is used to indicate whether the corresponding modality is available, downweighted, or unavailable. Collaborative assessment platform, used for: The preprocessed multimodal data is input into the cognitive load assessment model; the cognitive load assessment model includes multiple modality expert branches, a shared representation branch, a gating network, and an output head; wherein, the cognitive load assessment model is a model trained by consistency constraints between the full modality input path and the input path with missing modality or quality degradation. Based on the modal availability mask, for unavailable modes, the corresponding modal expert branches are closed; for deweighted modes, the weights of the corresponding modal expert branches are reduced. When there are unavailable modes or reduced-weight modes that meet the preset compensation triggering conditions, the shared representation branch generates a compensation query and extracts information from the features of the remaining available modes based on the compensation query to update the global representation of the available modes; Based on the original weights of each branch output by the gated network, the original weights are constrained and renormalized according to the modal availability mask to obtain the fused weights; The outputs of each modality expert branch, the output of the shared representation branch, and the updated global representation are fused according to the fusion weights to obtain a fused representation; the fused representation is input into the output head to output the cognitive load assessment result and reliability index of the current time window; When all modalities are unavailable, the fusion and evaluation of the current time window are not performed, and the status information, reasons and reliability indicators of the unmet evaluation conditions are output.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images