An industrial HMI anti-shake reconstruction and data flow scheduling method based on cognitive load

Abstract

The application discloses an industrial HMI anti-shake reconstruction and data flow scheduling method based on cognitive load and belongs to the technical field of industrial human-computer interaction, and comprises the following steps: constructing a continuous cognitive load correlation index after extracting features from EEG data and eye movement data; based on the cognitive load correlation index, adjusting the industrial control state through a double-threshold hysteresis state machine, and when the industrial control state is adjusted to an emergency state, based on a ternary mapping relationship among interface elements, data flow identifiers and graphic resource objects, the following is synchronously realized through a control strategy bound to the same interface element: retaining key data flow, reducing non-key data flow, maintaining a core interface layer corresponding to the key data flow, weakening or folding an auxiliary interface layer corresponding to the non-key data flow, so that the industrial HMI can not only reduce non-key interface interference in the high-cognitive-load scene of an operator, but also synchronously reduce the influence of non-key data and graphic resource reconstruction on a system processing link.

Description

Technical Field

[0001] This invention belongs to the field of industrial human-machine interaction technology, specifically relating to an industrial HMI anti-shake reconstruction and data flow scheduling method based on cognitive load. Background Technology

[0002] In safety-critical industrial scenarios such as large power grid dispatch centers, complex CNC machine tools, and unmanned system monitoring platforms, operators typically need to simultaneously monitor numerous sensor readings, trend curves, alarm information, task status, and control inputs. Industrial Human-Machine Interfaces (HMIs) are the primary interactive entry points for operators to obtain system status, identify abnormal operating conditions, and issue control commands. With the increasing automation level and the growing number of sensors in industrial systems, the information density carried by HMIs continues to rise. When emergency situations occur, alarm information, telemetry data, and control prompts flood the interface in a short period, easily increasing the operator's cognitive load and leading to missed critical alarm readings, delayed operational responses, or misoperations.

[0003] Cognitive load refers to the mental resources consumed by an operator when performing perception, judgment, decision-making, and operational tasks. Electroencephalography (EEG) and eye-tracking data are used to estimate an operator's working memory load, attentional allocation status, and visual search stress. Constructing continuous cognitive load-related indices based on this type of physiological or behavioral data can provide an engineering basis for the dynamic adjustment of HMIs. Meanwhile, in industrial HMI systems, the objects displayed on the interface typically correspond to continuously refreshed telemetry data streams, alarm data streams, and auxiliary status data streams; curves, icons, textures, vertex buffers, and shaders in the graphical interface also consume CPU, network, queue, video memory, and GPU rendering resources. Therefore, emergency simplification of industrial HMIs should not be understood merely as displaying fewer controls, but also involves data stream scheduling and graphical resource lifecycle management.

[0004] Existing technologies have proposed various solutions related to cognitive load, HMI adjustment, and interface information scheduling. Existing HMI adjustment solutions for changes in user cognitive load typically assess the user's current cognitive or task load by collecting data such as user status, device operating status, environmental perception information, and interaction behavior. Based on the assessment results, different processing is applied to the display priority, prompting methods, or interaction requests of interface information, and output strategies for visual, auditory, or tactile senses are dynamically adjusted to reduce unnecessary information interference under high cognitive load conditions.

[0005] However, existing solutions primarily focus on adjusting interface information display and interaction strategies, neglecting the management of relationships between interface elements, data flows, and graphical resources in industrial HMIs, as well as the coordinated control of data flow scheduling and graphical rendering under varying cognitive loads. They lack a unified mapping mechanism for managing the relationships between interface elements, data flow identifiers, and graphical resource objects, as well as a unified state triggering and consistency update mechanism. Furthermore, under high cognitive load conditions, there is still room for improvement in reducing the transmission and processing overhead of non-critical industrial data and avoiding frequent destruction and reconstruction of auxiliary interface graphical resources.

[0006] For data flow scheduling and graphics rendering resource management, there are numerous existing solutions addressing network traffic control, monitoring system information display, graphics processing pipelines, and graphics resource reuse. However, these solutions typically address network communication, control room display, graphics processing, or cache management separately, failing to consider the operator's cognitive load status as a unified trigger source, and neglecting to integrate critical / non-critical data flow reduction with HMI graphics resource preservation and reconstruction. In existing simplified HMI solutions, even if non-critical controls are hidden, their underlying non-critical telemetry data may still continuously enter the user-mode receiving process, UI thread, or rendering thread, causing queue backlog, CPU usage, and rendering refresh burden. On the other hand, if interface switching is performed by destroying controls, releasing textures, or rebuilding vertex buffers, GPU resource reconstruction and video memory reallocation overhead can easily occur when restoring to normal state, leading to frame time spikes and interface stuttering during the recovery phase.

[0007] Furthermore, existing adaptive HMIs or information scheduling schemes typically do not explicitly establish a mapping between "interface element priority—data flow identifier—graphics resource object." Therefore, even if the interface layer has weakened or hidden certain auxiliary elements, their corresponding non-critical data flows may still continue to enter the receiving queue or rendering thread in full. Conversely, individual data rate limiting or discarding strategies are difficult to synchronize with interface reconstruction and graphics resource lifecycle control. This design, lacking mapping coordination, is easily broken down into independent display control, network rate limiting, or graphics cache management measures, making it difficult to form a unified closed-loop control for safety-critical industrial HMIs.

[0008] Furthermore, in schemes lacking a ternary mapping relationship, even if interface display control, data flow reduction, and graphics resource management exist separately, mismatches may occur due to differences in state sources, policy versions, or update sequences. For example, auxiliary interface elements may be weakened, but their non-critical data flows may still enter the user-mode queue in full, or non-critical data flows may be reduced, but the corresponding auxiliary interface may still be displayed in its complete state.

[0009] In summary, existing technologies have disclosed technologies related to cognitive load assessment, cockpit or vehicle HMI adjustment, human-machine function allocation, interface information scheduling, eye-tracking or multi-physiological indicator load determination, network data processing, and graphics rendering resource management. However, a unified closed-loop solution for safety-critical industrial HMIs has yet to be found that can collaboratively combine multimodal cognitive load correlation indices, dual-threshold hysteresis stabilization state machines, critical data flow retention, non-critical data flow reduction, and GPU / memory-retaining interface reconstruction. This would simultaneously reduce visual information burden and system processing burden under high operator cognitive load conditions, and minimize stuttering caused by interface resource reconstruction when returning to normal operation. Summary of the Invention

[0010] In view of the above, this invention aims to solve the problems of frequent interface switching, continuous occupation of processing resources by non-critical data, recovery delay caused by ordinary hiding or destroying of controls, and large graphics resource reconstruction overhead in the existing industrial HMI under high cognitive load scenarios. It proposes an industrial HMI anti-jitter reconstruction and data flow scheduling method based on cognitive load. It constructs a continuous cognitive load correlation index (CCCI) through EEG features and eye-tracking features, suppresses interface jitter caused by critical fluctuations through a hysteresis state machine, retains key interface elements and key data flows in emergency situations, reduces non-critical data flows, and reduces resource reconstruction overhead during auxiliary interface recovery through GPU / video memory retention interface reconstruction.

[0011] To achieve the above-mentioned objectives, an embodiment provides an industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load, comprising the following steps: After extracting features from EEG and eye-tracking data, a continuous cognitive load-related index was constructed. Based on the cognitive load-related index, the industrial control state is adjusted through a dual-threshold hysteresis state machine. When the industrial control state is adjusted to an emergency state, based on the ternary mapping relationship between interface elements, data flow identifiers, and graphical resource objects, the data flow scheduling and industrial HMI reconstruction are synchronously realized through the control strategy bound to the same interface element. Specifically, this includes: retaining key data flows, reducing non-key data flows, maintaining the core interface layer corresponding to key data flows, and weakening or folding the auxiliary interface layer corresponding to non-key data flows.

[0012] Preferably, a continuous cognitive load-related index is constructed after extracting features from EEG data and eye-tracking data, including: Feature extraction includes: extraction based on EEG data Frequency band energy ratio, based on eye-tracking data, extracts pupil positive dilation differential, fixation dwell ratio, and saccade path entropy penalty; A continuous cognitive load correlation index was constructed based on extracted features using a cognitive load model.

[0013] Preferably, the industrial control state is adjusted using a dual-threshold hysteresis state machine based on a cognitive load correlation index, including: The state transition rules for the dual-threshold hysteresis state machine based on the cognitive load correlation index are as follows: in, To enter a state of emergency The upper threshold, To restore normal status The lower threshold, For the current moment Cognitive load related index, express Current state for Current state; Adjust the industrial control state according to the above state transition rules.

[0014] Preferably, the ternary mapping relationship between the interface elements, data flow identifiers, and graphics resource objects forms a mapping record. In each record, each type of interface element has at least an interface priority, an associated data flow identifier, a graphics resource object identifier, and an emergency state control strategy. The control strategy also includes a data flow scheduling strategy and graphics resource rendering control parameters.

[0015] Preferably, the key data stream includes security alarms, core operating parameters, emergency operation entry points, and equipment protection signals; The non-critical data streams include historical trend updates, auxiliary charts, low-priority logs, non-urgent sensor data, and decorative 3D component data.

[0016] Preferably, the reduction of non-critical data streams is achieved through downsampling, rate limiting, dropping, or delaying processing.

[0017] Preferably, the control policy is deployed in the user-space queue, edge gateway, network interface layer, TC ingress (Traffic Control ingress) layer, XDP (eXpress Data Path) layer, or eBPF (extended Berkeley Packet Filter) mapping control layer.

[0018] Preferably, a GPU or video memory-retaining interface reconstruction is adopted. During the initialization phase, GPU vertex buffer objects, texture objects, vertex array objects, and shader programs are created for the core interface layer and the auxiliary interface layer. When entering an emergency state, the auxiliary interface layer elements do not release the GPU vertex buffer objects, texture objects, vertex array objects, or shader programs, but only change the uniform variables of the shader programs, making the auxiliary interface layer transparent, weakened, or collapsed.

[0019] Preferably, the method further includes: treating data streams that are not identified, have not been prioritized, or have conflicting classification results as critical data streams by default.

[0020] Preferably, the method further includes: If EEG or eye-tracking data is missing, CCCI output is abnormal, the dual-threshold hysteresis state machine is unavailable, or the control strategy update fails, a preset safety fallback strategy is implemented to at least maintain full access to critical data streams and suspend the reduction of non-critical data streams or reduce them to the minimum required rate according to the non-critical stream reduction rate.

[0021] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention combines cognitive load-related continuous state, dual-threshold hysteresis debouncing, critical / non-critical data flow scheduling, and GPU / memory-retaining interface reconstruction into a unified closed loop. This enables industrial HMIs to not only reduce non-critical interface interference in scenarios with high operator cognitive load, but also to simultaneously reduce the impact of non-critical data and graphics resource reconstruction on the system processing link. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an industrial HMI anti-shake reconstruction and data stream scheduling method based on cognitive load, provided in the embodiment. Figure 2 This is a flowchart of the industrial HMI anti-shake reconstruction and data stream scheduling method based on cognitive load provided in the embodiment; Figure 3 This is a flowchart of the feature extraction and inference process for cognitive load-related indices provided in the embodiment; Figure 4 This is a schematic diagram of the dual-threshold hysteresis state machine transition provided in the embodiment; Figure 5This is a flowchart of the hierarchical scheduling process for critical / non-critical data flows provided in the embodiment; Figure 6 This is a schematic diagram of GPU / memory-retaining interface reconstruction provided in the embodiment; Figure 7 This is another schematic diagram of the GPU / memory-retaining interface reconstruction provided in the embodiment; Figure 8 This is a schematic diagram of ternary mapping collaborative control and mismatch avoidance provided in the embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] This invention provides an industrial HMI anti-jitter reconstruction and data flow scheduling method based on cognitive load. Through data path collaborative control and graphics resource lifecycle management, it reduces the burden on operator attention resources, user-mode processing queues, UI threads, and rendering pipelines by maintaining the continuous accessibility of critical alarms, core operating parameters, and emergency operation entry points, based on the operator's cognitive load-related continuous state. Furthermore, this invention applies cognitive load-related states to the industrial HMI's interface layer, data flow layer, and graphics resource layer. By mapping and recording the same interface element—data flow—graphics resource object, it associates and generates data flow scheduling strategies and rendering control parameters under state change events, thus forming a collaborative closed loop from state awareness, mode anti-jitter, data flow reduction, resource-preserving recovery to anomaly fallback.

[0026] like Figure 1 and Figure 2 As shown in the embodiment, an industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load is provided, including the following steps: S1. After extracting features from EEG and eye-tracking data, a continuous cognitive load-related index is constructed.

[0027] In the embodiments, such as Figure 3 As shown, EEG data and eye-tracking data are read, and an EEG window is generated based on the EEG data. Frequency band energy ratio Specifically, for each EEG channel Robust imputation of missing values ​​and mean removal are performed within the time window, and the power spectral density is estimated using the Welch method. .in Frequency band selection , Frequency band selection .

[0028] in, and They represent channels respectively. of Frequency band power and Frequency band power, It is a very small positive number, used to avoid the denominator being 0. The larger the value, the higher the working memory load or cognitive stress level.

[0029] In this embodiment, a positive pupil dilation differential is generated based on the pupil window corresponding to the eye movement data. This reflects the instantaneous dilation trend. Specifically, first, the effective mean value is taken from the left and right pupil diameters, or a single pupil column is read; then, robust filling and Savitzky-Golay smoothing are performed on the pupil sequence. The baseline pupil diameter is used as the reference. For reference, the smoothed pupil diameter of the current window Relative normalization is performed to obtain the normalized pupil diameter. and build .

[0030] in, The eye-tracking sampling frequency, The difference in pupil diameter between two adjacent sampling points and The product of these two factors represents the discrete-time rate of change of the normalized pupil diameter. The 95th quantile operator is used, preferably with a positive derivative, to reduce the impact of blinks, interpolation spikes, or local outliers on the feature scale. If a positive derivative does not exist, then... Take 0.

[0031] In this embodiment, a gaze duration ratio is also generated based on the eye-tracking coordinate window corresponding to the eye-tracking data. This is used to reflect the degree of gaze / pause. Specifically, a velocity threshold is preferred to approximate gaze dwell time, rather than a complete I-DT (Identification by Dispersion-Threshold) clustering algorithm. (The text then abruptly shifts to discussing eye movement coordinates.) Calculate eye movement velocity between adjacent sampling points : Among them, eye movement speed threshold The preferred speed is 80 px / s. This is an indicator function. This represents the number of sampling points within the current eye-tracking window. This is the index for the sampling points. The proportion of slow-speed or localized fixation within the window can serve as an approximate indicator of information encoding difficulties or sustained localized attention; it should be noted that... The subscript dt in the variable name is only part of the variable name and is used to represent the feature "gaze dwell ratio". This is different from the engineering playback time step dt = 0.2 seconds in the CCCI playback sequence. It should be... Treat it as a single variable name. Alternatively, the I-DT algorithm can be used to calculate the gaze duration based on statistics of gaze clustering duration. .

[0032] In this embodiment, a saccade path entropy penalty is also generated based on the scanning path transition corresponding to the eye-tracking data. This is used to reflect the degree of deviation from the saccade path. Specifically, the adaptive screen range, determined by the 1% and 99th percentiles of the eye-tracking coordinates, is divided into a 4×4 grid, resulting in 16 discrete states. The adjacent sampling points are statistically analyzed from state... to state Number of transfers This forms a first-order transition distribution.

[0033] in, This indicates that adjacent eye-tracking sampling points are represented by discrete grid states. i Transition to discrete grid state j normalized transition frequency, Indicates the state i Transition to state j Number of transfers, A very small positive number set to avoid the denominator being zero. It represents the first-order transition entropy of the scan path calculated based on the normalized transition frequency of each state transition pair, and is used to characterize the discreteness or uncertainty of the scan path transition mode within the current window. This is the median of the saccade path entropy in the calibration or baseline data. This quadratic penalty term is non-negative and can simultaneously characterize saccade path deviations that are excessively cluttered or excessively contracted relative to the baseline.

[0034] In this embodiment, the extracted data is used to form a four-dimensional feature vector. Calculate each feature mean and standard deviation Then, Z-score standardization is performed to obtain the standardized features. : in, i For feature indexing.

[0035] A continuous cognitive load correlation index (CCCI) was constructed based on four standardized features using a cognitive load model. The cognitive load model is a multilayer perceptron with a 4-dimensional input layer, a first hidden layer with 64 neurons, a second hidden layer with 32 neurons, and ReLU activation function used in the hidden layers. Dropout is set to 0.3. The output layer has one neuron connected to a Sigmoid function. It can be represented as follows: in, Represents the ReLU activation function. This represents a random deactivation operation with Dropout=0.3 during the training phase; during CCCI inference, playback, and online triggering phases, when the model is in inference, the Dropout layer is turned off or equivalently represented by an identity mapping. The Sigmoid output refers to any real number from the last layer of the model. Through the above Compress to the (0,1) interval. The output is not the binary classification label itself, but a continuous cognitive load index (CCCI), which is convenient for connection with a dual-threshold hysteresis state machine. The closer the CCCI is to 1, the closer the model believes the current window is to a high-load project trigger state; the closer it is to 0, the closer it is to a low-load project trigger state.

[0036] In this embodiment, the model undergoes supervised training using real cognitive load labels, with the mean squared error loss function as the training objective, and the AdamW optimizer is used to update the model parameters. After training, the model parameters, along with the mean and standard deviation of the features used for standardization, are saved for use in the CCCI inference phase. The number of training epochs and the learning rate can be configured based on the training data size and model convergence.

[0037] S2, based on the cognitive load-related index, adjusts the industrial control state through a dual-threshold hysteresis state machine. When the industrial control state is adjusted to an emergency state, it synchronously realizes data flow scheduling and industrial HMI reconstruction based on the ternary mapping relationship between interface elements, data flow identifiers, and graphical resource objects through the control strategy bound to the same interface element. Specifically, it includes: retaining key data flows, reducing non-key data flows, maintaining the core interface layer corresponding to key data flows, and weakening or folding the auxiliary interface layer corresponding to non-key data flows.

[0038] In this embodiment, an upper threshold for entering an emergency state is set. and the lower threshold for returning to normal state ,in Preferably, , .make express The current state can be either a normal state or an emergency state, such as... Figure 4 As shown, the state transition rules implemented by the dual-threshold hysteresis state machine are as follows: in, Indicates a state of emergency. Indicates a normal state. for Moment .

[0039] When in normal state ( When entering an emergency state, maintain the integrity of the interface to avoid accidental triggering. When this occurs, it corresponds to the minimalist emergency mode, retaining core alarms and critical operations while weakening non-critical components to reduce visual burden. Specifically, it retains key interface elements and key data flows, reduces non-critical data flows, and lowers the input load on the user state and UI thread. When in the hysteresis hold-up zone of engineering control (… When ), maintain state This can reduce interface flickering and repeated switching near the threshold, and suppress interface jitter caused by critical fluctuations. When the system returns to normal ( When this occurs, it corresponds to the normal monitoring mode, smoothly restoring non-critical charts and auxiliary menus, reducing interference from sudden changes in the recovery process.

[0040] By using the aforementioned dual-threshold hysteresis state machine, the problems of frequent switching, visual flickering, and secondary interference that easily occur in the critical load region when using single-threshold or regular interface switching can be solved, making the interface mode transition have a more stable hold area and recovery conditions.

[0041] In this embodiment, the system divides industrial input data streams into critical data streams and non-critical data streams based on stream identifiers or priorities. Critical data streams include safety alarms, core operating parameters, emergency operation entry points, and equipment protection signals; non-critical data streams include historical trend updates, auxiliary charts, low-priority logs, non-emergency sensor data, and decorative 3D component data.

[0042] like Figure 5As shown, in the normal state, both critical and non-critical data streams can enter the receiving process and UI rendering process at normal frequencies for normal passage / refresh. In the emergency state, critical streams still pass through in full; for non-critical data streams bound to non-critical interface elements, the system performs downsampling, rate limiting, dropping, or delay processing according to the emergency state handling strategy in the pre-established "interface element - data stream identifier - graphics resource object" mapping record. The scheduling of this control strategy can be deployed at the user-space queue, edge gateway, network interface layer, TC ingress layer, XDP layer, or eBPF mapping table control layer.

[0043] To avoid erroneous reduction of safety-related data in industrial safety scenarios, the system can set conservative classification rules: data streams that are not identified, have not been prioritized, or have conflicting classification results are treated as critical data streams by default; core operating parameters required for safety alarms, equipment protection signals, emergency operation entry points, and manual confirmation are not discarded or forcibly downsampled under any circumstances. If EEG data or eye-tracking data is missing, CCCI output is abnormal, the hysteresis state machine is unavailable, or the control strategy (data stream scheduling strategy and graphics resource rendering control parameters) fails to update, the system enters a preset safety fallback strategy, at least ensuring that critical data streams are fully allowed to pass, and can suspend the reduction of non-critical data streams or reduce them to the minimum required rate according to the non-critical stream reduction rate, in order to avoid introducing new safety risks due to abnormal state estimation.

[0044] in, Non-critical flow reduction rate and These represent the number of non-critical flows that passed through and the number of flows that were dropped.

[0045] In one implementation, the scheduling of the aforementioned emergency control strategy is implemented at the TC ingress layer or the eBPF mapping control layer. Specifically, a sender namespace ns_tx and a receiver namespace ns_rx are built within the Linux terminal and connected via a veth virtual network interface. The receiver veth_rx's TC ingress mounts the eBPF program. The user-space controller reads the CCCI playback sequence and updates the policy fields in the BPF map according to the normal / emergency state. Specific fields, exemplary values, and functions are shown in Table 1. Table 1 The eBPF program parses Ethernet frames, IPv4, and UDP headers; it returns TC_ACT_OK directly when the destination port is critical_port; it returns TC_ACT_OK when the destination port is auxiliary_port and emergency=0; and it performs deterministic sampling based on the sequence number seq in the packet payload when the destination port is auxiliary_port and emergency=1.

[0046] in, This indicates that the request is granted and TC_ACT_OK is returned. This indicates that TC_ACT_SHOT will be discarded and returned. It represents aux_keep_per_mille, and The value ranges from 0 to 1000. This rule theoretically ensures that the retention rate of non-critical flows is approximately aux_keep_per_mille / 1000. For example, when aux_keep_per_mille=200, the theoretical retention is about 20%, and the theoretical reduction is about 80%. Actual experimental results are affected by the duration of the emergency, state transition boundaries, and sampling statistics; therefore, state-specific statistics should be used as the basis for determination.

[0047] In this embodiment, the industrial HMI is also reconstructed for both critical and non-critical data flows. Specifically, the interface is reconstructed using GPU or memory-retaining methods. In the OpenGL / Shader implementation, the front-end industrial HMI is divided into a core interface layer and an auxiliary interface layer. The core interface layer displays safety alarms, core operating parameters, and emergency operation entry points; the auxiliary interface layer displays historical trend curves, low-priority logs, auxiliary charts, and unnecessary 3D decorative component data.

[0048] During the initialization phase, GPU resources are created for the core and auxiliary UI layers, including GPU vertex buffers, textures, vertex array objects (VAOs), and shaders. Upon entering the emergency state, auxiliary UI layer elements do not release GPU buffers, textures, VAOs, or shaders; only shader uniforms are modified, such as the auxiliary UI layer opacity parameter (aux_alpha) and fold factor, making the auxiliary UI layer transparent, weakened, or folded. Upon returning to the normal state, existing GPU resources are reused, and the auxiliary UI layer display is restored through gradual changes in opacity or fold factor.

[0049] in, For fold_factor, A fixed blending weight for vertex color and texture sampling. The vertex color, Sample color for texture. The color of the mixed fragments. For aux_alpha, FragColor (the final color vector output by the fragment shader). and These are the x-coordinates of the auxiliary object's vertices before and after the folding transformation, respectively. and These are the ordinates of the auxiliary object's vertices before and after the folding transformation, respectively. This indicates normal unfolding. This indicates that the auxiliary object shrinks towards the normalized coordinate origin or a preset shrinkage position according to a preset ratio. This indicates a normal, opaque display. This indicates a weakened display in emergency situations. The above parameters are updated as shaderuniforms during state transitions, without requiring the destruction or reconstruction of auxiliary layer GPU resources. In the OpenGL example code, and These correspond to uniform variables such as u_fold and u_alpha, respectively.

[0050] In this embodiment, by reconstructing the interface using GPU or video memory retention, the problems that may occur when restoring normal monitoring mode due to traditional methods of hiding controls, stopping refresh, or destroying auxiliary interface objects can be solved, such as concentrated reconstruction of graphics resources, frame time spikes, and recovery stuttering. This improves the continuity and recoverability of industrial HMI when switching between emergency mode and normal mode.

[0051] In this embodiment, a mapping relationship between interface elements, data flow identifiers, and graphical resource objects is pre-established, and a mapping record is formed, such as... Figure 8As shown, each record contains at least the interface priority, associated data flow identifier, graphics resource object identifier, and emergency state control strategy for each HMI element. After the hysteresis state machine outputs the normal / emergency state, the system updates the data flow scheduling strategy and rendering control parameters based on the mapping record, enabling the interface layer, data path, and graphics pipeline to coordinate actions based on the same state source. The mapping records are shown in Table 2. Table 2 Through the aforementioned mapping records, the data flow scheduling strategy and GPU rendering control parameters are generated in association with the same state change event during state changes, and can be recorded as a consistent strategy version within the same control cycle or the same coordination control loop. This mechanism is used to avoid inconsistent states where the interface has been weakened but non-critical data is still fully input, or where data has been reduced but the interface still displays complete auxiliary content.

[0052] The embodiments also provide experimental verification results of the above methods, specifically including: 1. CCCI playback sequence, hysteresis state machine, and data caliber verification results This embodiment uses the final CCCI playback sequence as the common input for the GPU / OpenGL prototype, TC / eBPF prototype, and cooperative control experiment. The ccci_sequence_used.csv files saved on both the GPU and TC / eBPF sides, as well as the ccci_sequence.csv file read by the cooperative control experiment, are consistent with the main sequence after SHA256 verification, indicating that the subsequent graphics resource experiment, kernel network path experiment, and cooperative control logic experiment are executed based on the same state sequence.

[0053] 2. Comparison of anti-shake strategies using dual-threshold hysteresis and single-threshold hysteresis The results of comparing the anti-shake strategies of dual-threshold hysteresis and single-threshold hysteresis are shown in Table 3: Table 3 Table 3 shows that, on the same final CCCI replay sequence, the dual-threshold hysteresis strategy reduces the number of normal / emergency state transitions from 70 to 24, a reduction of approximately 65.71%; and reduces the number of jittery transitions within a short period from 50 to 10, a reduction of approximately 80.00%. This result indicates that the dual-threshold hysteresis state machine can reduce repeated normal / emergency state transitions near critical loads, providing more stable state inputs for subsequent data stream scheduling and HMI graph reconstruction. It should be noted that the dual-threshold hysteresis strategy does not aim to reduce the total proportion of emergencies, but rather reduces frequent transitions by maintaining the state between the entry and recovery thresholds.

[0054] 3. TC / eBPF Prototype Experiment Results The results of the TC / eBPF prototype experiment are shown in Table 4. The TC / eBPF experiment was performed for 760 seconds based on the final CCCI replay sequence, and the experiment duration covered the complete replay sequence. When the experiment time exceeded the end of the sequence, the user-space controller could continue reading the state by taking the modulo of the replay time. The experimental mount point was TC ingress, the critical flow was UDP 5001, and the non-critical flow was UDP 5002. The results show that the critical flow drop rate was 0.00% in both emergency and normal states; the non-critical flow reduction rate was 70.87% in the emergency state and 99.37% in the normal state.

[0055] Table 4 The above results indicate that under high-load emergency conditions, critical flows pass through in full, while non-critical flows are significantly reduced; under normal conditions, non-critical flows pass through almost entirely. Since the emergency state accounts for a relatively small proportion of the entire replay, the overall reduction rate should not be used as the primary indicator. The emergency non-critical flow reduction rate in the state-specific statistics is the key evidence for evaluating this mechanism. The statement "critical flow drop rate of 0.00%" refers to the result under the statistical caliber of the TC / eBPF program in this embodiment, used to indicate that the strategy does not actively drop packets identified as critical flows; this statement is not equivalent to an absolute guarantee of UDP end-to-end arrival rate under all network environments.

[0056] 4. GPU / VRAM-reserved interface prototype results Figure 6 and Figure 7The diagram illustrates the GPU / memory-preserving interface reconstruction. The GPU / OpenGL experiment was conducted using the same final CCCI sequence, 3640 frames, and 300 auxiliary objects. Both strategies underwent 24 mode switches. The Destroy / Rebuild baseline strategy created buffers 3600 times and textures 3600 times during runtime, with 3600 destructions and 3600 reconstructions. The RetainedResource strategy had zero buffer creations, zero texture creations, zero destructions, and zero reconstructions during mode switches. Specific results are shown in Table 5.

[0057] Table 5 In this embodiment, the resource retention policy keeps runtime buffer / texture creation, destruction, and reconstruction at 0 during the 24 state transitions, and reduces the peak values ​​of maximum frame time, maximum frame time during recovery, and maximum frame time during switching.

[0058] 5. Results of Co-mapping Ablation Experiment To verify that the ternary cooperative control relationship is not a simple parallel of ordinary interface hiding, ordinary data rate limiting, and ordinary graphics resource reuse, a cooperative mapping ablation experiment was conducted. This experiment used the same CCCI playback sequence, the same dual-threshold hysteresis state output, and the same interface element-dataflow-graphics resource object mapping table to compare four strategies: display_only_hide (only interface weakening / hiding), data_only_shedding (only data reduction), display_data_unsynced (interface and data asynchronous updates), and full_coordination_mapping (full cooperative mapping). This experiment belongs to the cooperative control logic level and is used to demonstrate the consistency between mapping cooperation and strategy versions. The results are shown in Table 6: Table 6 Note: data_only_shedding does not involve auxiliary interface graphics resource weakening / restoration control, and its resource reconstruction count is not used as a basis for comparing GPU lifecycle advantages; the resource reconstruction counts in display_only_hide and display_data_unsynced correspond to the normal destruction / reconstruction interface switching baseline.

[0059] Experimental results show that, under conditions of 3640 control steps and 24 normal / emergency state transitions, the complete collaborative mapping strategy did not exhibit any interface / data inconsistency control cycles, nor did it show any strategy version mismatch. The number of critical flow drops was 0, and the number of times auxiliary graphics resources were created, destroyed, and rebuilt during runtime was also 0. In contrast, the display-only, data-only, and display+data unsynced control strategies exhibited 222, 222, and 58 inconsistency control cycles, respectively. These results indicate that associated updates under the same mapping record and the same state change event can reduce mismatches between the interface layer, data path, and resource lifecycle.

[0060] 6. Results of the safety fallback abnormal injection experiment To verify the conservative handling logic in industrial safety scenarios, a safety fallback anomaly injection experiment was conducted. The experiment injected eight types of anomalies into the coordination controller: CCCI NaN, CCCI inf, CCCI out of bounds, sensor missing, hysteresis state machine unavailable, unknown data flow, classification conflict, and policy update failure. The experimental goal was not to prove absolute safety under arbitrary fault scenarios, but rather to verify that under the aforementioned controllable anomaly injection conditions, the system can trigger a preset fallback strategy to avoid actively discarding critical flows and to maintain the visibility of core interface elements and emergency operation entry points.

[0061] Table 7 The experimental results are shown in Table 7. All eight abnormal test cases passed the preset assertions. The cumulative number of dropped critical data streams (critical_drop_total) was 0 for each test case, the cumulative number of dropped unknown data streams (unknown_stream_drop_total) and the cumulative number of dropped conflicting data streams (conflict_stream_drop_total) were both 0, the number of times core UI elements were invisible (core_ui_invisible_count) and the number of times emergency operation entries were invisible (emergency_entry_invisible_count) were both 0, and the number of times unsafe data was shedding (unsafe_shedding_count) was 0. Among them, the policy update failure test case recorded that the observed policy update failure (policy_update_failure_observed) = True, indicating that the anomaly was actually injected and handled by the fallback logic. This result is used to support the conservative handling of unidentified, unlabeled, or conflicting data streams, as well as the safety fallback mechanism to maintain critical streams and core interfaces when CCCI or state machine anomalies occur.

[0062] 7. Experimental Results of Synchronous Time-Sequence Log To verify that the data stream scheduling policy and rendering control parameters are generated by the same state change event, a synchronous time-series log experiment was conducted. This experiment only ran the full_coordination_mapping policy, recording the following fields for each normal / emergency state change event: state_change_event, stream_policy_version, render_policy_version, control_cycle_id, same_control_cycle, policy_version_match, and stream_render_skew_ms (time difference between data stream policy and rendering policy updates, in milliseconds), as shown in Table 8.

[0063] Table 8 Note: The precise nanosecond / microsecond timestamps of the synchronous timing experiment are affected by the operating environment. When reviewing, discrete logic indicators such as consistent strategy version, mismatch cycle of 0, and same_control_cycle_ratio of 1.0 should be used as the standard.

[0064] The experimental results demonstrate that, in this embodiment, all 24 state change events were associated with and generated consistent data stream strategy versions and rendering strategy versions, and no strategy version mismatch or interface / data inconsistency cycles were observed. These results support the technical feature of "associated updates under the same state change event."

[0065] Based on the above experimental results, the industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load of the present invention, combined with technical means, has the following effects: (1) By using CCCI, multimodal physiological and behavioral characteristics such as EEG data and eye-tracking data are transformed into unified trigger variables for HMI reconstruction and data flow scheduling, enabling the interface layer, data flow layer and graphics resource layer to coordinate actions based on the same state source.

[0066] (2) By using a dual-threshold hysteresis state machine, the current state is maintained within the threshold range, reducing the frequent switching, visual flickering and secondary interference that may occur in the critical area of ​​cognitive load.

[0067] (3) By retaining critical flows and reducing non-critical flows, the non-critical input load under high load conditions is reduced while maintaining the priority of critical data flows.

[0068] (4) Through GPU / video memory retention interface reconstruction, during mode switching, the resources such as GPU buffer, texture, VAO and shader corresponding to auxiliary interface elements are not destroyed or rebuilt, and the weakening, folding and restoration are controlled only by rendering parameters.

[0069] (5) Through the three-element mapping relationship of interface element - data flow identifier - graphics resource object, the same state change event can be associated with updating the data flow scheduling strategy and rendering control parameters.

[0070] (6) Through a safety fallback strategy, under abnormal injection conditions such as CCCI anomaly, sensor missing, state machine unavailability, unknown data flow, classification conflict and policy update failure, the critical flow is fully allowed, core interface elements are visible and emergency operation entry is accessible.

[0071] (7) By recording the data strategy version and rendering strategy version under the same state change event through synchronous time-series logs, the controller can detect and avoid long-term mismatch between interface and data strategy.

[0072] This invention reduces resource consumption and upgrade pressure on existing industrial hosts, edge gateways, or graphics terminals by reducing non-critical data flows under high load conditions, minimizing the chance of non-critical data entering the user-space processing queue and rendering thread, and reducing resource rebuilding during the recovery phase by preserving graphics resources. Collaborative control experiments also show that managing interface elements, data flows, and graphics resource objects using a unified mapping record helps reduce maintenance costs caused by separate configuration of multiple modules, asynchronous states, and repeated rebuilding during the recovery phase in engineering implementation. For already deployed industrial HMI systems, this solution can be implemented as an enhancement mechanism at the software or system configuration level.

[0073] In safety-critical scenarios such as nuclear power, power grids, rail transit, process industries, and complex manufacturing equipment, this invention helps maintain continuous access to critical alarms, core operating parameters, and emergency operation entry points when operators are under high cognitive load, while reducing the occupation of attentional resources by non-critical information and data. Anomaly injection experiments further demonstrate that under controllable anomaly conditions such as CCCI anomalies, sensor missingness, unknown flows, and policy update failures, the system can enter a conservative fallback processing path, avoiding accidental damage to critical flows or hiding of core interfaces due to automated reconfiguration strategies. This effect is beneficial for improving the stability of human-machine collaboration in complex industrial systems.

[0074] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for anti-jitter reconstruction and data stream scheduling in industrial HMIs based on cognitive load, characterized in that, Includes the following steps: After extracting features from EEG and eye-tracking data, a continuous cognitive load-related index was constructed. Based on the cognitive load-related index, the industrial control state is adjusted through a dual-threshold hysteresis state machine. When the industrial control state is adjusted to an emergency state, based on the ternary mapping relationship between interface elements, data flow identifiers, and graphical resource objects, the data flow scheduling and industrial HMI reconstruction are synchronously realized through the control strategy bound to the same interface element. Specifically, this includes: retaining key data flows, reducing non-key data flows, maintaining the core interface layer corresponding to key data flows, and weakening or folding the auxiliary interface layer corresponding to non-key data flows.

2. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, After extracting features from EEG and eye-tracking data, a continuous cognitive load-related index was constructed, including: Feature extraction includes: extraction based on EEG data Frequency band energy ratio, based on eye-tracking data, extracts pupil positive dilation differential, fixation dwell ratio, and saccade path entropy penalty; A continuous cognitive load correlation index was constructed based on extracted features using a cognitive load model.

3. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, Based on cognitive load-related indices, industrial control states are adjusted using a dual-threshold hysteresis state machine, including: The state transition rules for the dual-threshold hysteresis state machine based on the cognitive load correlation index are as follows: in, To enter a state of emergency The upper threshold, To restore normal status The lower threshold, For the current moment Cognitive load related index, express Current state for Current state; Adjust the industrial control state according to the above state transition rules.

4. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The ternary mapping relationship between the interface elements, data flow identifiers, and graphics resource objects forms a mapping record. In each record, each type of interface element has at least an interface priority, an associated data flow identifier, a graphics resource object identifier, and an emergency state control strategy. The control strategy also includes a data flow scheduling strategy and graphics resource rendering control parameters.

5. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The key data streams include safety alarms, core operating parameters, emergency operation entry points, and equipment protection signals. The non-critical data streams include historical trend updates, auxiliary charts, low-priority logs, non-urgent sensor data, and decorative 3D component data.

6. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, Non-critical data streams can be reduced by downsampling, rate limiting, dropping, or delaying processing.

7. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The control strategy is deployed in the user-space queue, edge gateway, network interface layer, TC ingress layer, XDP layer, or eBPF mapping table control layer.

8. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The interface is reconstructed using GPU or video memory retention. During the initialization phase, GPU vertex buffer objects, texture objects, vertex array objects, and shader programs are created for the core interface layer and auxiliary interface layer. When entering an emergency state, the auxiliary interface layer elements do not release GPU vertex buffer objects, texture objects, vertex array objects, or shader programs. They only change the uniform variables of the shader programs, making the auxiliary interface layer transparent, weakened, or collapsed.

9. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The method also includes: treating data streams that are not identified, have not been prioritized, or have conflicting classification results as critical data streams by default.

10. The industrial HMI anti-jitter reconstruction and data stream scheduling method based on cognitive load according to claim 1, characterized in that, The method further includes: If EEG or eye-tracking data is missing, CCCI output is abnormal, the dual-threshold hysteresis state machine is unavailable, or the control strategy update fails, a preset safety fallback strategy is implemented to at least maintain full access to critical data streams and suspend the reduction of non-critical data streams or reduce them to the minimum required rate according to the non-critical stream reduction rate.

Images