A method and apparatus for predicting human-machine conflicts

Abstract

This disclosure provides a method and apparatus for predicting human-machine conflict, capable of proactively predicting human-machine conflict from multiple dimensions. By acquiring and analyzing various physiological and behavioral data of operators in historical human-machine interaction scenarios, significant characterization factors corresponding to fatigue states are screened, enabling a more accurate quantitative representation of operator status. Simultaneously, the significant characterization factors of operators are combined with environmental and machine characteristic dimensions (including the complexity of interface interaction) to establish a human-machine conflict prediction model suitable for the target scenario. This method can dynamically reflect changes in multiple factors during system operation, proactively predicting the probability of human-machine conflict occurrence, thereby improving the comprehensiveness, timeliness, and reliability of identification, avoiding lag and bias caused by single data or rules, and ultimately contributing to improving the safety and stability of complex human-machine systems.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method and apparatus for predicting human-computer conflict. Background Technology

[0002] Human-machine conflict refers to situations where, during system operation, the operator's behavior and intentions conflict with the system's state, functions, and outputs, resulting in incoordination, inconsistency, or even mutual interference. For example, conflicts may arise between the operation of an aircraft's autopilot system and the pilot's manual operation. Currently, methods for identifying human-machine conflicts are limited to passive response and local optimization, often based on fixed patterns and rules. They fail to fully consider the various dynamic factors that change during system operation, making it impossible to comprehensively understand the nature and influencing factors of human-machine conflict. This affects the effectiveness and reliability of identification methods, leading to untimely identification. Summary of the Invention

[0003] In view of this, the present disclosure provides a method and apparatus for predicting human-machine conflict, so as to improve the accuracy and timeliness of human-machine conflict identification.

[0004] Specifically, this disclosure is achieved through the following technical solution:

[0005] In a first aspect, embodiments of this disclosure provide a method for predicting human-machine conflict, including:

[0006] Acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios;

[0007] Based on the physiological and behavioral data, various data types associated with fatigue state are identified, and these data types are used as significant characterization factors corresponding to the fatigue state.

[0008] Based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario, a human-computer conflict prediction model corresponding to the target human-computer interaction scenario is constructed; the machine feature dimension includes the complexity of interface interaction operations.

[0009] The human-computer conflict prediction model is used to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario.

[0010] Optionally, the human-computer conflict prediction model includes a first sub-model and a second sub-model; the step of using the human-computer conflict prediction model to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario includes:

[0011] In the current target human-computer interaction scenario, acquire the operator's significant representation data under the aforementioned significant representation factors;

[0012] The significant characterization data is input into the first sub-model to predict the operator's fatigue state;

[0013] The fatigue state, environmental features of the target human-computer interaction scenario, and machine features of the target human-computer interaction scenario are input into the second sub-model to predict the probability of human-computer conflict occurring in the target human-computer interaction scenario.

[0014] Optionally, the first sub-model can be constructed through the following steps:

[0015] Based on the aforementioned significant characterization factors, a feature vector of significant characterization factors is constructed;

[0016] Based on the initial long short-term memory neural network, a first sub-model is constructed; the first sub-model includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the feature vector of the significant representation factor; the hidden layer is used to learn the temporal features of the feature vector of the significant representation factor; and the output layer is used to output the fatigue state of the operator.

[0017] Optionally, the second sub-model can be constructed through the following steps:

[0018] Construct multiple Bayesian network nodes; the Bayesian network nodes are used to represent the fatigue state, the environmental features of the target human-computer interaction scenario, and the machine features of the target human-computer interaction scenario;

[0019] Based on the dependencies between the Bayesian network nodes, directed edges connecting the Bayesian network nodes are determined; the starting position of the directed edge is the parent node, and the ending position is the child node.

[0020] Determine the prior probability of each Bayesian network node and the conditional probability of the directed edge; the prior probability is used to represent the probability that each Bayesian network node takes a specific state without considering other Bayesian network nodes; the conditional probability is used to represent the probability that the child node takes a specific state given the state of the parent node.

[0021] Based on the Bayesian network nodes, the directed edges, the prior probabilities, and the conditional probabilities, an initial second sub-model is constructed;

[0022] The initial second sub-model is trained to obtain the second sub-model.

[0023] Optionally, based on the physiological and behavioral data, the determination of various data types associated with fatigue states includes:

[0024] The physiological and behavioral data are standardized to obtain standard data under various standard data types;

[0025] Based on the standard data and the forward stepwise regression model, a variety of first target data types that affect fatigue state are selected from the standard data types.

[0026] Determine the variance inflation factor of the first target data type, and based on the variance inflation factor, select a variety of second target data types that affect fatigue state from the first target data type.

[0027] Optionally, the physiological and behavioral data includes electroencephalogram (EEG) data and eye movement data;

[0028] The standardization process for the physiological and behavioral data includes:

[0029] Determine the causal relationship between the electroencephalogram (EEG) data and the eye movement (EMG) data;

[0030] When there is a causal relationship between the EEG data and the eye movement data, the EEG data and the eye movement data are combined to obtain standard data under the EEG-eye movement data type.

[0031] Optionally, the physiological and behavioral data may also include at least one of the following:

[0032] Facial data, eye aspect ratio data, electroencephalogram (EEG) data, and electrocardiogram (ECG) data.

[0033] Optionally, the method further includes:

[0034] If the probability of human-computer conflict is greater than the target threshold, the human-computer interaction interface in the current target human-computer interaction scenario will be switched to a simplified interface.

[0035] Optionally, the simplified interface can be generated through the following steps:

[0036] The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface, and multiple initial simplified interfaces are generated using a generative adversarial network.

[0037] Based on the distance of the target operation button from the center of the interface in the initial simplified interface, and the size information of the target operation button, the operation efficiency of the initial simplified interface is determined;

[0038] Based on the operational efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the plurality of initial simplified interfaces.

[0039] Optionally, the method further includes:

[0040] Based on the probability of human-computer conflict, the task difficulty in the current target human-computer interaction scenario, the machine performance information in the current target human-computer interaction scenario, and the trained task optimization model, a task optimization strategy is determined; the task optimization model is trained based on the near-end strategy optimization model.

[0041] Based on the task optimization strategy, the operator's tasks in the current target human-computer interaction scenario are optimized; the optimization includes tasks with full machine takeover, tasks with partial machine takeover, and tasks that maintain manual operation.

[0042] Secondly, embodiments of this disclosure also provide a human-machine conflict prediction device, comprising:

[0043] The acquisition module is used to acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios;

[0044] The determination module is used to determine multiple data types associated with fatigue state based on the physiological and behavioral data, and to use the data types as significant characterization factors corresponding to the fatigue state.

[0045] The construction module is used to construct a human-computer conflict prediction model corresponding to the target human-computer interaction scenario based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario; the machine feature dimension includes the complexity of interface interaction operation.

[0046] The prediction module is used to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario using the human-computer conflict prediction model.

[0047] Optionally, the human-machine conflict prediction model includes a first sub-model and a second sub-model; the prediction module is specifically used for:

[0048] In the current target human-computer interaction scenario, acquire the operator's significant representation data under the aforementioned significant representation factors;

[0049] The significant characterization data is input into the first sub-model to predict the operator's fatigue state;

[0050] The fatigue state, environmental features of the target human-computer interaction scenario, and machine features of the target human-computer interaction scenario are input into the second sub-model to predict the probability of human-computer conflict occurring in the target human-computer interaction scenario.

[0051] Optionally, the building module is specifically used for:

[0052] Based on the aforementioned significant characterization factors, a feature vector of significant characterization factors is constructed;

[0053] Based on the initial long short-term memory neural network, a first sub-model is constructed; the first sub-model includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the feature vector of the significant representation factor; the hidden layer is used to learn the temporal features of the feature vector of the significant representation factor; and the output layer is used to output the fatigue state of the operator.

[0054] Optionally, the building module is specifically used for:

[0055] Construct multiple Bayesian network nodes; the Bayesian network nodes are used to represent the fatigue state, the environmental features of the target human-computer interaction scenario, and the machine features of the target human-computer interaction scenario;

[0056] Based on the dependencies between the Bayesian network nodes, directed edges connecting the Bayesian network nodes are determined; the starting position of the directed edge is the parent node, and the ending position is the child node.

[0057] Determine the prior probability of each Bayesian network node and the conditional probability of the directed edge; the prior probability is used to represent the probability that each Bayesian network node takes a specific state without considering other Bayesian network nodes; the conditional probability is used to represent the probability that the child node takes a specific state given the state of the parent node.

[0058] Based on the Bayesian network nodes, the directed edges, the prior probabilities, and the conditional probabilities, an initial second sub-model is constructed;

[0059] The initial second sub-model is trained to obtain the second sub-model.

[0060] Optionally, the determining module is specifically used for:

[0061] The physiological and behavioral data are standardized to obtain standard data under various standard data types;

[0062] Based on the standard data and the forward stepwise regression model, a variety of first target data types that affect fatigue state are selected from the standard data types.

[0063] Determine the variance inflation factor of the first target data type, and based on the variance inflation factor, select a variety of second target data types that affect fatigue state from the first target data type.

[0064] Optionally, the physiological and behavioral data includes electroencephalogram (EEG) data and eye movement data;

[0065] The determining module is specifically used for:

[0066] Determine the causal relationship between the electroencephalogram (EEG) data and the eye movement (EMG) data;

[0067] When there is a causal relationship between the EEG data and the eye movement data, the EEG data and the eye movement data are combined to obtain standard data under the EEG-eye movement data type.

[0068] Optionally, the physiological and behavioral data may also include at least one of the following:

[0069] Facial data, eye aspect ratio data, electroencephalogram (EEG) data, and electrocardiogram (ECG) data.

[0070] Optionally, the device further includes a switching module for:

[0071] If the probability of human-computer conflict is greater than the target threshold, the human-computer interaction interface in the current target human-computer interaction scenario will be switched to a simplified interface.

[0072] Optionally, the apparatus further includes a generation module for:

[0073] The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface, and multiple initial simplified interfaces are generated using a generative adversarial network.

[0074] Based on the distance of the target operation button from the center of the interface in the initial simplified interface, and the size information of the target operation button, the operation efficiency of the initial simplified interface is determined;

[0075] Based on the operational efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the plurality of initial simplified interfaces.

[0076] Optionally, the device further includes an optimization module for:

[0077] Based on the probability of human-computer conflict, the task difficulty in the current target human-computer interaction scenario, the machine performance information in the current target human-computer interaction scenario, and the trained task optimization model, a task optimization strategy is determined; the task optimization model is trained based on the near-end strategy optimization model.

[0078] Based on the task optimization strategy, the operator's tasks in the current target human-computer interaction scenario are optimized; the optimization includes tasks with full machine takeover, tasks with partial machine takeover, and tasks that maintain manual operation.

[0079] Thirdly, an optional implementation of this disclosure also provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, they perform the steps of the first aspect above, or any possible implementation of the first aspect.

[0080] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium storing a computer program that, when run, performs the steps of the first aspect or any possible implementation of the first aspect.

[0081] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0082] The human-machine conflict prediction method and apparatus provided in this disclosure can proactively predict human-machine conflicts from multiple dimensions. By acquiring and analyzing various physiological and behavioral data of operators in historical human-machine interaction scenarios, significant characterization factors corresponding to fatigue states are screened out, enabling a more accurate quantitative representation of personnel states. Simultaneously, the significant characterization factors of operators are combined with environmental and machine characteristic dimensions (including the complexity of interface interaction operations) to establish a human-machine conflict prediction model suitable for the target scenario. This method can dynamically reflect changes in multiple factors during system operation, proactively predict the probability of human-machine conflicts, thereby improving the comprehensiveness, timeliness, and reliability of identification, avoiding lag and bias caused by single data or rules, and ultimately contributing to improving the safety and stability of complex human-machine systems. Attached Figure Description

[0083] Figure 1 A flowchart of a human-machine conflict prediction method provided by an embodiment of this disclosure is shown;

[0084] Figure 2 A schematic diagram of an eye image provided in an embodiment of this disclosure is shown;

[0085] Figure 3 A schematic diagram of a human-machine conflict prediction device provided in an embodiment of this disclosure is shown;

[0086] Figure 4 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0087] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0088] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0089] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0090] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0091] Research has revealed that current human-machine conflict identification methods are limited to passive responses and localized optimizations, often based on fixed patterns and rules. They fail to adequately consider the dynamic changes occurring during system operation, thus hindering a comprehensive understanding of the nature and influencing factors of human-machine conflicts. For example, they cannot flexibly adjust identification strategies when faced with different operating scenarios, operator states, and environmental conditions, resulting in inaccurate and untimely identification of human-machine conflicts. Furthermore, single-dimensional evaluation—assessing human-machine conflicts from only a single perspective, such as focusing solely on operator behavior data or system performance indicators without comprehensive analysis of multiple dimensions—failes to fully grasp the nature and influencing factors of human-machine conflicts, thereby impacting the effectiveness and reliability of the identification methods.

[0092] In view of this, embodiments of this disclosure provide a human-machine conflict prediction method and apparatus, capable of proactively predicting human-machine conflicts from multiple dimensions. By acquiring and analyzing various physiological and behavioral data of operators in historical human-machine interaction scenarios, significant characterization factors corresponding to fatigue states are screened out, enabling a more accurate quantitative representation of personnel states. Simultaneously, the significant characterization factors of operators are combined with environmental feature dimensions and machine feature dimensions (including the complexity of interface interaction operations) to establish a human-machine conflict prediction model suitable for the target scenario. This method can dynamically reflect changes in multiple factors during system operation, proactively predicting the probability of human-machine conflict occurrence, thereby improving the comprehensiveness, timeliness, and reliability of identification, avoiding lag and bias caused by single data or rules, and ultimately contributing to improving the safety and stability of complex human-machine systems.

[0093] The deficiencies of the existing technical solutions are the result of the inventors' practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventors' contributions to this disclosure.

[0094] To facilitate understanding of this embodiment, a human-computer conflict prediction method disclosed in this disclosure will first be described in detail. The execution subject of the human-computer conflict prediction method provided in this disclosure is generally a computer device with certain computing power.

[0095] See Figure 1 The diagram shown is a flowchart of a human-machine conflict prediction method provided in an embodiment of this disclosure. The method includes:

[0096] S101. Obtain various physiological and behavioral data of testers in historical target human-computer interaction scenarios.

[0097] Human-machine interaction (HMI) scenarios refer to specific situations in which operators (humans) interact with systems or equipment (machines) during the operation of a particular task or system. For example, in a car driving scenario, HMI could be the interaction between a driver and an automated driving assistance system while driving on a highway, including the relationship between the driver's steering and braking actions and the vehicle's automatic control system's responses. In an aviation scenario, HMI could be the interaction between a pilot and an autopilot in complex weather conditions, such as a pilot attempting to manually correct the course. In an industrial control scenario, HMI could be an operator adjusting equipment parameters via a control console in a monitoring room.

[0098] For example, the aforementioned physiological and behavioral data may include:

[0099] Electroencephalogram (EEG) signals include time-domain and frequency-domain signals. The time domain includes whole-brain EEG at different time periods, while the frequency domain analyzes the energy distribution of different frequency components, such as alpha waves, theta waves, beta waves, and gamma waves, and performs conversions between frequency components, such as theta / beta.

[0100] Eye characteristics, including saccade count, saccade duration, saccade amplitude, saccade distance, blink count, blink frequency, total fixation time, average fixation time, fixation count, pupil diameter, and eye aspect ratio;

[0101] ECG data, including heart rate and heart rate variability (HRV), includes time-domain metrics such as Standard Deviation of Neural Intervals (SDNN) (NN interval standard deviation), RMSSD (root mean square of the difference between consecutive NN intervals), and pNN50 (the proportion of logarithmic NN interval differences greater than 50 milliseconds). Frequency-domain metrics include LF (Low Frequency), HF (High Frequency, reflecting the effect of respiratory rhythm on heart rate), and LF / HF Ratio (the ratio of low frequency to high frequency, used to assess sympathetic-parasympathetic balance).

[0102] Facial expression data can be categorized into neutral, surprised, happy, sad, fearful, disgusted, and angry. Further, these can be divided into positive emotions (mean of surprise and happiness) and negative emotions (mean of sadness, fear, disgust, and anger).

[0103] The aforementioned target human-computer interaction scenarios could be driving scenarios in transportation systems, flight missions in flight simulators, or monitoring and operation tasks in industrial control consoles, etc., using sensor devices and monitoring platforms to collect multimodal data from operators.

[0104] S102. Based on the physiological and behavioral data, determine a variety of data types associated with fatigue state, and use the data types as significant characterization factors corresponding to the fatigue state.

[0105] In this step, data processing and feature selection methods, such as principal component analysis, correlation analysis, cluster analysis, and feature importance assessment (e.g., based on random forest), can be used to screen out indicators that are significantly related to the operator's fatigue state.

[0106] For example, if heart rate variability, eye movement persistence, and operational delay are found to significantly characterize fatigue, these indicators can be used as significant characterizing factors of fatigue. This approach avoids introducing redundant data and improves the accuracy and efficiency of subsequent modeling.

[0107] After obtaining the initial physiological and behavioral data, these data can be preprocessed to obtain data that can be used in subsequent steps.

[0108] For example, eye-tracking data can be processed using standardized eye-tracking methods to clean the raw eye-tracking data, remove invalid data, and use a velocity-based I-VT fixation point extraction algorithm to extract states such as fixation point, blink, and saccade. Through a series of operations such as velocity calculation, merging adjacent fixations, and discarding abnormal fixations, fixation point extraction can be further optimized. Principal component analysis can be used to remove electrooculography interference, such as artifacts caused by blinking or muscle activity, and a velocity threshold algorithm can be used to filter saccade artifacts.

[0109] ECG data can be purified by bandpass filtering (e.g., 0.5-30Hz) and possible notch filtering to remove baseline drift, power line interference, and high-frequency noise. Next, algorithms such as Pan-Tompkins are used to accurately detect the R-wave position, and abnormal beats are eliminated based on dynamic thresholds or morphological rules to ensure the accuracy of rhythm analysis. Subsequently, independent component analysis or adaptive filtering is used to remove artifacts from electrooculography (EOG) and other sources. The processed continuous signal is then divided into meaningful segments based on R-wave detection or experimental events, and these segments are finally standardized (e.g., Z-score) to eliminate amplitude differences.

[0110] EEG signals can be labeled with event tags at corresponding time points according to experimental conditions, and the data can be divided into different segments based on these tags. Next, a finite impulse response (FIR) filter is used to perform high-pass (0.5Hz) and low-pass (49Hz) filtering on the data to remove low-frequency and high-frequency noise. Subsequently, whole-brain average rereference calculation is performed to eliminate common-mode interference. Potential bad leads are identified by observing the EEG signal heatmap, and data replacement is performed using an interpolation algorithm. Finally, independent component analysis is performed to decompose the EEG signal into multiple independent components, and artifacts such as mains voltage, eye movement, and electromyography (EMG) are identified and removed. The results are then analyzed in the time-frequency domain using a short-time Fourier transform algorithm to extract the power in the 4-8Hz (θ wave) and 13-30Hz (β wave) frequency bands at each time point, and the θ / β wave power ratio is extracted (θ waves reflect high workload, β waves reflect focused state, and the ratio is used to assess cognitive workload).

[0111] Facial expression data can be used to perform face detection and key point localization on video frames using algorithms. Based on the localized facial key points, active facial action units (AUs) in each frame are automatically detected. The appearance, disappearance, and intensity changes of these AUs are tracked over time to construct an AU dynamic transition matrix. Transient micro-expression (<500ms) conflict features are detected. Combining machine learning algorithms and image recognition technology, and considering the intensity, combination pattern, and geometric relationship of facial key points, expression classification is performed.

[0112] To facilitate data use, the preprocessed data can be fused into multimodal data to form an optimized physiological signal and behavioral dataset.

[0113] The aspect ratio of the eye can be determined based on an image of the eye. For example, see [link to example]. Figure 2 As shown, Figure 2 This is a schematic diagram of an eye image provided in an embodiment of this disclosure. A two-dimensional coordinate system can be established with the center of the eyeball in the eye image as the origin. A set of eye feature points is obtained based on this two-dimensional coordinate system. The set of eye feature points includes a set of feature points for the left eye and a set for the right eye; unilateral eye feature points are shown below. Figure 2 The diagram may include P1, P2, P3, P4, P5, and P6. The aspect ratio R of the eye can then be expressed as:

[0114]

[0115] The aforementioned eye aspect ratio represents the length-to-width ratio of the eye. When fatigued, operators' eyes may be closed or partially closed to varying degrees, causing changes in eye shape. By analyzing the eye aspect ratio, abnormal eye conditions can be detected. This method is used to detect blinking in facial data. When the eyes are open, the eye aspect ratio fluctuates within a certain range. When the eyes are closed, the eye aspect ratio drops rapidly. If the eye aspect ratio value is below 0.1, the eyes are considered closed. If the eye aspect ratio rapidly drops from a certain value to below 0.1 and then rapidly rises above a threshold, it is considered a blinking state.

[0116] It's important to note that the purpose of extracting the theta / β wave power ratio is that theta waves are closely related to memory encoding, learning processes, attention allocation, and brain resource integration. When cognitive load increases (such as complex task processing or information overload), theta wave power usually increases significantly, reflecting that the brain is using more neural resources for information processing. β waves are related to motor control, cognitive activity in a waking state, and concentration. Under low cognitive load, β wave power is high, indicating that the brain is in a highly efficient and focused state; however, when the load is too high, β wave power may decrease, reflecting inhibition of neural activity or fatigue. Under low load: β waves dominate, and the theta / β ratio is low; under moderate to high load: theta waves increase, β waves decrease, and the theta / β ratio increases significantly. This dynamic change can sensitively distinguish different load levels; therefore, the theta / β wave power ratio is used as an indicator of cognitive load.

[0117] Therefore, the above physiological and behavioral data can reflect the operator's fatigue state to a certain extent.

[0118] In one possible implementation, the physiological and behavioral data can be standardized to obtain standard data under multiple standard data types; then, based on the standard data and a forward stepwise regression model, multiple first target data types that affect fatigue state are selected from the standard data types; finally, the variance inflation factor of the first target data type is determined, and based on the variance inflation factor, multiple second target data types that affect fatigue state are selected from the first target data type.

[0119] In this step, physiological and behavioral data from different sources may have different dimensions and numerical ranges. For example, directly adding or comparing indicators such as heart rate (bpm), skin conductance (μS), and reaction time (ms) may lead to deviations. Therefore, it is necessary to standardize the data.

[0120] Using the aforementioned standard data, a forward stepwise regression model can be constructed to statistically test the relationship between various data types and operator fatigue status.

[0121] In this process, the model starts from an empty model, and each time the most significant independent variable is introduced, features are gradually added until the newly added variables no longer significantly improve the explanatory power of the model.

[0122] This method allows for the selection of multiple primary target data types that are significantly related to fatigue states from standard data types, such as heart rate variability, eye movement duration, and operational reaction delay.

[0123] For the selected first target data type, its variance inflation factor (VIF) is further calculated. The VIF value can be used to measure multicollinearity among different variables. When the VIF value of a certain data type exceeds a preset threshold (such as 5), it indicates that the variable has a strong correlation with other variables, which may cause the model estimation to be unstable.

[0124] Therefore, the first target data type can be screened a second time based on the VIF value to remove redundant features and obtain the second target data type that is finally used to characterize the fatigue state.

[0125] When performing forward stepwise regression, the data can be transformed using the Box-Cox transformation to amplify the difference between outliers and the baseline.

[0126] The standardization process described above may include the aforementioned preprocessing. In some possible implementations, specific data may also be specially processed to obtain new data.

[0127] For example, physiological and behavioral data include electroencephalogram (EEG) data (i.e., brain signals) and eye-tracking data; standardization processing may include:

[0128] Determine the causal relationship between the EEG data and the eye movement data; if a causal relationship exists between the EEG data and the eye movement data, combine the EEG data and the eye movement data to obtain standard data under the EEG-eye movement data type.

[0129] When a causal relationship exists between EEG and eye-tracking data, the two types of data can be combined to form a new composite data type: the EEG-eye-tracking data type. This standard data can better characterize the neural and behavioral coupling features of operators under fatigue.

[0130] In this step, the causal relationship between modalities can be determined by comparing the autoregressive model with the joint model, reducing misclassification of single modalities. After clarifying the causal relationship between eye movement and EEG data, feature combinations with real correlations can be prioritized for retention during multimodal data fusion, while synchronous abnormal data without actual causal relationship can be removed, improving the effectiveness of the fused dataset and providing high-quality input for subsequent models.

[0131] In one possible implementation, ECG data and facial expression data can be time-aligned, and both can be processed into time series. A dynamic time warping algorithm is used to calculate the optimal alignment path between the two time series, obtaining the best matching path. The time alignment of the ECG data and facial micro-expression data is then adjusted based on the best matching path to obtain optimized data. Precise adjustment of the time alignment between the ECG data and facial micro-expression data yields the optimized modal relationship.

[0132] In the process of identifying significant representative data, we can use elastic network regression, balance the L1 / L2 regularization of the forward stepwise regression model, and determine the optimal parameters of the multi-source regression model through leave-one-out cross-validation to construct an optimized multi-source regression model (i.e., the aforementioned forward stepwise regression model).

[0133] The aforementioned forward stepwise regression model can be simulated using particle filtering combined with Monte Carlo simulation to simulate extreme conflict scenarios, calculate KL divergence to assess model bias, and adjust the model accordingly.

[0134] For example, attention mechanisms can be introduced to analyze feature importance, mark conflict risk points, improve model interpretability, and facilitate model adjustment.

[0135] S103. Based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario, construct a human-computer conflict prediction model corresponding to the target human-computer interaction scenario; the machine feature dimension includes the complexity of interface interaction operations.

[0136] In this step, the environmental characteristics dimension can include external factors such as noise level, lighting conditions, task duration, and workload; the machine characteristics dimension can include task interface layout, number of task steps, operation feedback delay, and operation complexity, among which operation complexity can be quantified by the number of operation steps, operation path length, or information processing difficulty.

[0137] Based on this, a predictive model is established by combining significant representative factors describing human condition (fatigue-related indicators), environmental characteristics, and machine characteristics, using machine learning or deep learning methods (such as support vector machines, random forests, convolutional neural networks, and long short-term memory networks). This predictive model can reflect the comprehensive impact of different dimensional factors on the risk of human-machine conflict.

[0138] For example, the human-computer conflict prediction model includes a first sub-model and a second sub-model; the step of using the human-computer conflict prediction model to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario includes:

[0139] In the current target human-computer interaction scenario, acquire the operator's salient representation data under the salient representation factor; input the salient representation data into the first sub-model to predict the operator's fatigue state; input the fatigue state, the environmental features of the target human-computer interaction scenario, and the machine features of the target human-computer interaction scenario into the second sub-model to predict the probability of human-machine conflict occurring in the target human-computer interaction scenario.

[0140] In this way, we can first use significant characterization factors and the first sub-model to predict the operator's fatigue state, and then predict the probability of human-machine conflict based on fatigue state, environmental characteristics, machine characteristics, and the second sub-model, thereby improving accuracy.

[0141] The first sub-model can be constructed through the following steps:

[0142] Based on the salient representation factors, a salient representation factor feature vector is constructed; based on an initial long short-term memory neural network, a first sub-model is constructed; the first sub-model includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the salient representation factor feature vector; the hidden layer is used to learn the temporal features of the salient representation factor feature vector; the output layer is used to output the fatigue state of the operator.

[0143] In this step, feature vectors can be constructed based on salient representation factors to form the input sequence. Then, an initial Long Short-Term Memory (LSTM) neural network is used as the prediction framework. The LSTM network includes an input layer, hidden layers, and an output layer: the input layer receives the feature vectors of salient representation factors; the hidden layer learns the time-series features of the input data through memory units and gating mechanisms, avoiding the gradient vanishing problem that occurs in traditional neural networks during training on long-sequence data; the output layer outputs the prediction result, i.e., the operator's fatigue state (e.g., alert, mildly fatigued, severely fatigued).

[0144] When training the LSTM network, supervised learning can be used to optimize the network weight parameters based on historical sample data labeled with fatigue levels.

[0145] The fatigue state output by the first sub-model is combined with environmental features (such as noise level, lighting conditions, and task load) and machine features (such as interface complexity and feedback delay) in the target human-computer interaction scenario and input into the second sub-model to predict the probability of human-computer conflict occurring in the target scenario.

[0146] The second sub-model can be constructed through the following steps:

[0147] Multiple Bayesian network nodes are constructed; these nodes represent the fatigue state, environmental features of the target human-computer interaction scenario, and machine features of the target human-computer interaction scenario. Based on the dependencies between the Bayesian network nodes, directed edges connecting them are determined; the starting position of each directed edge is the parent node, and the ending position is the child node. The prior probability of each Bayesian network node and the conditional probability of each directed edge are determined; the prior probability represents the probability that each Bayesian network node takes a specific state without considering other Bayesian network nodes; the conditional probability represents the probability that a child node takes a specific state given the state of its parent node. An initial second sub-model is constructed based on the Bayesian network nodes, the directed edges, the prior probabilities, and the conditional probabilities. The initial second sub-model is trained to obtain the second sub-model.

[0148] Multiple Bayesian network nodes are constructed to represent fatigue states, environmental features, and machine characteristics. Based on the dependencies between these nodes, directed edges are determined, with parent nodes representing the dependent variable and child nodes representing variables affected by them. The prior probabilities of each node and the conditional probabilities of the directed edges are determined. For example, the impact of fatigue states on operational error rates can be modeled using conditional probability distributions. Based on the nodes, edges, prior probabilities, and conditional probabilities, an initial Bayesian network structure is constructed, forming the second sub-model. The initial Bayesian network is trained and its parameters updated using historical data to obtain the converged second sub-model.

[0149] For example, a Bayesian network consists of nodes and edges. Nodes represent random variables; here, nodes related to fatigue state and human error need to be defined. For instance, a "fatigue state" node can have two states: "fatigued" and "not fatigued"; a "human error" node (a type of human-computer conflict) can have two states: "error" and "no error". Furthermore, other relevant nodes can be introduced, such as "operational complexity" and "work environment," which may affect the probability of human error. Edges represent causal relationships between nodes. Based on domain knowledge and experience, the dependencies between nodes are determined. For example, "fatigue state" may directly affect "human error," so a directed edge can be drawn from the "fatigue state" node to the "human error" node. Similarly, "operational complexity" and "work environment" may also affect "fatigue state" and "human error," and edges can be added accordingly.

[0150] To construct a Bayesian network, necessary data can be obtained, such as fatigue data of different operators in different scenarios, and records of operator errors during actual operation, including the type of error, the time of occurrence, and the task being performed. Data can be obtained from sources such as system logs and records.

[0151] When determining prior probabilities, the prior probability of each node can be estimated based on collected data. For example, the probability of an operator being fatigued can be statistically analyzed based on historical data, along with the probability distribution of fatigue under different operational complexities and working environments. For nodes connected by directed edges, the conditional probability distribution can be estimated. For instance, given that the operator is fatigued, the probability of human error can be calculated; given operational complexity and working environment, the probability of fatigue can be calculated, and so on. Maximum likelihood estimation and other methods can be used to estimate these conditional probabilities.

[0152] In this step, a Bayesian network model can be constructed using the collected data and estimated conditional probability distribution. A subset of data can then be used to validate the constructed Bayesian network model, evaluating its accuracy and reliability. Methods such as cross-validation can be used for model validation. Models that pass validation can be applied in practice, while those that fail require further training.

[0153] S104. Using the human-computer conflict prediction model, predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario.

[0154] In practical applications, physiological and behavioral data of operators in the current scenario can be obtained and combined with current environmental parameters and machine parameters, and then input into the constructed human-machine conflict prediction model to output the probability value of human-machine conflict.

[0155] For example, in car driving scenarios, the system can monitor the driver's eye movements, heart rate, and operation frequency in real time, and combine road environment (such as weather and traffic flow) and vehicle interaction characteristics (such as the operational complexity of driver assistance functions) to predict and prompt potential human-machine conflicts (such as operational contradictions between the driver and the autonomous driving system).

[0156] In some possible implementations, the human-machine conflict prediction method may further include an adaptive intervention step to reduce the probability of human-machine conflict when the risk of conflict is high.

[0157] If the predicted probability of a human-computer conflict exceeds a target threshold, the system can automatically switch the interface of the target human-computer interaction scenario to a simplified interface. The target threshold can be a critical value preset based on historical experimental data or safety standards, such as 0.6 or 0.7. Compared to the original interface, the simplified interface reduces redundant information and operational complexity, thereby alleviating the cognitive load on the operator.

[0158] In one possible implementation, the simplified interface can be generated through the following steps:

[0159] The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface. Multiple initial simplified interfaces are generated using a generative adversarial network. Based on the distance of the target operation button from the center of the interface and the size information of the target operation button in the initial simplified interface, the operation efficiency of the initial simplified interface is determined. Based on the operation efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the multiple initial simplified interfaces.

[0160] For example, the pre-labeled important operation buttons could be "Emergency Braking" or "Confirmation Command". The human-computer conflict and the pre-labeled important operation buttons are encoded as conditional vectors. These vectors are then combined with the layout parameters of the current human-computer interaction interface to generate an input feature matrix. A conditional generative adversarial network (GAN) is used, with the conditional vectors serving as additional input to the generator to constrain the generation direction. The input feature matrix is ​​then sent to the generator to generate candidate interface layouts. The generator's goal is to generate an interface centered on the "pre-labeled important operation buttons" and simplifying other information.

[0161] In this step, the operation efficiency can be expressed as S = log2(D / W + 1). Here, S is the operation efficiency index; D represents the distance from the center of the interface; and W represents the size of the control. A smaller S value indicates a faster operation.

[0162] Since the operator is in an abnormal state, an overly complex interface is more likely to cause misoperation. Therefore, this step also includes evaluating the information density of the interface (the number of controls included in the interface) and selecting the interface with the lowest density and lowest operation efficiency index from the candidate interface layouts as the optimized interface.

[0163] In one possible implementation, important operation buttons in the simplified interface can be highlighted with a prominent color (such as red) to effectively stimulate the operator's senses and further reduce the probability of human error.

[0164] In another embodiment, when the probability of a current human-machine conflict is predicted to be high, the system can also perform task optimization steps:

[0165] Based on the probability of human-computer conflict, the task difficulty in the current target human-computer interaction scenario, the machine performance information in the current target human-computer interaction scenario, and the trained task optimization model, a task optimization strategy is determined; the task optimization model is trained based on the near-end strategy optimization model.

[0166] Based on the task optimization strategy, the operator's tasks in the current target human-computer interaction scenario are optimized; the optimization includes tasks with full machine takeover, tasks with partial machine takeover, and tasks that maintain manual operation.

[0167] For example, a task optimization model is trained based on the Proximal Policy Optimization (PPO) algorithm. Model inputs include conflict probability, task difficulty, and machine performance information; the model output is the optimal task allocation strategy. The system generates the current optimal task optimization strategy based on the prediction results and the trained model. The reward function can be set according to task completion status; for example, a successful task can be rewarded with +10, and a reduction in latency with +5, as positive rewards; a failed task can be rewarded with -20, and a misoperation with -10, as negative rewards.

[0168] Based on the task optimization strategy, the system can adopt one of the following three methods:

[0169] Machines take full control of tasks: When the risk of conflict is extremely high or the difficulty of the task exceeds the human capacity, the machine takes over directly; Machines partially take over tasks: When the risk of conflict is moderate, the machine and the operator work together to complete the task; Human operation is maintained: When the risk is controllable, the operator still completes the task to maintain operational proficiency and a sense of control.

[0170] Through the above embodiments, not only can the probability of human-computer conflict be predicted, but also proactive intervention can be carried out through interface adaptive optimization and task allocation optimization when the risk is too high, thereby effectively reducing the risk of human-computer conflict and improving the overall security and reliability of the system.

[0171] The human-machine conflict prediction method provided in this disclosure can proactively predict human-machine conflicts from multiple dimensions. By acquiring and analyzing various physiological and behavioral data of operators in historical human-machine interaction scenarios, significant characterization factors corresponding to fatigue states are screened out, enabling a more accurate quantitative representation of personnel states. Simultaneously, the significant characterization factors of operators are combined with environmental and machine characteristic dimensions (including the complexity of interface interaction operations) to establish a human-machine conflict prediction model suitable for the target scenario. This method can dynamically reflect changes in multiple factors during system operation, proactively predicting the probability of human-machine conflicts, thereby improving the comprehensiveness, timeliness, and reliability of identification, avoiding lag and bias caused by single data or rules, and ultimately contributing to improving the safety and stability of complex human-machine systems.

[0172] Corresponding to the embodiments of the aforementioned human-machine conflict prediction method, this disclosure also provides embodiments of a human-machine conflict prediction device.

[0173] See Figure 3 The diagram shown is a schematic representation of a human-machine conflict prediction device provided in an embodiment of this disclosure. The device includes:

[0174] The acquisition module 310 is used to acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios.

[0175] The determination module 320 is used to determine multiple data types associated with fatigue state based on the physiological and behavioral data, and to use the data types as significant characterization factors corresponding to the fatigue state.

[0176] The construction module 330 is used to construct a human-computer conflict prediction model corresponding to the target human-computer interaction scenario based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario; the machine feature dimension includes the complexity of interface interaction operation.

[0177] The prediction module 340 is used to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario using the human-computer conflict prediction model.

[0178] Optionally, the human-machine conflict prediction model includes a first sub-model and a second sub-model; the prediction module 340 is specifically used for:

[0179] In the current target human-computer interaction scenario, acquire the operator's significant representation data under the aforementioned significant representation factors;

[0180] The significant characterization data is input into the first sub-model to predict the operator's fatigue state;

[0181] The fatigue state, environmental features of the target human-computer interaction scenario, and machine features of the target human-computer interaction scenario are input into the second sub-model to predict the probability of human-computer conflict occurring in the target human-computer interaction scenario.

[0182] Optionally, the building module 330 is specifically used for:

[0183] Based on the aforementioned significant characterization factors, a feature vector of significant characterization factors is constructed;

[0184] Based on the initial long short-term memory neural network, a first sub-model is constructed; the first sub-model includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the feature vector of the significant representation factor; the hidden layer is used to learn the temporal features of the feature vector of the significant representation factor; and the output layer is used to output the fatigue state of the operator.

[0185] Optionally, the building module 330 is specifically used for:

[0186] Construct multiple Bayesian network nodes; the Bayesian network nodes are used to represent the fatigue state, the environmental features of the target human-computer interaction scenario, and the machine features of the target human-computer interaction scenario;

[0187] Based on the dependencies between the Bayesian network nodes, directed edges connecting the Bayesian network nodes are determined; the starting position of the directed edge is the parent node, and the ending position is the child node.

[0188] Determine the prior probability of each Bayesian network node and the conditional probability of the directed edge; the prior probability is used to represent the probability that each Bayesian network node takes a specific state without considering other Bayesian network nodes; the conditional probability is used to represent the probability that the child node takes a specific state given the state of the parent node.

[0189] Based on the Bayesian network nodes, the directed edges, the prior probabilities, and the conditional probabilities, an initial second sub-model is constructed;

[0190] The initial second sub-model is trained to obtain the second sub-model.

[0191] Optionally, the determining module 320 is specifically used for:

[0192] The physiological and behavioral data are standardized to obtain standard data under various standard data types;

[0193] Based on the standard data and the forward stepwise regression model, a variety of first target data types that affect fatigue state are selected from the standard data types.

[0194] Determine the variance inflation factor of the first target data type, and based on the variance inflation factor, select a variety of second target data types that affect fatigue state from the first target data type.

[0195] Optionally, the physiological and behavioral data includes electroencephalogram (EEG) data and eye movement data;

[0196] The determining module 320 is specifically used for:

[0197] Determine the causal relationship between the electroencephalogram (EEG) data and the eye movement (EMG) data;

[0198] When there is a causal relationship between the EEG data and the eye movement data, the EEG data and the eye movement data are combined to obtain standard data under the EEG-eye movement data type.

[0199] Optionally, the physiological and behavioral data may also include at least one of the following:

[0200] Facial data, eye aspect ratio data, electroencephalogram (EEG) data, and electrocardiogram (ECG) data.

[0201] Optionally, the device further includes a switching module for:

[0202] If the probability of human-computer conflict is greater than the target threshold, the human-computer interaction interface in the current target human-computer interaction scenario will be switched to a simplified interface.

[0203] Optionally, the apparatus further includes a generation module for:

[0204] The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface, and multiple initial simplified interfaces are generated using a generative adversarial network.

[0205] Based on the distance of the target operation button from the center of the interface in the initial simplified interface, and the size information of the target operation button, the operation efficiency of the initial simplified interface is determined;

[0206] Based on the operational efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the plurality of initial simplified interfaces.

[0207] Optionally, the device further includes an optimization module for:

[0208] Based on the probability of human-computer conflict, the task difficulty in the current target human-computer interaction scenario, the machine performance information in the current target human-computer interaction scenario, and the trained task optimization model, a task optimization strategy is determined; the task optimization model is trained based on the near-end strategy optimization model.

[0209] Based on the task optimization strategy, the operator's tasks in the current target human-computer interaction scenario are optimized; the optimization includes tasks with full machine takeover, tasks with partial machine takeover, and tasks that maintain manual operation.

[0210] The human-machine conflict prediction device provided in this disclosure can proactively predict human-machine conflicts from multiple dimensions. By acquiring and analyzing various physiological and behavioral data of operators in historical human-machine interaction scenarios, significant characterization factors corresponding to fatigue states are screened out, enabling a more accurate quantitative representation of personnel states. Simultaneously, the significant characterization factors of operators are combined with environmental and machine feature dimensions (including the complexity of interface interaction operations) to establish a human-machine conflict prediction model suitable for the target scenario. This method can dynamically reflect changes in multiple factors during system operation, proactively predict the probability of human-machine conflicts, thereby improving the comprehensiveness, timeliness, and reliability of identification, avoiding lag and bias caused by single data or rules, and ultimately contributing to improving the safety and stability of complex human-machine systems.

[0211] This disclosure also provides a computer device, such as... Figure 4 The diagram shown is a schematic representation of a computer device structure provided in an embodiment of this disclosure, including:

[0212] Processor 41 and memory 42; the memory 42 stores machine-readable instructions executable by the processor 41, and the processor 41 executes the machine-readable instructions stored in the memory 42. When the machine-readable instructions are executed by the processor 41, the processor 41 performs the following steps:

[0213] Acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios;

[0214] Based on the physiological and behavioral data, various data types associated with fatigue state are identified, and these data types are used as significant characterization factors corresponding to the fatigue state.

[0215] Based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario, a human-computer conflict prediction model corresponding to the target human-computer interaction scenario is constructed; the machine feature dimension includes the complexity of interface interaction operations.

[0216] The human-computer conflict prediction model is used to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario.

[0217] The aforementioned memory 42 includes a main memory 421 and an external memory 422; the main memory 421, also known as internal memory, is used to temporarily store the computational data in the processor 41, as well as the data exchanged with external memory 422 such as a hard disk. The processor 41 exchanges data with the external memory 422 through the main memory 421.

[0218] The specific execution process of the above instructions can be referred to the steps of the human-machine conflict prediction method described in the embodiments of this disclosure, and will not be repeated here.

[0219] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0220] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the human-machine conflict prediction method described in the above-described method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0221] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the human-machine conflict prediction method provided in the various embodiments of this disclosure.

[0222] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0223] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0226] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0227] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

[0228] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for predicting human-machine conflict, characterized in that, The method includes: Acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios; Based on the physiological and behavioral data, various data types associated with fatigue state are identified, and these data types are used as significant characterization factors corresponding to the fatigue state. Based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario, a human-computer conflict prediction model corresponding to the target human-computer interaction scenario is constructed; the machine feature dimension includes the complexity of interface interaction operation; the complexity of interface interaction operation is quantified based on at least one of the number of operation steps, operation path length, or information processing difficulty. Using the aforementioned human-computer conflict prediction model, the probability of human-computer conflict occurring in the current target human-computer interaction scenario is predicted; If the probability of human-computer conflict is greater than the target threshold, the human-computer interaction interface in the current target human-computer interaction scenario will be switched to a simplified interface. The simplified interface is generated through the following steps: The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface. This vector is then combined with the layout parameters of the current human-computer interaction interface to generate an input feature matrix. Finally, a generative adversarial network is used to generate multiple initial simplified interfaces. Based on the distance of the target operation button from the center of the interface in the initial simplified interface, and the size information of the target operation button, the operation efficiency of the initial simplified interface is determined; Based on the operational efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the plurality of initial simplified interfaces.

2. The method according to claim 1, characterized in that, The human-computer conflict prediction model includes a first sub-model and a second sub-model; the step of using the human-computer conflict prediction model to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario includes: In the current target human-computer interaction scenario, acquire the operator's significant representation data under the aforementioned significant representation factors; The significant characterization data is input into the first sub-model to predict the operator's fatigue state; The fatigue state, environmental features of the target human-computer interaction scenario, and machine features of the target human-computer interaction scenario are input into the second sub-model to predict the probability of human-computer conflict occurring in the target human-computer interaction scenario.

3. The method according to claim 2, characterized in that, The first sub-model is constructed using the following steps: Based on the aforementioned significant characterization factors, a feature vector of significant characterization factors is constructed; Based on the initial long short-term memory neural network, a first sub-model is constructed; the first sub-model includes an input layer, a hidden layer, and an output layer; the input layer is used to receive the feature vector of the significant representation factor; the hidden layer is used to learn the temporal features of the feature vector of the significant representation factor. The output layer is used to output the operator's fatigue status.

4. The method according to claim 2, characterized in that, The second sub-model is constructed using the following steps: Construct multiple Bayesian network nodes; the Bayesian network nodes are used to represent the fatigue state, the environmental features of the target human-computer interaction scenario, and the machine features of the target human-computer interaction scenario; Based on the dependencies between the Bayesian network nodes, directed edges connecting the Bayesian network nodes are determined; the starting position of the directed edge is the parent node, and the ending position is the child node. Determine the prior probability of each Bayesian network node and the conditional probability of the directed edge; the prior probability is used to represent the probability that each Bayesian network node takes a specific state without considering other Bayesian network nodes; the conditional probability is used to represent the probability that the child node takes a specific state given the state of the parent node. Based on the Bayesian network nodes, the directed edges, the prior probabilities, and the conditional probabilities, an initial second sub-model is constructed; The initial second sub-model is trained to obtain the second sub-model.

5. The method according to claim 1, characterized in that, Based on the physiological and behavioral data, various data types associated with fatigue states are identified, including: The physiological and behavioral data are standardized to obtain standard data under various standard data types; Based on the standard data and the forward stepwise regression model, a variety of first target data types that affect fatigue state are selected from the standard data types. Determine the variance inflation factor of the first target data type, and based on the variance inflation factor, select a variety of second target data types that affect fatigue state from the first target data type.

6. The method according to claim 5, characterized in that, The physiological and behavioral data include electroencephalogram (EEG) data and eye movement data; The standardization process for the physiological and behavioral data includes: Determine the causal relationship between the electroencephalogram (EEG) data and the eye movement (EMG) data; When there is a causal relationship between the EEG data and the eye movement data, the EEG data and the eye movement data are combined to obtain standard data under the EEG-eye movement data type.

7. The method according to claim 1, characterized in that, The physiological and behavioral data also include at least one of the following: Facial data, eye aspect ratio data, electroencephalogram (EEG) data, and electrocardiogram (ECG) data.

8. The method according to claim 1, characterized in that, The method further includes: Based on the probability of human-computer conflict, the task difficulty in the current target human-computer interaction scenario, the machine performance information in the current target human-computer interaction scenario, and the trained task optimization model, a task optimization strategy is determined; the task optimization model is trained based on the near-end strategy optimization model. Based on the task optimization strategy, the operator's tasks in the current target human-computer interaction scenario are optimized; the optimization includes tasks with full machine takeover, tasks with partial machine takeover, and tasks that maintain manual operation.

9. A human-machine conflict prediction device, characterized in that, The device includes: The acquisition module is used to acquire various physiological and behavioral data of testers in historical target human-computer interaction scenarios; The determination module is used to determine multiple data types associated with fatigue state based on the physiological and behavioral data, and to use the data types as significant characterization factors corresponding to the fatigue state. The construction module is used to construct a human-computer conflict prediction model corresponding to the target human-computer interaction scenario based on the significant characterization factor, the environmental feature dimension of the target human-computer interaction scenario, and the machine feature dimension of the target human-computer interaction scenario; the machine feature dimension includes the complexity of interface interaction operation; the complexity of interface interaction operation is quantified based on at least one of the number of operation steps, operation path length, or information processing difficulty. The prediction module is used to predict the probability of human-computer conflict occurring in the current target human-computer interaction scenario using the human-computer conflict prediction model. The switching module is used to switch the human-computer interaction interface in the current target human-computer interaction scenario to a simplified interface when the probability of human-computer conflict is greater than the target threshold. Generate modules for: The probability of human-computer conflict is encoded into a condition vector along with the target operation button in the human-computer interaction interface. This vector is then combined with the layout parameters of the current human-computer interaction interface to generate an input feature matrix. Finally, a generative adversarial network is used to generate multiple initial simplified interfaces. Based on the distance of the target operation button from the center of the interface in the initial simplified interface, and the size information of the target operation button, the operation efficiency of the initial simplified interface is determined; Based on the operational efficiency and information density of the initial simplified interface, the simplified interface for switching is selected from the plurality of initial simplified interfaces.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8.

Images