Multi-level human-computer interaction intention recognition method oriented to complex task situation

Abstract

The invention discloses a multi-level man-machine interaction intention recognition method oriented to a complex task situation, relates to the technical field of man-machine interaction intention recognition, and solves the technical problems that multi-level man-machine interaction intention recognition under the complex task situation is low in accuracy and low in task stage interpretation. According to the technical scheme, the multi-level intention recognition framework suitable for the complex task situation is provided, a task analysis method and a machine learning method are combined, and sensitivity to human behaviors and contextual information is achieved, and interpretability is achieved. The input data is the behavior data of a person which is easy to capture and the situation data which can be acquired by a machine sensor, and the method has the advantage and operability of not interfering the working process of a man-machine system. By simultaneously fusing the behavior, environment and task context features of the person, the multi-level intention of the person is recognized from bottom data layer by layer to top, the system can comprehensively understand the real intention of the person, the limitation of the existing method in the aspects of intention recognition from a single level and only according to the behavior features of the person and the like is effectively solved, and the user experience is improved. The accuracy of multi-level intention recognition is improved, and meanwhile interpretable modeling of the multi-level intention is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of human-computer interaction intention recognition, and in particular to a multi-level human-computer interaction intention recognition method for complex task scenarios. Background Art

[0002] With the development of machine intelligence, human-machine interaction has gradually developed into human-intelligent system interaction (Duric Z, Gray WD, Heishman R, et al. Integrating perceptual and cognitive modeling for adaptive and intelligent human-computer interaction [J]. Proceedings of the IEEE, 2002, 90 (7): 1272-1289.), and this kind of interaction must be based on mutual understanding and trust between the two parties. Therefore, realizing the recognition of human intentions by intelligent systems is the key foundation of intelligent human-computer interaction. Intent recognition is a technology that uses internal sensors of the system to perceive the contextual information of user behavior, system and environment, thereby inferring human intentions or predicting their next actions. At present, intent recognition technology has been widely studied and applied to many human-computer interaction scenarios such as ubiquitous computing, human-computer collaboration, intelligent driving and operator-centric complex systems.

[0003] In reality, many real work scenarios involve complex task situations. In the process of complex tasks, there is close information interaction and mutual influence between people, systems and the environment. Changes in human status, system status and environmental status, that is, contextual information, will cause changes in people's current task intentions. Existing research on intention recognition for complex task scenarios often recognizes the intention of complex tasks from a single scale, which limits its usability. By observing the sequence of people's interactive actions within a certain period of time, people's simple interactive intentions can be identified, but the complex task intentions reflected behind the intentions cannot be represented; the complex task intentions inferred from people's hand, eye and other interactive behaviors lack interpretability and have a greater risk of misjudgment of intentions.

[0004] In fact, in the human-computer interaction system with complex task scenarios, human intentions show the characteristics of multi-level structure and strong correlation with context. The context features include environmental situation context and task context. Environmental situation context represents the machine system data, environmental data, and situation data that are closely related to the task scenario, and is an external factor that causes human intention transfer. Task context represents the task process performed by people, which is related to the task procedure and human ability and preference. It can be considered as an internal factor of human intention transfer and provides a priori rules for reasoning about human intention. For multi-level intention recognition of complex tasks, behavioral features and context features are indispensable and can provide the necessary information for intention recognition.

[0005] In the prior art, task analysis methods suitable for complex tasks include Hierarchical Task Analysis (HTA), Goal-Directed Task Analysis (GDTA), Cognitive Task Analysis (CTA), and Hierarchical Goal Analysis (HGA). HTA (Stanton N A. Hierarchical task analysis: Developments, applications, and extensions [J]. Applied ergonomics, 2006, 37 (1): 55-79.) decomposes complex tasks into smaller and smaller subtasks through a hierarchical structure diagram, so it is suitable for task hierarchy sorting of a wide range of tasks. GDTA (Nasser-dine A, A, Lapalme J. Does explicit categorization taxonomy facilitate performing goal-directed task analysis? [J]. IEEE Transactions on Human-Machine Systems, 2021, 51(3): 177-187.) Based on Endsley's three-level situational awareness theory, it aims to decompose the goal into sub-goals based on the target task, and then sort out the situational awareness variables under each sub-goal. This method focuses on sorting out the situational awareness needs of people in the process of performing complex tasks. CTA (Lercel D, Andrews DH. Cognitive task analysis of unmanned aircraft system pilots [J]. The International Journal of Aerospace Psychology, 2021, 31(4): 319-342.) emphasizes starting from the cognitive level of people, breaking down cognitive tasks into low-level task units and analyzing the cognitive needs of each low-level task unit. Compared with HTA, HGA based on perception control theory (Kobierski B. Hierarchical Goal Analysis and performance modelling for the control of multiple UAVs / UCAVs from an airborne platform [R]. DRDC Technical Report CR 2004-063 (DRDC Toronto), 2004.) emphasizes the hierarchical structure of task goals rather than the disassembly of the task itself. In addition, HGA can decompose task goals without assigning operators, so even if the operator role changes, the overall task goal hierarchy does not need to be modified. In addition to the top-down task analysis method mentioned above, some studies have conducted task analysis from a bottom-up perspective through Operational Sequence Diagrams (OSDs) to sort out the operator's operation sequence in the entire complex task process.

[0006] The input features of multi-level intent recognition mainly come from two aspects: relevant behavioral features of people and contextual features related to the task environment. In terms of behavioral features, existing research mainly focuses on the extraction of multimodal behavioral features of people. At present, many researchers have collected eye information (Lukander K, Toivanen M, K.Inferring intent and action from gaze in naturalistic behavior:A review[J].International Journal of Mobile Human Computer Interaction(IJMHCI),2017,9(4):41-57.), hand interaction actions (Wei D,Chen L,Zhao L,et al.A vision-based measure ofenvironmental effects on inferring human intention during human robotinteraction[J].IEEE Sensors Journal,2021,22(5):4246-4256.) and other human behavior information to infer intention. In addition to hand-eye behavior, some researchers have used physiological parameters such as EEG (Electroencephalography, EEG) (Wenbo Huang FA, Changyuan Wang SB, Hongbo Jia T C. Ergonomics analysis based on intention inference [J]. Journal of Intelligent & Fuzzy Systems, 2021, 41 (1): 1281-1296.), electromyography (Electromyography, EMG) (Feleke AG, Bi L, Fei W. EMG-based 3D hand motor intention prediction for information transfer from human to robot [J]. Sensors, 2021, 21 (4): 1316.), and heart rate (Wang H, Pan T, Si H, et al. Research on influencing factor selection of pilot's intention [J]. International Journal of Aerospace Engineering, 2020, 2020 (1): 4294538.) to assist intention recognition. In terms of task environment context features, existing research mainly focuses on environmental situation context and task context. Environmental situation context represents environmental data and situation data that are closely related to the task situation and is an external factor that causes the operator's intention transfer.Some researchers (Xia J, Chen M, Fang W. Air combat intention recognition with incomplete information based on decision tree and GRU network [J]. Entropy, 2023, 25 (4): 671.) use machine system data as a type of situation context. For example, in the process of identifying the pilot's intention, the aircraft's altitude, speed, pitch angle, heading and other machine system information are also included in the recognition model as an important situation context feature. Task context represents the task process performed by the operator, which is related to the task procedure and human capabilities and preferences. It can be considered as an intrinsic factor in the transfer of human intentions, providing a priori rules for reasoning about human intentions. In general, for multi-level intention recognition of complex tasks, behavioral features and contextual features are indispensable, and both can provide the necessary information for intention recognition.

[0007] According to the interpretability of the model, the existing intent recognition algorithms can be divided into four types: based on expert knowledge, based on deep learning, based on machine learning, and based on probability theory. Intent recognition methods based on expert knowledge usually use expert-defined rules and template libraries, and judge intent by inferring the degree of match between feature information and intent templates through methods such as maximum similarity and DS evidence theory (Tang Z, Li S, Dou Y. Research on Pilot's Intention Reasoning Method Based on D–S Evidence Theory [C] / / Man–Machine–Environment System Engineering: Proceedings of the 17th International Conference on MMESE 17. SpringerSingapore, 2018: 23-31.). Such methods are fully interpretable, but require explicit organization, abstraction, and description of the experiential knowledge of domain experts. It is difficult to acquire and represent knowledge, and lacks flexibility with changes in the environment. Intent recognition methods based on deep learning are purely driven by feature data and aim to train neural networks with intent classification capabilities. Since human intentions are related to contextual time series, researchers generally use recurrent neural network (RNN) methods with time series processing capabilities, including: Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and Bi-directional Long Short-Term Memory (Bi-LSTM). With the improvement of general computing power, deep learning methods can often achieve high accuracy in intention recognition, but they are prone to overfitting, difficult to effectively integrate domain expert knowledge, and lack interpretability. Machine learning-based intention recognition methods are another type of method that is purely driven by feature data. They aim to use typical machine learning classifiers to complete intent classification. Commonly used classifiers include Support Vector Machine (SVM) and Random Forest (RF). Compared with deep learning neural network models, such methods are simpler and consume less computing resources, but they are often difficult to process time series data and large-scale data sets, and they are also not interpretable.The intention recognition method based on probability theory uses Bayesian theory to carry out uncertainty intention reasoning, such as Hidden Markov Model (HMM), Dynamic Bayesian Network (DBN), etc. This type of method belongs to the probabilistic graph model that describes the causal relationship of variables with the help of Directed Acyclic Graph (DAG). It can often define the network structure of the model through expert modeling and calculate the network parameters in a data-driven way, thereby realizing the intention recognition that integrates expert knowledge and data-driven.

[0008] How to accurately identify multi-level human-computer interaction intentions in complex task scenarios remains to be solved. Summary of the invention

[0009] The present application provides a multi-level human-computer interaction intention recognition method for complex task scenarios, and its technical purpose is to accurately recognize the multi-level human-computer interaction intention in complex task scenarios.

[0010] The above technical objectives of this application are achieved through the following technical solutions:

[0011] A multi-level human-computer interaction intention recognition method for complex task scenarios, including:

[0012] Step S1: Analyze the task objectives of complex tasks and extract feature categories of multi-level intentions;

[0013] Step S2: Conducting experiments in a simulation task environment, and collecting multi-level intention feature data sets of human behavior and context situation information according to feature categories;

[0014] Step S3: Train and test the intention recognition model through a multi-level intention feature data set, and use the trained intention recognition model to identify intentions from low-level to high-level layers from the underlying feature data to achieve multi-level human-computer interaction intention recognition.

[0015] Furthermore, the step S1 comprises:

[0016] Through hierarchical goal analysis, the goals of complex tasks are decomposed layer by layer, and the hierarchical task goals are matched with the intentions of people at multiple levels to obtain the hierarchical goal analysis results; the operation sequence of people in the complex task process is sorted out through the operation sequence diagram, and then the underlying features related to the multi-level intentions are sorted out according to the operation sequence to obtain the analysis results of the operation sequence diagram;

[0017] Extract high-level task intentions and low-level task interaction intentions based on the hierarchical goal analysis results;

[0018] According to the hierarchical goal analysis results and the operation sequence diagram analysis results, the relevant features of the high-level task intention and the low-level task interaction intention are extracted to obtain feature categories; wherein the relevant features of the high-level task intention and the low-level task interaction intention include behavioral features and contextual features, the behavioral features include behavioral data, and the behavioral data include hand behavior data, eye behavior data and human physiological data; the contextual features include system data and situation data, the system data includes local system data, local motion data and local alarm data, and the situation data includes geographic situation data, spatiotemporal situation data and environmental situation data.

[0019] Furthermore, the step S2 comprises:

[0020] Build complex mission scenarios based on the 3D engine;

[0021] According to the feature categories, multi-level intention feature data in the constructed complex task scenarios are extracted to obtain a multi-level intention feature dataset.

[0022] Furthermore, the step S3 comprises:

[0023] Step S31: performing a first preprocessing on the multi-level intention feature data set to obtain first preprocessed data, and dividing the first preprocessed data into a first training set and a first test set; training a low-level interaction intention recognition model by using the first training set, and testing the trained low-level interaction intention recognition model by using the first test set, until a final low-level interaction intention recognition model is obtained;

[0024] Step S32: performing a second preprocessing on the multi-level intention feature data set to obtain second preprocessed data, and dividing the second preprocessed data into a second training set and a second test set; training the high-level task intention recognition model through the second training set, and testing the trained high-level task intention recognition model through the second test set, until a final high-level task intention recognition model is obtained;

[0025] Step S33: Input the recognition result of the final low-level interaction intention recognition model into the final high-level task intention recognition model for further intention recognition, so as to realize multi-level human-computer interaction intention recognition.

[0026] Furthermore, in step S31, the multi-level intention feature data set is subjected to a first preprocessing to obtain first preprocessed data, including:

[0027] One-Hot encoding is used for discrete data in the multi-level intention feature data set to convert categorical variables in the discrete data into binary variables to obtain first feature data;

[0028] Normalizing the continuous data in the multi-level intention feature data set to unify the dimensions of the continuous data to obtain second feature data;

[0029] A time sliding window is used to divide the first feature data and the second feature data into continuous subsequences of a fixed size to obtain first preprocessed data.

[0030] Furthermore, in step S32, the multi-level intention feature data set is subjected to a second preprocessing to obtain second preprocessed data, including:

[0031] The data in the multi-level intention feature data set is discretely processed and divided into time windows to obtain second preprocessed data.

[0032] Furthermore, the low-level interaction intention recognition model includes a 1D-CNN module and a Bi-LSTM module, and the high-level task intention recognition model includes a dynamic Bayesian network structure.

[0033] The beneficial effects of this application are:

[0034] (1) The multi-level human-computer interaction intention recognition method for complex task scenarios proposed in this application proposes a multi-level intention recognition framework suitable for complex task scenarios, which combines task analysis methods with machine learning methods. It is sensitive to human behavior and contextual information and is interpretable. Its input data is situation data that can be easily captured by human and machine sensors, and has the advantages of not interfering with human work progress and operability. By simultaneously integrating human behavior, environment, and task context features, the multi-level intentions of people are identified layer by layer from the bottom-level data, achieving a more comprehensive understanding of the system's true intentions of people, and effectively solving the limitations of existing methods in that they conduct intention recognition from a single level and only based on human behavior characteristics.

[0035] (2) This method can ensure the accuracy of multi-level intention recognition for more categories, and at the same time realize the interpretable modeling of multi-level intentions. In the example scenario based on the helicopter-UAV wildfire exploration task, the method described in this application achieves an interactive intention recognition accuracy of 93.96% for 16 types of interactive intentions and a task intention recognition accuracy of 97.12% for 5 types of task intentions, ensuring the accuracy requirements of intention recognition. In addition, the "context-intention-behavior" dynamic Bayesian network in the method described in this application completes the interpretable modeling of multi-level intentions, which is in line with the intention generation mechanism in cognitive psychology theory, namely: when people perform complex tasks, they perceive the task situation context, generate intentions through brain cognitive processing, and the intentions drive people to perform a series of operational actions and ultimately drive the helicopter state changes.

[0036] (3) The multi-level intent recognition framework of the method described in this application is universal. Based on the basic framework of complex task scenario analysis, underlying feature data collection and multi-level intent recognition, the task analysis method and the specific algorithm model used in intent recognition are interchangeable. In the actual application process, the corresponding method can be selected in combination with the characteristics of each level of intent to achieve the best recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A framework diagram of a multi-level human-computer interaction intention recognition method for complex task scenarios in an embodiment of the present application;

[0038] Figure 2 A multi-level intent tree taking a helicopter-UAV wildfire exploration mission as an example in an embodiment of the present application;

[0039] Figure 3 This is a schematic diagram of multi-level intent recognition related features in an embodiment of the present application;

[0040] Figure 4 This is a framework diagram of a low-level interaction intention recognition model based on 1D-CNN+Bi-LSTM in an embodiment of the present application;

[0041] Figure 5 This is a framework diagram of a high-level task intent recognition model based on a dynamic Bayesian network in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the multi-level human-computer interaction intention recognition method for complex task scenarios described in this application includes:

[0044] Step S1: Analyze the task objectives of complex tasks and extract feature categories of multi-level intentions.

[0045] Complex task situation analysis is the first step in multi-level intention recognition in complex task human-computer interaction. Its purpose is to sort out the standard operating procedures of operators in performing complex tasks, and then analyze people's multi-level intentions and intention-related features, providing a basis for subsequent simulation task data collection and multi-level intention recognition models. Complex task situation analysis includes three parts: complex task analysis, extraction of multi-level intentions, and extraction of intention-related features.

[0046] In view of the correlation between human intention and complex task goals, this application uses a combination of Hierarchical Goal Analysis (HGA) and Operational Sequence Diagrams (OSDs) to carry out overall task analysis of complex tasks, and parses the tasks into a three-layer structure of "task intention-interaction intention-feature data". In the embodiment of this application, the helicopter-UAV wildfire exploration task is used as an example scenario of a complex task, and the specific process is described as follows:

[0047] First, we use hierarchical goal analysis to start from the overall root goal of the complex system, which is "wildfire exploration" in this example. We decompose the task goals layer by layer from a top-down perspective, decompose each goal layer by layer, correspond the hierarchical task goals to the intentions of people at multiple levels, and add controlled variables to each type of goal.

[0048] Afterwards, the operation sequence diagram is used to sort out the operation sequence of people in the entire task process, supplement the task analysis details from a top-down perspective, and sort out the underlying features related to multi-level intentions based on the operation sequence of people. The operation sequence diagram is used to sort out the logical order of tasks along the timeline process, and describes the task status of each operator through icons such as information transmission, information reception, operation, inspection, and decision.

[0049] Finally, human factors engineering experts and system domain experts jointly organized the analysis results of the hierarchical goal analysis and operation sequence diagram, and sorted out the multi-level intentions and intention-related feature classifications.

[0050] Multi-level intentions are extracted from the hierarchical target analysis results. The present invention simplifies and decomposes human intentions into two layers: high-level task intentions and low-level task interaction intentions. Taking the helicopter-UAV wildfire exploration task as an example scenario of a complex task, the schematic diagram of extracting multi-level intentions is shown in the following figure: Figure 2 shown.

[0051] The feature classification of intention-related features is obtained by combing through the results of hierarchical target analysis and operation sequence diagram analysis, and can be generally divided into two categories: human behavior features and context features. Human behavior features represent the relevant information of hands, eyes and other interaction modes when people perform tasks. Context features represent the human-machine system information and task environment information in complex tasks. This information is often represented by the display elements of the user interface, including system data and situation data. Taking the helicopter-UAV wildfire exploration task as an example scenario of a complex task, the extracted intention-related features are as follows: Figure 3The human behavior features include the operator's hand behavior data, eye behavior data, physiological data, etc. during the task. The context features include system data such as the local system data, local motion data, local alarm data, and situation data such as geographic situation information, spatiotemporal situation information, and environmental situation information during the task.

[0052] Step S2: Conduct experiments in a simulation task environment and collect multi-level intention feature data sets of human behavior and contextual situation information according to feature categories.

[0053] The collection of underlying feature data is the data basis for multi-level intention recognition in complex task human-computer interaction. Its purpose is to reproduce the task scenario in a laboratory environment based on the aforementioned complex task analysis, and to gather participants to collect feature data related to the operator's intention during the task execution, establish a multi-level intention feature data set, and provide data samples for subsequent algorithm research. Taking the helicopter-UAV wildfire exploration task as an example scenario of a complex task, the underlying data collection process is as follows:

[0054] First, we use Unity3D and other 3D engines to build and reproduce complex helicopter-UAV wildfire exploration mission scenarios, and use the engine’s built-in C# script to collect human hand behavior data, helicopter system data, motion data, and situation data at a fixed frequency. We also use an external eye tracker to collect human eye behavior data at a fixed frequency.

[0055] After that, we will enter the preparation stage before the underlying feature data collection experiment. The experimenter will introduce the experimental process to the participants in advance. After that, the participants will repeat the training for at least 3 trials. The formal experiment can only begin after the participants independently complete the entire task process under the supervision of the experimenter and verbally report that they have fully mastered the operation and task process.

[0056] Then, it entered the formal experiment stage. The formal experiment includes three parts: eye movement calibration, simulated flight tasks, and intention labeling. First, the subjects will perform eye movement calibration before the experiment begins and turn on screen recording; after that, the subjects will carry out a helicopter-UAV collaborative wildfire exploration task in a certain area. The subjects are required to abide by the basic task process, and the specific task implementation details and timing such as planning the UAV exploration area, selecting UAVs, judging the exploration results, and rescuing trapped people will be decided by the subjects themselves. Finally, after completing the exploration task of the entire area, the subjects are required to view and record the screen immediately, and use key marking to mark the starting and ending points of high-level task intentions and low-level task interaction intentions.

[0057] Finally, the experimental data are processed and the experimental data of all subjects are summarized. After data cleaning processes such as aligning timestamps, merging data, and filling in default values, they are finally summarized into a structured data set.

[0058] Step S3: Train and test the intention recognition model through a multi-level intention feature data set, and use the trained intention recognition model to identify intentions from low-level to high-level layers from the underlying feature data to achieve multi-level human-computer interaction intention recognition.

[0059] Specifically, step S3 includes:

[0060] Step S31: perform a first preprocessing on the multi-level intention feature data set to obtain first preprocessed data, and divide the first preprocessed data into a first training set and a first test set; train the low-level interaction intention recognition model through the first training set, and test the trained low-level interaction intention recognition model through the first test set until the final low-level interaction intention recognition model is obtained.

[0061] The low-level interaction intent recognition model uses a 1DCNN+Bi-LSTM deep learning neural network structure to obtain the most accurate multi-category interaction intent classification effect. Its framework diagram is as follows: Figure 4 shown.

[0062] First, preprocessing is performed on the input data. The discrete data in the structured multi-level intention feature data set collected in the experiment adopts One-Hot encoding, and the continuous data is normalized to improve the convergence speed and training efficiency of the neural network. Among them, One-Hot encoding converts categorical variables into binary variables. Each unique value of a given categorical variable will be represented as a vector composed of 0 and 1, and only one position is 1, and the rest are 0; normalization converts feature variables of different dimensions and magnitudes to the same scale between 0 and 1. In this application, the minimum-maximum normalization (Min-Max Normalization) method is used to unify the dimensions of continuous variables. Finally, the preprocessed feature data uses a time sliding window method to divide the feature data into a fixed-size continuous subsequence, which represents the preceding continuous time information fragment of the operator's intention.

[0063] After that, the structure of the low-level interactive intention recognition model is determined. In order to reduce the number of model parameters and reduce the model complexity of the low-level interactive intention recognition model, the input data needs to be convolved through a single-direction 1D-CNN layer with high computational efficiency. In order to realize the intention recognition of time series data and meet the characteristics of bidirectional dependence between time series data, it is necessary to select the Bi-LSTM network as the main part of the model. According to the above analysis, the 1D-CNN module and the Bi-LSTM module are integrated into a low-level interactive intention recognition model in turn, and the forward propagation process of the model is determined as follows: first, the feature data is input into the multi-layer 1D-CNN network, and the local features of various feature variables in the time dimension are extracted through convolution operations, and the gradient disappearance is avoided through the Relu nonlinear activation function; then, the multi-layer Bi-LSTM network is input to capture the correlation between the feature data time series and the interactive intention from the forward and reverse perspectives; finally, the input dimension is mapped to the output dimension through the linear layer, and the output data is normalized to the probability distribution of the intent category from 0 to 1 through the Softmax layer.

[0064] Finally, the low-level interaction intention recognition model is trained and verified. The basic process is as follows: First, the structured data is preprocessed to obtain the time series of feature data, and 80% of the time series is used as the training set and 20% of the time series is used as the test set. After that, the training set is used to train the above model. Finally, the test set is input into the training model to evaluate the effect of the low-level interaction intention recognition model.

[0065] Step S32: Perform a second preprocessing on the multi-level intention feature data set to obtain second preprocessed data, and divide the second preprocessed data into a second training set and a second test set; train the high-level task intention recognition model through the second training set, and test the trained high-level task intention recognition model through the second test set until the final high-level task intention recognition model is obtained.

[0066] The high-level task intent recognition model adopts a dynamic Bayesian network structure. This application establishes a causal model of "context-intention-behavior" to represent the causal relationship between human intention and context and behavior, so as to achieve interpretable modeling of intention. In addition, this application inputs the recognition results of the low-level interactive intent recognition model into the dynamic Bayesian network to enhance the accuracy of the overall task intent recognition. Figure 5 shown.

[0067] First, determine the network structure of the dynamic Bayes. This application performs intent modeling based on the influence relationship between context-intention-behavior. The Bayesian network under a single time slice mainly includes three types of nodes: context, intention, and behavior: context nodes include drone information-related states, local information-related states, and task information-related states, etc. The attribute values ​​of these nodes are directly obtained through observation; intention nodes include task intentions and interaction intentions. Task intentions are defined as implicit, and interaction intentions are driven by the recognition results of low-level interaction intention recognition models; behavior nodes include human hand behaviors and eye behaviors, specifically including key information and eye gaze AOI information during the operation of Hotas. These nodes are also defined as observable nodes. In addition, in the embodiments of this application, it is assumed that all nodes meet the first-order hidden Markov conditions.

[0068] Afterwards, the high-level task intent recognition model is trained and verified, including three steps: data processing, parameter learning, and intent reasoning. First, due to technical limitations, the dynamic Bayesian network only supports discrete input, so the data of the structured data set is discretized and divided into time windows, and 80% of the time series data set is used as the training set, and the remaining 20% ​​is used as the test set. Afterwards, the feature data of the training set is assigned to each DBN node, and parameter learning is performed under the specified time window to calculate the conditional probability distribution of each node in the training set. Finally, the time series feature state is input to the observation nodes such as the situation node and the behavior node, and the low-level interaction intention recognition result is input to the interaction intention node, so as to infer the task intent at the current moment and evaluate the task intent recognition accuracy.

[0069] Step S33: Input the recognition result of the final low-level interaction intention recognition model into the final high-level task intention recognition model for further intention recognition, so as to realize multi-level human-computer interaction intention recognition.

[0070] The above are exemplary embodiments of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-level human-computer interaction intention recognition method for complex task scenarios, characterized in that: include: Step S1: Analyze the task objectives of complex tasks and extract feature categories of multi-level intentions; Step S2: Conducting experiments in a simulation task environment, and collecting multi-level intention feature data sets of human behavior and context situation information according to feature categories; Step S3: Train and test the intention recognition model through a multi-level intention feature data set, and use the trained intention recognition model to identify intentions from low-level to high-level layers from the underlying feature data to achieve multi-level human-computer interaction intention recognition.

2. The multi-level human-computer interaction intention recognition method according to claim 1, characterized in that: The step S1 comprises: Through hierarchical goal analysis, the goals of complex tasks are decomposed layer by layer, and the hierarchical task goals are matched with the intentions of people at multiple levels to obtain the hierarchical goal analysis results; the operation sequence of people in the complex task process is sorted out through the operation sequence diagram, and then the underlying features related to the multi-level intentions are sorted out according to the operation sequence to obtain the analysis results of the operation sequence diagram; Extract high-level task intentions and low-level task interaction intentions based on the hierarchical goal analysis results; According to the hierarchical goal analysis results and the operation sequence diagram analysis results, the relevant features of the high-level task intention and the low-level task interaction intention are extracted to obtain feature categories; wherein the relevant features of the high-level task intention and the low-level task interaction intention include behavioral features and contextual features, the behavioral features include behavioral data, and the behavioral data include hand behavior data, eye behavior data and human physiological data; the contextual features include system data and situation data, the system data includes local system data, local motion data and local alarm data, and the situation data includes geographic situation data, spatiotemporal situation data and environmental situation data.

3. The multi-level human-computer interaction intention recognition method according to claim 2, characterized in that: The step S2 comprises: Build complex mission scenarios based on the 3D engine; According to the feature categories, multi-level intention feature data in the constructed complex task scenarios are extracted to obtain a multi-level intention feature dataset.

4. The multi-level human-computer interaction intention recognition method according to claim 3, characterized in that: The step S3 comprises: Step S31: performing a first preprocessing on the multi-level intention feature data set to obtain first preprocessed data, and dividing the first preprocessed data into a first training set and a first test set; training a low-level interaction intention recognition model by using the first training set, and testing the trained low-level interaction intention recognition model by using the first test set, until a final low-level interaction intention recognition model is obtained; Step S32: performing a second preprocessing on the multi-level intention feature data set to obtain second preprocessed data, and dividing the second preprocessed data into a second training set and a second test set; training the high-level task intention recognition model through the second training set, and testing the trained high-level task intention recognition model through the second test set, until a final high-level task intention recognition model is obtained; Step S33: Input the recognition result of the final low-level interaction intention recognition model into the final high-level task intention recognition model for further intention recognition, so as to realize multi-level human-computer interaction intention recognition.

5. The multi-level human-computer interaction intention recognition method according to claim 4, characterized in that: In the step S31, the multi-level intention feature data set is subjected to a first preprocessing to obtain first preprocessed data, including: One-Hot encoding is used for discrete data in the multi-level intention feature data set to convert categorical variables in the discrete data into binary variables to obtain first feature data; Normalizing the continuous data in the multi-level intention feature data set to unify the dimensions of the continuous data to obtain second feature data; A time sliding window is used to divide the first feature data and the second feature data into continuous subsequences of a fixed size to obtain first preprocessed data.

6. The multi-level human-computer interaction intention recognition method according to claim 5, characterized in that: In the step S32, the multi-level intention feature data set is subjected to a second preprocessing to obtain second preprocessed data, including: The data in the multi-level intention feature data set is discretely processed and divided into time windows to obtain second preprocessed data.

7. The multi-level human-computer interaction intention recognition method according to claim 6, characterized in that: The low-level interaction intention recognition model includes a 1D-CNN module and a Bi-LSTM module, and the high-level task intention recognition model includes a dynamic Bayesian network structure.

Images