Knowledge graph-based cockpit man-machine interaction method and system and electronic equipment

Abstract

The invention discloses a cockpit man-machine interaction method and system based on a knowledge graph and electronic equipment, and the method comprises the steps: after obtaining a typical task scene of cockpit man-machine interaction, and a knowledge graph and a task plan in the typical task scene, analyzing the typical task scene and the task plan, and then building a task model, a task plan is decomposed into a task network through a task model in combination with a knowledge graph, the task network is converted into a task list according to changes of action information and environment situation information of a pilot after the action information and the environment situation information of the pilot are obtained, then load data of the pilot are monitored, and a load evaluation result and an intention prediction result are finally generated in combination with the task list. The problem that pilot interaction requirements cannot be met since pilot intention prediction cannot be performed by combining task scenes, task plans and task decomposition results in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a cockpit human-computer interaction method, system and electronic equipment based on a knowledge graph. Background Art

[0002] In the field of helicopter driving, the human-computer interaction system in the flight environment constitutes the core framework of safe flight. As the key subject in the system, the pilot's status will directly affect the safety and efficiency of the flight mission. Especially when facing complex and changing flight environments, such as low-visibility flights in severe weather conditions, emergency rescue operations and other tasks, pilots face not only physiological challenges, but also huge psychological pressure and cognitive load. In this case, how to provide an efficient and stable human-computer interaction system is an urgent problem to be solved.

[0003] The mental workload of pilots has long been a concern in the field of flight safety. In a high-pressure environment, pilots need to quickly process a large amount of information, including but not limited to meteorological data, terrain information, flight instrument indications, radio communications, and emergency response strategies. This high-intensity cognitive demand on pilots can easily lead to information processing errors, which in turn lead to wrong decisions and are the direct cause of many flight accidents. Especially in takeoff, approach and landing, which are recognized as the more dangerous links in the flight process, pilots need to make accurate judgments and operations in a short period of time, and any small mistakes may lead to serious consequences.

[0004] Therefore, there is an urgent need for an advanced helicopter cockpit human-machine collaboration technology that can meet the pilot's interaction requirements while achieving load assessment and intention prediction for the pilot. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a cockpit human-computer interaction method, system and electronic equipment based on a knowledge graph, so as to solve the problem in the prior art that the pilot's load assessment and intention prediction cannot be performed in a timely manner, resulting in low flight stability and safety.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a cockpit human-computer interaction method based on a knowledge graph, the method comprising: Obtain typical mission scenarios of cockpit human-computer interaction, as well as knowledge graphs and mission plans under the typical mission scenarios; Analyze the typical mission scenarios and parse the mission plans, and establish a mission model based on the analysis and parsing results; In combination with the knowledge graph, the task plan is decomposed into a task network through the task model; Obtain pilot action information and environmental situation information; converting the task network into a task list according to changes in the pilot's action information and environmental situation information; monitoring load data of the pilot; Based on the load data and task list, load assessment results and intention prediction results are generated.

[0007] On the basis of the above technical solution, the present invention can also be improved as follows: Furthermore, in one implementation, the analyzing of typical task scenarios and parsing of task plans, and establishing a task model according to the analysis results and the parsing results, include: Analyze the typical task scenarios to obtain a basic task set; Performing unified knowledge representation on the multimodal data in the basic task set; Parsing and structuring the text content in the task plan; The task model is constructed based on the multimodal data after the unified knowledge representation and the text content after parsing and structuring.

[0008] Furthermore, in one implementation, the combining of the knowledge graph and decomposing the task plan into a task network through the task model includes: Performing knowledge extraction on the task plan to obtain a knowledge extraction result, wherein the knowledge extraction includes: performing entity extraction, relationship extraction and event extraction on the task plan; The entity extraction includes the extraction of key entities, the relationship extraction includes the extraction of relationships between key entities, and the event extraction includes the extraction of meta-events to obtain event extraction results; Performing knowledge matching on the task plan to obtain a knowledge matching result, wherein the knowledge matching includes: matching the knowledge extraction result with a knowledge graph; Performing knowledge analysis on the task plan to obtain a knowledge analysis result, wherein the knowledge analysis includes: parsing and structuring the text content in the task plan; The knowledge extraction result, the knowledge matching result and the knowledge parsing result are associated to generate a task network corresponding to the task plan.

[0009] Furthermore, in one implementation, converting the task network into a task list according to changes in the pilot's action information and environmental situation information includes: Determine the time nodes of each subtask in the task plan according to the event extraction results obtained in the process of decomposing the task plan into a task network; According to the relationship between the changes in the pilot's action information and environmental situation information and the time nodes of each subtask, the task network is converted into a task list containing time nodes, and the task list is updated in real time. The task list includes a completed task list, a current task list and an expected task list.

[0010] Furthermore, in one implementation, converting the task network into a task list according to changes in the pilot's action information and environmental situation information includes: If, according to the event extraction result obtained in the process of decomposing the task plan into a task network, there are subtasks without time nodes in the task plan, the subtasks are marked as tasks without sequence relationship; The tasks without sequence relationship are sorted into a task list, which is distinguished from the task list containing time nodes and is set as a parallel task list.

[0011] Furthermore, in one implementation, monitoring the pilot's load data includes: Acquiring load data of the pilot through monitoring equipment, wherein the load data includes physiological load data and psychological load data; The monitoring device includes feature extractors of different modes for receiving load data of different modes.

[0012] Furthermore, in one implementation, generating a load assessment result and an intention prediction result according to the load data and the task list includes: Generating a psychological load assessment result according to the load data of the different modes by using a preset automated psychological load assessment method; At the same time, according to the load data of different modes, combined with the knowledge in the task network, feature calculation is performed on the load data to generate a multi-channel load feature fusion result; According to the current task list in the task list, the mental load assessment result and the multi-channel load feature fusion result are integrated into a load assessment result corresponding to the current task list.

[0013] Furthermore, in one implementation, generating a load assessment result and an intention prediction result according to the load data and the task list includes: Combining the task changes and load assessment results of the task list, generating the intention prediction result according to a multi-dimensional intention recognition algorithm; The multi-dimensional intention recognition algorithm includes: an intention recognition method based on a convolutional neural network, an intention recognition method based on a long short-term memory neural network, an intention recognition method based on semantic features, and an intention recognition method based on a fusion strategy.

[0014] In a second aspect, the present invention provides a cockpit human-computer interaction system based on a knowledge graph, the system comprising: an acquisition module, a task model building module, a task network acquisition module, a pilot information acquisition module, a task list acquisition module, a load data monitoring module, and a load evaluation and intention recognition module; The acquisition module is used to acquire typical task scenarios of cockpit human-computer interaction, as well as knowledge graphs and task plans under the typical task scenarios; The task model building module is used to analyze the typical task scenario and parse the task plan, and build a task model based on the analysis and parsing results; The task network generation module is used to combine the knowledge graph and decompose the task plan into a task network through the task model; The pilot information acquisition module is used to acquire the pilot's action information and environmental situation information; The task list generation module is used to convert the task network into a task list according to the changes of the pilot's action information and environmental situation information; The load data acquisition module is used to monitor the load data of the pilot; The load assessment and intention recognition module is used to generate load assessment results and intention prediction results based on the load data and task list.

[0015] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method described in the first aspect when executing the computer program.

[0016] The present invention discloses a cockpit human-computer interaction method, system and electronic device based on knowledge graph, wherein the method comprises: after obtaining typical task scenarios of cockpit human-computer interaction, as well as knowledge graphs and task plans under typical task scenarios, analyzing typical task scenarios and parsing task plans, and then establishing a task model, combining the knowledge graph, decomposing the task plan into a task network through the task model, after obtaining the pilot's action information and environmental situation information, according to their changes, converting the task network into a task list, and then monitoring the pilot's load data, and then combining the task list, finally generating a load assessment result and an intention prediction result. The problem that the prior art cannot combine the task scenario, task plan and task decomposition results to predict the pilot's intention, and thus cannot meet the pilot's interaction requirements, is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0018] Figure 1 A schematic diagram of the flow of a cockpit human-computer interaction method based on a knowledge graph provided in an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a cockpit human-computer interaction system based on a knowledge graph provided in an embodiment of the present invention.

[0019] Among them, 10-acquisition module, 20-task model building module, 30-task network generation module, 40-pilot information acquisition module, 50-task list generation module, 60-load data acquisition module, 70-load assessment and intention recognition module. DETAILED DESCRIPTION

[0020] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Figure 1 A flow chart of a cockpit human-computer interaction method based on a knowledge graph provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps: S101, obtaining a typical task scenario of cockpit human-machine interaction, and a knowledge graph and a task plan under the typical task scenario; specifically, in an embodiment of the present invention, the typical task scenario and the knowledge graph and the task plan under the typical task scenario need to be pre-set according to the specific environment in which the pilot is located, and are generally provided by relevant technical units; S102, analyzing the typical task scenario and parsing the task plan, and establishing a task model according to the analysis and parsing results; S103, combining the knowledge graph and decomposing the task plan into a task network through the task model; S104, obtaining the pilot's action information and environmental situation information; specifically, in the embodiment of the present invention, the pilot's action information includes action sequence, voice text, etc., and the environmental situation information includes weather information, terrain information, flight instrument indication information, radio communication information, etc.; S105, converting the task network into a task list according to changes in the pilot's action information and environmental situation information; S106, monitoring the load data of the pilot; S107, generating a load assessment result and an intention prediction result according to the load data and the task list.

[0022] The cockpit human-computer interaction method based on knowledge graph provided by the embodiment of the present invention realizes, through S101~S107, in a complex helicopter cockpit human-computer interaction scene, after acquiring multimodal data related to the pilot and multimodal data related to the mission plan, a reasonable description and knowledge representation of data of various types and complex structures are performed by establishing a task model, a task network and a task list; wherein the multimodal data related to the pilot includes pilot action information, environmental situation information and load data, etc., and the multimodal data related to the mission plan includes nodes, associations, conditions, constraints, time and space in the task process, etc.; in addition, through the present invention, the acquired multimodal data is associated and fused with data of other dimensions in the cockpit human-computer interaction scene, and the data of other dimensions includes pilot portraits, avionics system composition, etc.; the present invention can associate and represent these knowledge of different dimensions and structures, and can provide a good foundation for the analysis and processing of multimodal data, thereby better realizing the intelligent application of intention recognition and load assessment based on multimodal data, and meeting the pilot interaction requirements in the helicopter driving environment.

[0023] After this step, it also includes: in combination with the task list, the load assessment results and the intention prediction results are pushed; in actual applications, the push method of the load assessment results and the intention prediction results and the display method on the cockpit display panel can be adjusted according to the specific application scenario and the needs of the pilot, including the push content, push channel, push position, display size, etc.

[0024] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S102, analyzing typical mission scenarios and parsing mission plans, and establishing a mission model according to the analysis results and parsing results, includes: S1021, analyzing the typical mission scenario to obtain a basic mission set; in an embodiment of the present invention, the basic mission set includes process elements in a mission plan; specifically, the process elements in the mission plan may cover a complete process from mission planning, takeoff preparation, low-altitude operation to return recovery; each step must clearly specify the personnel roles involved, equipment configuration, communication requirements, energy consumption and other details; S1022, performing unified knowledge representation on the multimodal data in the basic task set; in addition, in S1022 provided in the embodiment of the present invention, it is also necessary to associate and fuse the multimodal data with data of other dimensions in the cockpit human-computer interaction scene; the data of other dimensions, for example: pilot portrait, avionics system composition, environmental situation information, etc.; through this step, the knowledge in the helicopter cockpit human-computer interaction scene is uniformly represented and integrated, effectively eliminating ambiguity between the knowledge, and laying a foundation for realizing intelligent applications such as pilot intention recognition, load assessment, and information push based on cross-modal data in subsequent steps; S1023, parsing and structuring the text content in the task plan; specifically, S1023 provided by the embodiment of the present invention includes extracting and annotating key information, instructions, time series, geographic location and other elements in the task plan; the parsing and structuring can be performed based on information extraction, entity recognition, relationship extraction and other technologies, and finally constructing a machine-readable representation of the document content; S1024, constructing the task model according to the multimodal data after the unified knowledge representation and the text content after parsing and structuring; specifically, the attributes of the task model include but are not limited to task load and task automation level.

[0025] S1021~S1024 provided in the embodiment of the present invention can realize the construction of the task model through the joint representation technology of multimodal data and knowledge such as text, voice, image, video, signal, etc.; the multimodal data can provide more and more useful feature information than single modal data for many tasks in human-computer interaction scenarios, obtain effective representation learning results from multimodal data, convert the original data of each modality into feature representation, and realize the unified joint representation of multimodal knowledge, thereby breaking the barriers of multi-source heterogeneous knowledge.

[0026] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S103, in combination with the knowledge graph, decomposes the task plan into a task network through the task model, including: S1031, extracting knowledge from the task plan to obtain a knowledge extraction result, wherein the knowledge extraction includes: extracting entities, extracting relations, and extracting events from the task plan; The entity extraction includes the extraction of key entities, the relationship extraction includes the extraction of relationships between key entities, and the event extraction includes the extraction of meta-events to obtain event extraction results; S1032, performing knowledge matching on the task plan to obtain a knowledge matching result; the knowledge matching includes: matching the knowledge extraction result with a knowledge graph; S1033, performing knowledge analysis on the task plan to obtain a knowledge analysis result, wherein the knowledge analysis includes: performing analysis and structural processing on the text content in the task plan; S1034, associating the knowledge extraction results, knowledge matching results and knowledge parsing results to generate a task network corresponding to the task plan; the task network is used to support the functional implementation of subsequent steps, such as pilot load assessment, human-machine function allocation, etc., to achieve task decomposition and monitoring based on the task network.

[0027] Specifically, in the present invention, the knowledge extraction in S1031 includes: entity extraction, relationship extraction and event extraction of the task plan, wherein entity extraction includes the extraction of key entities such as tasks and goals, relationship extraction is used to analyze the association between entities, and provide a basis for the association between the various results in S1034 and the subsequent task execution, and event extraction is used to abstract the entire task plan to form an operational event node.

[0028] The knowledge matching in S1032 includes: matching the knowledge extraction result with the pre-built knowledge graph, specifically, implementing subgraph query on the knowledge graph by entity alignment of the knowledge extraction result with the knowledge graph; then the system can generate a task network corresponding to the task plan, and the task network includes key knowledge such as the task and environmental situation; The knowledge analysis in S1033 includes parsing and structuring the text content in the task plan; specifically, it includes extracting and labeling key information, instructions, time series, geographic locations and other elements of the text content of the task plan; specifically, the analysis and structuring can be based on information extraction, entity recognition, relationship extraction and other technologies to achieve the construction and representation of machine-readable text content.

[0029] In addition, the task network is presented in a visual chart, including various attributes of the task model; specifically, the visual chart can be represented as a dual-axis chart, wherein one axis is used to represent time, and the other axis is used to represent various attributes in the task model.

[0030] The present invention performs knowledge extraction, knowledge matching and knowledge analysis on the mission plan through S1031~S1034, firstly automatically identifies named entities from the multimodal data set related to the mission plan, that is, the extracted and labeled entities; then, on the basis of entity extraction, extracts the semantic association relationship between entities from the corpus related to the multimodal data set, and finally associates the entities through the relationship, that is, can form a task network corresponding to the mission plan; through this step, the complex mission plan issued by the superior to the pilot can be decomposed into an executable task network; the intention of the mission instruction is accurately understood, and the macro task is refined into a series of specific subtasks in combination with the established mission plan, so as to form a task execution link with clear logic and strong operability; and a good foundation is laid for the realization of the subsequent load assessment and intention prediction functions of the pilot.

[0031] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S105, according to the changes of the pilot's action information and environmental situation information, converts the task network into a task list, including: S1051, determining the time nodes of each subtask in the task plan according to the event extraction result obtained in the process of decomposing the task plan into a task network; S1052, according to the relationship between the changes in the pilot's action information and environmental situation information and the time nodes of each subtask, convert the task network into a task list containing time nodes, and update the task list in real time. The task list includes a completed task list, a current task list and an expected task list.

[0032] In the present invention, the relationship between the change of the pilot's action information and environmental situation information and the time nodes of each subtask includes: the change of the pilot's action information and / or environmental situation information occurs at a certain time node, and the change of the pilot's action information and environmental situation information occurs during a time period from a certain time node to another time node, etc.; In the process of the pilot executing the mission plan, the mission network will be dynamically adjusted according to the actual situation; when the mission plan is updated, the system will synchronize the latest mission subgraph to ensure that the pilot can grasp the mission situation in real time; in the specific implementation process, during the mission, the pilot's action or environmental situation changes, that is, the pilot's action information such as action sequence, voice text, and the pilot's environmental situation information such as weather information and terrain information change, which will trigger the activation and completion conditions of the mission, so that the mission network can maintain the current active, expected and completed task list in real time at the mission time point; this highly intelligent task management mechanism is realized through S1051~S1052, so that the mission network can maintain the current, expected and completed task list in real time at the mission time point; Therefore, when performing missions, pilots can make full use of the rich information provided by this system; for example, when performing a reconnaissance mission, pilots can choose the best flight route and reconnaissance area based on the environmental situation information in the mission network; in addition, the mission network can also monitor the pilot's actions and environmental situation in real time, provide decision-making support for the pilot, and ensure the smooth completion of the mission; while improving the pilot's ability to operate in complex environments, it also reduces the pilot's workload; through intelligent analysis of mission instructions and real-time maintenance of the mission network, pilots can focus more on flight operations and decision-making, thereby improving flight efficiency.

[0033] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S105, according to the changes of the pilot's action information and environmental situation information, converts the task network into a task list, including: S1053, if, according to the event extraction result obtained in the process of decomposing the task plan into a task network, there are subtasks without time nodes in the task plan, mark the subtasks as tasks without sequence relationship; S1054: Arrange the tasks without sequence relationship into a task list, separate it from the task list containing time nodes, and set it as a parallel task list.

[0034] In the present invention, through S1053~S1054, the task list includes sequential tasks related to time nodes and parallel tasks unrelated to time nodes, so that the pilot can adopt different processing strategies according to the different characteristics of such sequential tasks and parallel tasks; specifically, if there is a sequence between tasks and they cannot be executed in parallel, then these tasks are defined as sequential tasks, and the predecessor task must be completed before the next task can be carried out, so as to ensure that the entire take-off process is carried out in an orderly manner; and for tasks without a sequential relationship, they are defined as parallel tasks; in the subsequent steps, when the system coordinates task resources, different message channels can be used to provide real-time prompts to the pilot, thereby meeting the pilot's human-computer interaction requirements in the cockpit environment, so that the pilot can efficiently manage multiple tasks.

[0035] In addition, when the pilot's action information and environmental situation information change, the system will immediately prompt the pilot to inform him whether the sequential tasks are completed within the specified time; if it is found that a task is not completed within the specified time, the system will make a judgment based on the priority of the task and push the judgment result to the pilot for confirmation through the task list; the pilot needs to decide whether to terminate or continue the current unfinished task based on the actual situation; this flexible task management mechanism meets the pilot's human-computer interaction requirements in the cockpit environment, allowing pilots to make decisions quickly in a complex and changing environment to ensure the smooth progress of the mission.

[0036] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the step S106, monitoring the pilot's load data, includes: S1061, obtaining load data of the pilot through a monitoring device, where the load data includes physiological load data and psychological load data; S1062, the monitoring device includes feature extractors of different modes, which are used to receive load data of different modes.

[0037] In the present invention, the physiological load data and psychological load data of the pilot are monitored in real time through the monitoring equipment of S1061~S1062, that is, feature extractors of different modes; the physiological load data include: visual fatigue, heart rate / pulse, electromyography / electric skin, body temperature, blood oxygen saturation, etc.; the psychological load data include: attention span, working memory capacity, attention allocation, attention transfer and other indicators.

[0038] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S107 generates load assessment results and intention prediction results according to load data and task list, including: S1071, generating a psychological load assessment result according to the load data of the different modes through a preset automated psychological load assessment method; specifically, the preset automated psychological load assessment method includes a psychological load assessment scale, such as the NASA-TLX scale, the SWAT scale, the WP scale, the multiple resource questionnaire (MRQ) and the modified Cooper-Harper (MCH) scale, etc., for automatically calculating the psychological load assessment result according to the psychological load data.

[0039] S1072, at the same time, according to the load data of different modes, combined with the knowledge in the task network, feature calculation is performed on the load data to generate a multi-channel load feature fusion result; S1073, according to the current task list in the task list, integrating the psychological load assessment result and the multi-channel load feature fusion result into a load assessment result corresponding to the current task list.

[0040] In the existing technology, the monitoring data of the pilot's face, voice, movement, indicators and other dimensions obtained by sensors at different points are often not able to be efficiently fused in a single way due to the different modalities, structures and features of the monitoring data. The extraction of features of different modalities is difficult, and there is a problem that multi-channel features cannot be effectively normalized, which leads to the limitations of feature fusion and the singleness of feature selection. Therefore, the final accuracy of the pilot's load assessment is low and cannot meet the pilot's interaction requirements.

[0041] In the present invention, through S1071~S1073, after obtaining the typical task scenario of cockpit human-computer interaction, as well as the knowledge graph and task plan under the typical task scenario, the typical task scenario is analyzed and the task plan is parsed, and a task model is established according to the analysis results and the parsing results. Then, combined with the knowledge graph, the task plan is decomposed into a task network through the task model. After obtaining the pilot's action information and environmental situation information, the task network is converted into a task list according to their changes, and then the pilot's load data is monitored. Combined with the task list, the load assessment result and the intention prediction result are finally generated; the problem that the prior art cannot combine the task scenario, task plan and task decomposition results to predict the pilot's intention, resulting in the inability to meet the pilot's interaction requirements, is solved.

[0042] In the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, the S107 generates load assessment results and intention prediction results according to load data and task list, including: S1074, combining the task changes and the load assessment results of the task list, generating the intention prediction result according to a multi-dimensional intention recognition algorithm; The multi-dimensional intention recognition algorithm includes: an intention recognition method based on a convolutional neural network, an intention recognition method based on a long short-term memory neural network, an intention recognition method based on semantic features, and an intention recognition method based on a fusion strategy.

[0043] Specifically, the present invention provides the capability of identifying pilot intentions, analyzes the pilot's task operation behavior according to the task changes in the task list, and combines various physiological indicators in the load assessment results to automatically analyze their operation intentions using a multi-dimensional intention recognition algorithm, accurately identify the behavioral orientation in the mission plan, provide intelligent auxiliary decision support for flight missions, and enhance real-time response and task execution.

[0044] In the specific implementation of S1074, a multi-dimensional data fusion method can be used to realize the recognition of the pilot's intention. For example, the pilot's operating action sequence, such as flight control input, switch operation rhythm and pattern, etc., is collected, and voice communication content such as voice recognition to text is collected for semantic and emotional analysis. At the same time, physiological parameters such as heart rate, galvanic skin response, electromyographic activity, etc. are collected to build a comprehensive real-time data stream.

[0045] In the existing technology, intent recognition can be achieved in a variety of ways. For example, the rule method based on dictionary templates has the advantage of being able to start quickly, but its disadvantage is that if all rules are written manually, a lot of manual work is required, and the coverage and scalability are difficult to meet the requirements of practical applications. The method based on machine learning (deep neural network) has high coverage and strong scalability, but its main disadvantage is that a large amount of corpus is required to train the model.

[0046] The intention recognition in the present invention adopts a multi-dimensional intention recognition algorithm fusion method, while considering the sequence features and semantic features for the intention recognition task. In the initial stage, a rule matching method based on the intention recognition method based on semantic features is used. On the one hand, a small number of template rules are manually pre-written, and on the other hand, intention recognition rules are automatically generated based on the pre-built cockpit human-computer interaction knowledge graph to reduce the manual workload, so that the problem of cold start of the system can be solved. For the corpus required by deep neural network models such as CNN and LSTM, on the one hand, a rule-based method can be used to automatically generate part of the corpus; on the other hand, it can be combined with a method based on historical log analysis to extract from the past flight log records of the system; according to specific typical mission scenarios, by using the combination of these intention recognition algorithms, the automatic construction of the corpus can be realized, forming a remote supervised learning method. Using machine learning and deep learning technology, an association model between pilots' operating habits, physiological reactions and specific intentions is established to learn and understand pilots' operating modes in different situations, such as mission execution preparation, emergency avoidance, reconnaissance and exploration, so as to predict their next possible action intentions.

[0047] The intention recognition results can be used to further conduct dynamic mission intention analysis. Specifically, the analysis strategy can be adjusted dynamically based on the mission progress and environmental changes to evaluate in real time whether the pilot's immediate reaction and decision-making intention match the current mission objectives. For example, it can be identified whether the pilot is preparing to perform a maneuver or prepare to respond to an emergency.

[0048] In addition, through the intention recognition results, auxiliary decision support for the pilot can be achieved: based on the accurate recognition of the pilot's intention, the information display can be optimized or the auxiliary driving parameters can be automatically adjusted to provide an intention basis for subsequent function allocation and information push.

[0049] like Figure 2 As shown, based on the cockpit human-computer interaction method based on knowledge graph provided by an embodiment of the present invention, an embodiment of the present invention also provides a cockpit human-computer interaction system based on knowledge graph, the system comprising: an acquisition module 10, a task model establishment module 20, a task network acquisition module 30, a pilot information acquisition module 40, a task list acquisition module 50, a load data monitoring module 60, and a load assessment and intention recognition module 70; The acquisition module 10 is used to acquire typical task scenarios of cockpit human-computer interaction, as well as knowledge graphs and task plans under the typical task scenarios; The task model building module 20 is used to analyze the typical task scenario and parse the task plan, and build a task model based on the analysis and parsing results; The task network generation module 30 is used to combine the knowledge graph and decompose the task plan into a task network through the task model; The pilot information acquisition module 40 is used to acquire the pilot's action information and environmental situation information; The task list generation module 50 is used to convert the task network into a task list according to the changes of the pilot's action information and environmental situation information; The load data acquisition module 60 is used to monitor the load data of the pilot; The load assessment and intention recognition module 70 is used to generate a load assessment result and an intention prediction result according to the load data and the task list.

[0050] Based on the knowledge graph-based cockpit human-computer interaction method and system provided in an embodiment of the present invention, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any method described in the knowledge graph-based cockpit human-computer interaction method are implemented.

[0051] In addition, based on the above methods, systems and electronic devices, the embodiments of the present invention also provide a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above method embodiments.

[0052] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0053] In combination with the above embodiments, the present invention discloses a cockpit human-computer interaction method, system and electronic device based on knowledge graph, wherein the method comprises: after obtaining typical task scenarios of cockpit human-computer interaction, as well as knowledge graphs and task plans under typical task scenarios, analyzing typical task scenarios and parsing task plans, establishing task models according to the analysis results and parsing results, combining knowledge graphs, decomposing task plans into task networks through task models, after obtaining pilot action information and environmental situation information, converting the task network into task lists according to their changes, and then monitoring the pilot's load data, and then combining the task lists to finally generate load assessment results and intention prediction results. This solves the problem that the prior art cannot combine task scenarios, task plans and task decomposition results to predict pilot intentions, and thus cannot meet pilot interaction requirements.

[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0056] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. The cockpit human-computer interaction method based on knowledge graph is characterized by: The method comprises: Obtain typical mission scenarios of cockpit human-computer interaction, as well as knowledge graphs and mission plans under the typical mission scenarios; Analyze the typical mission scenarios and parse the mission plans, and establish a mission model based on the analysis and parsing results; In combination with the knowledge graph, the task plan is decomposed into a task network through the task model; Obtain pilot action information and environmental situation information; converting the task network into a task list according to changes in the pilot's action information and environmental situation information; monitoring load data of the pilot; Based on the load data and task list, load assessment results and intention prediction results are generated.

2. The cockpit human-computer interaction method based on knowledge graph according to claim 1 is characterized in that: The typical task scenario is analyzed and the task plan is parsed, and a task model is established according to the analysis and parsing results, including: Analyze the typical task scenarios to obtain a basic task set; Performing unified knowledge representation on the multimodal data in the basic task set; Parsing and structuring the text content in the task plan; The task model is constructed based on the multimodal data after the unified knowledge representation and the text content after parsing and structuring.

3. The cockpit human-computer interaction method based on knowledge graph according to claim 1 is characterized in that: The combined knowledge graph and the task model are used to decompose the task plan into a task network, including: Performing knowledge extraction on the task plan to obtain a knowledge extraction result, wherein the knowledge extraction includes: performing entity extraction, relationship extraction and event extraction on the task plan; The entity extraction includes the extraction of key entities, the relationship extraction includes the extraction of relationships between key entities, and the event extraction includes the extraction of meta-events to obtain event extraction results; Performing knowledge matching on the task plan to obtain a knowledge matching result, wherein the knowledge matching includes: matching the knowledge extraction result with a knowledge graph; Performing knowledge analysis on the task plan to obtain a knowledge analysis result, wherein the knowledge analysis includes: parsing and structuring the text content in the task plan; The knowledge extraction result, the knowledge matching result and the knowledge parsing result are associated to generate a task network corresponding to the task plan.

4. The cockpit human-computer interaction method based on knowledge graph according to claim 3 is characterized in that: The step of converting the task network into a task list according to the changes in the pilot's action information and the environmental situation information includes: Determine the time nodes of each subtask in the task plan according to the event extraction results obtained in the process of decomposing the task plan into a task network; According to the relationship between the changes in the pilot's action information and environmental situation information and the time nodes of each subtask, the task network is converted into a task list containing time nodes, and the task list is updated in real time. The task list includes a completed task list, a current task list and an expected task list.

5. The cockpit human-computer interaction method based on knowledge graph according to claim 4 is characterized in that: The step of converting the task network into a task list according to the changes in the pilot's action information and the environmental situation information includes: If, according to the event extraction result obtained in the process of decomposing the task plan into a task network, there are subtasks without time nodes in the task plan, the subtasks are marked as tasks without sequence relationship; The tasks without sequence relationship are sorted into a task list, which is distinguished from the task list containing time nodes and is set as a parallel task list.

6. The cockpit human-computer interaction method based on knowledge graph according to claim 1 is characterized in that: The monitoring of the pilot's load data includes: Acquiring load data of the pilot through monitoring equipment, wherein the load data includes physiological load data and psychological load data; The monitoring device includes feature extractors of different modes for receiving load data of different modes.

7. The cockpit human-computer interaction method based on knowledge graph according to claim 6 is characterized in that: The generating of load assessment results and intention prediction results according to the load data and task list includes: Generating a psychological load assessment result according to the load data of the different modes by using a preset automated psychological load assessment method; At the same time, according to the load data of different modes, combined with the knowledge in the task network, feature calculation is performed on the load data to generate a multi-channel load feature fusion result; According to the current task list in the task list, the psychological load assessment result and the multi-channel load feature fusion result are integrated into a load assessment result corresponding to the current task list.

8. The cockpit human-computer interaction method based on knowledge graph according to claim 7 is characterized in that: The generating of load assessment results and intention prediction results according to the load data and task list includes: Combining the task changes and load assessment results of the task list, generating the intention prediction result according to a multi-dimensional intention recognition algorithm; The multi-dimensional intention recognition algorithm includes: an intention recognition method based on a convolutional neural network, an intention recognition method based on a long short-term memory neural network, an intention recognition method based on semantic features, and an intention recognition method based on a fusion strategy.

9. A cockpit human-computer interaction system based on knowledge graph, characterized in that: The system is used to execute the method according to any one of claims 1 to 8, the system comprising: an acquisition module, a task model building module, a task network acquisition module, a pilot information acquisition module, a task list acquisition module, a load data monitoring module, and a load assessment and intention recognition module; The acquisition module is used to acquire typical task scenarios of cockpit human-computer interaction, as well as knowledge graphs and task plans under the typical task scenarios; The task model building module is used to analyze the typical task scenario and parse the task plan, and build a task model based on the analysis and parsing results; The task network generation module is used to combine the knowledge graph and decompose the task plan into a task network through the task model; The pilot information acquisition module is used to acquire the pilot's action information and environmental situation information; The task list generation module is used to convert the task network into a task list according to the changes of the pilot's action information and environmental situation information; The load data acquisition module is used to monitor the load data of the pilot; The load assessment and intention recognition module is used to generate load assessment results and intention prediction results based on the load data and task list.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images