A hierarchical knowledge graph construction method and system for cockpit human-computer interaction

Abstract

The present invention discloses a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction, wherein the method comprises: first, determining the type of knowledge acquired in the cockpit human-computer interaction scene, if the knowledge type is determined to be basic static knowledge, performing triple knowledge modeling on the basic static knowledge, and constructing a basic static knowledge graph; if the knowledge type is determined to be spatiotemporal dynamic knowledge, performing spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge, and constructing a spatiotemporal dynamic knowledge graph; if the knowledge type is determined to be multimodal knowledge, performing multimodal knowledge modeling on the multimodal knowledge, and constructing a multimodal knowledge graph; according to the cockpit human-computer interaction scene, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph, and the multimodal knowledge graph are cut and merged, and finally an airborne knowledge graph is generated. The problem that the prior art cannot fully utilize the combination of multiple modal knowledge, and thus cannot realize the airborne deployment of large-scale knowledge graphs is solved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction, and in particular to a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction. Background Art

[0002] In the highly professional and complex field of helicopter driving, the "man-machine-environment" system constitutes the core framework of safe flight. In the process of helicopter driving, there are many types of knowledge with complex structures, such as images, texts, videos, audio and other information related to the helicopter driving environment. Especially in the scene of human-machine interaction in the cockpit, time information, spatial information and task process information related to the helicopter driving task are involved. At the same time, it is also necessary to process knowledge such as pilot portraits, avionics system composition, and task process decomposition. This kind of complex and changeable multimodal knowledge comes from different dimensions and has different characteristics and structures, resulting in low efficiency and poor accuracy of the data analysis and processing process in the existing technology.

[0003] Therefore, there is an urgent need for an advanced hierarchical knowledge graph construction technology for cockpit human-computer interaction to efficiently and accurately process multimodal knowledge and realize airborne deployment of large-scale knowledge graphs. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction, so as to solve the problem in the prior art that multimodal knowledge cannot be processed efficiently and accurately in a timely manner, resulting in low flight stability and safety.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for constructing a hierarchical knowledge graph for cockpit human-computer interaction, the method comprising:

[0006] Determine the types of knowledge acquired in the cockpit human-computer interaction scenario, the types of knowledge including: basic static knowledge, spatiotemporal dynamic knowledge, and multimodal knowledge;

[0007] If the knowledge type is determined to be basic static knowledge, the basic static knowledge is modeled as a triplet knowledge to construct a basic static knowledge graph;

[0008] If it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph;

[0009] If it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph;

[0010] According to the cockpit human-computer interaction scenario, the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph are cut and fused to generate an airborne knowledge graph.

[0011] On the basis of the above technical solution, the present invention can also be improved as follows:

[0012] Furthermore, in one implementation, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph, including:

[0013] According to the cockpit human-machine interaction scenario, establishing spatiotemporal coding rules;

[0014] The spatiotemporal coding rule divides the geographical locations involved in the cockpit human-machine interaction scenario according to time on the basis of a global grid model;

[0015] According to the spatiotemporal coding rules, the spatiotemporal dynamic knowledge is converted into information indexes.

[0016] Furthermore, in one implementation, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph, including:

[0017] If the spatiotemporal dynamic knowledge is unstructured spatiotemporal information, extracting information from the unstructured spatiotemporal information to obtain key spatiotemporal information, wherein the key spatiotemporal information includes position information and time information;

[0018] According to the key spatiotemporal information obtained by information extraction and in combination with the spatiotemporal coding rules, the unstructured spatiotemporal information is converted into the information index.

[0019] Furthermore, in one implementation, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph, including:

[0020] In combination with the information index, the spatiotemporal dynamic knowledge is represented and modeled to obtain the spatiotemporal dynamic knowledge graph, including:

[0021] Determining the categories of the spatiotemporal dynamic knowledge, wherein the categories of the spatiotemporal dynamic knowledge include time concepts, space concepts, and spatiotemporal concepts;

[0022] Determine the time concept, space concept and the relationship between the time and space concepts in the time and space dynamic knowledge;

[0023] The time concept includes time point, time period and time series;

[0024] The spatial concepts include points of interest, areas of interest, and paths;

[0025] The space-time concept includes space-time sequences.

[0026] Furthermore, in one implementation, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph, including:

[0027] Determining the type of the cockpit human-computer interaction scene, the type of the cockpit human-computer interaction scene includes: a time theme model, a space theme model and a spatiotemporal theme model;

[0028] If the type of the cockpit human-computer interaction scene is the time theme model, a recurrent neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a time theme spatiotemporal dynamic knowledge graph;

[0029] If the type of the cockpit human-computer interaction scene is the spatial theme model, a convolutional neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatial theme spatiotemporal dynamic knowledge graph;

[0030] If the type of the cockpit human-computer interaction scene is the spatiotemporal topic model, a convolutional neural network and a recurrent neural network are combined as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatiotemporal topic spatiotemporal dynamic knowledge graph.

[0031] Furthermore, in one implementation, if it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph, including:

[0032] By using a multimodal knowledge representation learning method, the multimodal knowledge is represented as an entity, and the association relationship between the multimodal knowledge is obtained;

[0033] That is, through the multimodal knowledge representation learning method, the information of all modalities in the multimodal knowledge is mapped to a unified language-oriented embedding space to align the semantics of the multimodal knowledge.

[0034] Furthermore, in one implementation, if it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph, including:

[0035] According to the logical architecture of the multimodal knowledge graph, the multimodal knowledge graph is constructed in combination with the multimodal knowledge learned by the multimodal knowledge representation;

[0036] The logical architecture of the multimodal knowledge graph includes: ontology elements, entity classes, event classes, multimodal classes, label classes and rule classes.

[0037] Furthermore, in one implementation, if it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph, including:

[0038] The relationship between the entity class and the multimodal class is obtained by extracting the multimodal knowledge;

[0039] The relationship between the multimodal classes is obtained by processing and aligning the multimodal knowledge;

[0040] The relationship between the multimodal class and the tag class is obtained by extracting and semantically fusing the multimodal knowledge to obtain the business features of the multimodal knowledge.

[0041] Furthermore, in one implementation, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cut and merged according to the cockpit human-computer interaction scenario to generate an airborne knowledge graph, including:

[0042] After obtaining the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph, obtaining the display requirements of the onboard knowledge graph in the cockpit human-computer interaction scene;

[0043] According to the display requirements, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cropped and fused through the pattern references of the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph to obtain the airborne knowledge graph.

[0044] In a second aspect, the present invention provides a hierarchical knowledge graph construction system for cockpit human-computer interaction, the system being used to implement the hierarchical knowledge graph construction method for cockpit human-computer interaction as described in any one of the first aspects, the system comprising:

[0045] A knowledge type determination module is used to determine the type of knowledge acquired in the cockpit human-computer interaction scenario, wherein the knowledge types include: basic static knowledge, spatiotemporal dynamic knowledge, and multimodal knowledge;

[0046] A basic static knowledge graph construction module is used to perform triple knowledge modeling on the basic static knowledge and construct a basic static knowledge graph if the knowledge type is determined to be basic static knowledge;

[0047] A spatiotemporal dynamic knowledge graph construction module, for, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, performing spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge and constructing a spatiotemporal dynamic knowledge graph;

[0048] A multimodal knowledge graph construction module, for, if it is determined that the knowledge type is multimodal knowledge, performing multimodal knowledge modeling on the multimodal knowledge and constructing a multimodal knowledge graph;

[0049] The airborne knowledge graph generation module is used to cut and fuse the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph according to the cockpit human-computer interaction scenario to generate an airborne knowledge graph.

[0050] The present invention discloses a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction, wherein the method comprises: first, determining the type of knowledge acquired in the cockpit human-computer interaction scene, if the knowledge type is determined to be basic static knowledge, performing triple knowledge modeling on the basic static knowledge, and constructing a basic static knowledge graph; if the knowledge type is determined to be spatiotemporal dynamic knowledge, performing spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge, and constructing a spatiotemporal dynamic knowledge graph; if the knowledge type is determined to be multimodal knowledge, performing multimodal knowledge modeling on the multimodal knowledge, and constructing a multimodal knowledge graph; according to the cockpit human-computer interaction scene, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph, and the multimodal knowledge graph are cut and merged, and finally an airborne knowledge graph is generated. The problem that the prior art cannot fully utilize the combination of multiple modal knowledge to realize load assessment and intention prediction of pilots, and thus cannot realize airborne deployment of large-scale knowledge graphs is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0052] Figure 1 A schematic diagram of a process of constructing a hierarchical knowledge graph for cockpit human-computer interaction provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the structure of a hierarchical knowledge graph construction system for cockpit human-computer interaction provided by an embodiment of the present invention;

[0054] Among them, 10-knowledge type determination module, 20-basic static knowledge graph construction module, 30-spatiotemporal dynamic knowledge graph construction module, 40-multimodal knowledge graph construction module, and 50-airborne knowledge graph generation module. DETAILED DESCRIPTION

[0055] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Figure 1 A schematic diagram of a process for constructing a hierarchical knowledge graph for cockpit human-computer interaction provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0057] S101, determining the type of knowledge acquired in the cockpit human-machine interaction scenario, where the knowledge type includes: basic static knowledge, spatiotemporal dynamic knowledge, and multimodal knowledge;

[0058] S102, if it is determined that the knowledge type is basic static knowledge, triple knowledge modeling is performed on the basic static knowledge to construct a basic static knowledge graph;

[0059] S103, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, perform spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph;

[0060] S104: if it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph;

[0061] S105, according to the cockpit human-computer interaction scenario, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cut and merged to generate an airborne knowledge graph.

[0062] The embodiment of the present invention first determines the type of knowledge acquired in the cockpit human-computer interaction scene through S101~S105. If the knowledge type is determined to be basic static knowledge, the basic static knowledge is modeled as a triple knowledge to construct a basic static knowledge graph; if the knowledge type is determined to be spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled as spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph; if the knowledge type is determined to be multimodal knowledge, the multimodal knowledge is modeled as multimodal knowledge to construct a multimodal knowledge graph; according to the cockpit human-computer interaction scene, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cut and merged to finally generate an airborne knowledge graph. This solves the problem that the prior art cannot fully utilize the combination of multiple modal knowledge, and thus cannot realize the airborne deployment of large-scale knowledge graphs.

[0063] The embodiment of the present invention targets the multimodal knowledge objectively existing in the human-computer interaction scenario in the helicopter cockpit, performs hierarchical modeling through different knowledge types, introduces complex knowledge types such as spatiotemporal dynamic knowledge and multimodal knowledge on the basis of the traditional static knowledge graph, forms a knowledge graph framework for typical helicopter mission scenarios, and realizes the effective expression of multimodal knowledge.

[0064] In this embodiment, if the knowledge type is determined to be basic static knowledge, the basic static knowledge is modeled as a triple knowledge to construct a basic static knowledge graph, including: the basic static knowledge graph modeling method for basic static knowledge is composed of four basic elements: "concept-entity-attribute-relationship", where concept refers to the abstraction of a class of things; entity represents an existing entity; attribute is the basic feature of a concept or entity, expressed in triple form <object-attribute-value>; Relationship refers to the association relationship between entities. It should be noted that since triple knowledge modeling is a relatively mature technology used in traditional static knowledge graphs, only basic ideas are provided in this embodiment, and specific implementation can be achieved by using relevant existing means.

[0065] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S103, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, performs spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph, includes:

[0066] S1031, establishing a spatiotemporal coding rule according to the cockpit human-machine interaction scenario;

[0067] S1032, the spatiotemporal coding rule divides the geographical locations involved in the cockpit human-machine interaction scene according to time on the basis of the global grid model;

[0068] S1033: Convert the spatiotemporal dynamic knowledge into an information index according to the spatiotemporal coding rule.

[0069] In the embodiment of the present invention, through S1031~S1033, on the basis of the global grid model, the concept of time division is introduced, and the current geographical location is represented by spatiotemporal coding to form a unified information index, which facilitates and quickly performs data search and association, and lays a good data foundation for constructing a spatiotemporal dynamic knowledge graph.

[0070] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S103, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, performs spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph, includes:

[0071] S1034: if the spatiotemporal dynamic knowledge is unstructured spatiotemporal information, extract information from the unstructured spatiotemporal information to obtain key spatiotemporal information, where the key spatiotemporal information includes location information and time information;

[0072] S1035: According to the key spatiotemporal information obtained by information extraction and in combination with the spatiotemporal coding rule, the unstructured spatiotemporal information is converted into the information index.

[0073] Specifically, in this embodiment, the unstructured spatiotemporal information refers to spatiotemporal data that is opposite to structured data and has no fixed model or format, such as text, images, videos, audio, etc. containing spatiotemporal information obtained in the human-computer interaction scene in the helicopter cockpit. Unstructured spatiotemporal information usually does not follow a strict row and column form and cannot be processed using a simple query language. Therefore, the present invention extracts information from unstructured spatiotemporal information, and then converts key spatiotemporal information into information indexes in combination with spatiotemporal coding rules, thereby improving the efficiency of processing unstructured spatiotemporal information data, and thereby accelerating the efficiency of building a spatiotemporal dynamic knowledge graph.

[0074] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S103, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, performs spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph, includes:

[0075] S1036, combining the information index, performing representation modeling on the spatiotemporal dynamic knowledge to obtain the spatiotemporal dynamic knowledge graph, including:

[0076] S10361, determining the categories of the spatiotemporal dynamic knowledge, where the categories of the spatiotemporal dynamic knowledge include time concepts, space concepts, and spatiotemporal concepts;

[0077] S10362, determining the time concept, space concept and the relationship between the time and space concepts in the time and space dynamic knowledge;

[0078] The time concept includes time point, time period and time series; in this embodiment, the time period refers to the time period from a first time point to a second time point, wherein the second time point is greater than or equal to the first time point; the time series refers to the observed data sequence arranged in time order;

[0079] The spatial concept includes points of interest, areas of interest, and paths; in this embodiment, the points of interest are Point of Interest, referred to as POI, or Point of Information, also referred to as POI, which can be translated as information points. Points of interest are generally represented by bubble icons on electronic maps, such as scenic spots, government agencies, companies, shopping malls, restaurants, etc. on electronic maps. The latitude and longitude positions, attributes, and other data of the plot can be obtained through the points of interest.

[0080] The region of interest refers to an area in a map; the area to be processed is usually outlined in a map in the form of a box, circle, ellipse, irregular polygon, etc., which is called the region of interest, or ROI for short. In the electronic information system GIS, the machine vision software Halcon, the cross-platform computer vision library OpenCV and other software, various operators and functions are often used to obtain the region of interest ROI and perform the next step of image processing.

[0081] The path refers to a route from one location to another, and is usually composed of a series of points; a road in a real map is a typical path.

[0082] The space-time concept includes space-time sequences. In this embodiment, the time series refers to an extension in space, which refers to a collection of multiple time series that are spatially related. For example, the best path of a typhoon includes information such as time, center position, maximum wind speed, and central air pressure.

[0083] In the embodiment of the present invention, through S1031 to S1036, based on spatiotemporal coding, the concepts of time and space are subdivided, the spatiotemporal dynamic knowledge is represented and modeled, and the possible relationship between time and space is explored.

[0084] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S103, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, performs spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph, includes:

[0085] S1037, determining the type of the cockpit human-computer interaction scene, where the types of the cockpit human-computer interaction scene include: a time theme model, a space theme model, and a spatiotemporal theme model;

[0086] S10371: If the type of the cockpit human-machine interaction scenario is the time theme model, a recurrent neural network is used as a model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a time theme spatiotemporal dynamic knowledge graph;

[0087] Specifically, in this embodiment, the temporal topic model introduces the time dimension in topic modeling, thereby tracking and analyzing the dynamic changes of topics over time, such as new generation, strengthening or weakening, or tracking the changes in the content of the topic. It can more timely represent the presentation of relevant information of events over time. Since the temporal topic model pays more attention to the representation and modeling of time series, a recurrent neural network RNN ​​is used as the model structure.

[0088] S10372: If the type of the cockpit human-machine interaction scene is the spatial theme model, a convolutional neural network is used as a model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatial theme spatiotemporal dynamic knowledge graph;

[0089] Specifically, in this embodiment, the spatial topic model takes into account the scenarios of different regions in the same topic, strengthens the distinction of geographical area information, and emphasizes the different performances of different regions in the same topic, thereby achieving the purpose of regional analysis. Since the spatial topic model mainly focuses on the representation modeling of information in each spatial region, a convolutional neural network (CNN) is used as the model structure.

[0090] S10373, if the type of the cockpit human-computer interaction scene is the spatiotemporal topic model, a convolutional neural network and a recurrent neural network are combined as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatiotemporal topic spatiotemporal dynamic knowledge graph.

[0091] Specifically, in this embodiment, the topic is described from both time and space dimensions, taking into account the changes in semantic information in time and space, that is, the implicit semantics of the same area will change with time, and the regional scope of similar semantics will also change with time. The spatiotemporal topic model not only performs representation learning for the spatial attributes of various types of knowledge, but also needs to consider the dynamic change process based on the time dimension. Therefore, for the spatiotemporal topic module, the present invention is based on the combination of convolutional neural network and recurrent neural network for representation and learning.

[0092] In the prior art, a set of specifications are usually used to define the expression of time and location, including date, longitude and latitude, etc. However, in actual applications, time and space data are of various types and complex relationships, and different standards exist in different industries, units and even regions, resulting in the problem of inconsistent data specifications between different regions and different historical stages, such as the lunar calendar and the Gregorian calendar, the time difference between regions, and the global space grid division standards. In response to this problem, the present invention uses spatiotemporal coding technology to uniformly compare and transform time and space data, that is, to represent spatiotemporal information in a coded manner. After establishing the spatiotemporal coding rules, for the received non-formatted information, intelligent information extraction is first performed to obtain key information such as the current position, time, and attributes of the target, wherein the current position of the target and the reconnaissance time are used to establish spatiotemporal coding. Then, according to the type of cockpit human-computer interaction scene, the model structure is selected, and finally a spatiotemporal dynamic knowledge graph is generated.

[0093] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S104, if it is determined that the knowledge type is multimodal knowledge, performs multimodal knowledge modeling on the multimodal knowledge to construct a multimodal knowledge graph, includes:

[0094] S1041, representing the multimodal knowledge as entities through a multimodal knowledge representation learning method, and obtaining association relationships between the multimodal knowledge;

[0095] That is, through the multimodal knowledge representation learning method, the information of all modalities in the multimodal knowledge is mapped to a unified language-oriented embedding space, and the semantics of the multimodal knowledge are aligned. Specifically, in this embodiment, the multimodal knowledge representation learning method can be implemented using a framework such as languageBind.

[0096] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S104, if it is determined that the knowledge type is multimodal knowledge, performs multimodal knowledge modeling on the multimodal knowledge to construct a multimodal knowledge graph, includes:

[0097] According to the logical architecture of the multimodal knowledge graph, the multimodal knowledge graph is constructed in combination with the multimodal knowledge learned by the multimodal knowledge representation;

[0098] S1042, the logical architecture of the multimodal knowledge graph includes: ontology elements, entity classes, event classes, multimodal classes, label classes and rule classes.

[0099] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S104, if it is determined that the knowledge type is multimodal knowledge, performs multimodal knowledge modeling on the multimodal knowledge to construct a multimodal knowledge graph, includes:

[0100] S1043, the relationship between the entity class and the multimodal class is obtained by extracting the multimodal knowledge;

[0101] S1044, the relationship between the multimodal classes is obtained by processing and aligning the multimodal knowledge;

[0102] S1045, the relationship between the multimodal class and the tag class is obtained by extracting and semantically fusing the multimodal knowledge to obtain the business features of the multimodal knowledge.

[0103] Specifically, in the embodiment of the present invention, the logical architecture of the multimodal knowledge graph is mainly composed of ontology elements, entities, events, multimodalities, labels, and rules, wherein:

[0104] Ontology elements: describe the top-level ontology framework of the multimodal knowledge graph, constrain the definition of the knowledge system, attributes and relationships in the graph; based on the traditional knowledge graph, expand the attributes and association representation of multimodal documents;

[0105] Entity class: describes the relationship between specific instances, which is consistent with the traditional knowledge graph;

[0106] Event class: describes the relationship between events, event elements and instances, which is consistent with the traditional knowledge graph;

[0107] Multimodal class: describes various types of multimodal files and their associations in the knowledge graph; types include images, videos, sequences, texts, etc. For each file, each multimodal file is recorded through entities, and its meta information is recorded through attributes, including size, resolution, semantic vector (obtained through multimodal alignment), etc., and the association between multimodal and entity, and the similarity, identity, splitting, etc. between multimodal.

[0108] Tag class: describes the business-oriented tag system, which is generally divided into multiple dimensions and is divided and constructed according to the application. It is semantically associated with various elements in the entity class and multimodal entity class;

[0109] Rule class: describes business-oriented rule information, is divided and constructed according to the business, and is associated with each element in the entity class.

[0110] The relationships in the multimodal knowledge graph include the following categories:

[0111] The relationship between entities and multimodality: It is mainly divided into three categories: strong semantic relationship and weak semantic relationship (correlation relationship) from entities to documents, and inclusion relationship from documents to entities; generally obtained through multimodal extraction technology;

[0112] The relationship between entities and labels: describes the business characteristics of entities for specific scenarios, which is generally obtained through data statistics and analysis methods;

[0113] Multimodal relations: describe the semantic correlation between multimodal files, such as inclusion, similarity, identity, splitting, etc.; generally obtained through multimodal data processing and alignment technology;

[0114] The relationship between multimodality and tags: describes the business characteristics of multimodal file content. Files can be associated with a single tag or a tag group. This is usually obtained through multimodal extraction and semantic fusion technology.

[0115] Relationship between tags: describes the hierarchical and related relationships between business tags; generally obtained through data mapping and text extraction technology.

[0116] In addition to S104, multimodal knowledge can be pre-trained by anchoring the language modality and adopting a series of optimized contrastive learning strategies. In this process, the multimodal knowledge graph can encode data from different modalities into representations compatible with the language modality, and can ensure semantic consistency between multimodal knowledge.

[0117] In the method for constructing a hierarchical knowledge graph for cockpit human-computer interaction described in an embodiment of the present invention, the step S105, according to the cockpit human-computer interaction scenario, cutting and fusing the basic static knowledge graph, the spatiotemporal dynamic knowledge graph, and the multimodal knowledge graph to generate an airborne knowledge graph, includes:

[0118] S1051, after obtaining the basic static knowledge graph, the spatiotemporal dynamic knowledge graph, and the multimodal knowledge graph, obtaining the display requirements of the onboard knowledge graph in the cockpit human-computer interaction scene;

[0119] S1052, according to the display requirements, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cropped and merged through the pattern references of the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph to obtain the airborne knowledge graph.

[0120] Specifically, in the present embodiment, a graph pattern reference function is provided through S105 to realize the cropping and fusion of graphs. Through this step, a target graph to be generated can be newly created or selected, and concepts in one or more graphs in the existing graph pattern library can be selected and cropped, and they can be hung or mapped under the target graph to realize the fusion of multiple graph patterns. After the concept is introduced, the attributes defined under the introduced concept are displayed in the pattern introduction information area. Through this step, the attributes to be imported into the target graph can be customized and selected. After the introduction operation is completed, the latest cropped or fused airborne graph can be viewed in the human-computer interaction interface. In addition, in the present embodiment, the airborne knowledge graph is displayed in a visual manner in the human-computer interaction interface, and the visual manner supports display in different forms.

[0121] like Figure 2 As shown, based on a method for constructing a hierarchical knowledge graph for cockpit human-computer interaction provided by an embodiment of the present invention, an embodiment of the present invention further provides a system for constructing a hierarchical knowledge graph for cockpit human-computer interaction, the system comprising:

[0122] The knowledge type determination module 10 is used to determine the knowledge type acquired in the cockpit human-computer interaction scene, and the knowledge type includes: basic static knowledge, spatiotemporal dynamic knowledge and multimodal knowledge;

[0123] A basic static knowledge graph construction module 20 is used to perform triple knowledge modeling on the basic static knowledge and construct a basic static knowledge graph if it is determined that the knowledge type is basic static knowledge;

[0124] A spatiotemporal dynamic knowledge graph construction module 30 is used to perform spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge and construct a spatiotemporal dynamic knowledge graph if it is determined that the knowledge type is spatiotemporal dynamic knowledge;

[0125] A multimodal knowledge graph construction module 40 is used to perform multimodal knowledge modeling on the multimodal knowledge and construct a multimodal knowledge graph if it is determined that the knowledge type is multimodal knowledge;

[0126] The airborne knowledge graph generation module 50 is used to cut and fuse the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph according to the cockpit human-computer interaction scenario to generate an airborne knowledge graph.

[0127] In addition, based on a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction provided in an embodiment of the present invention, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the methods described in a method for constructing a hierarchical knowledge graph for cockpit human-computer interaction are implemented.

[0128] In addition, based on the above methods, systems and electronic devices, the embodiments of the present invention also provide a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above method embodiments.

[0129] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0130] In combination with the above embodiments, the present invention discloses a method and system for constructing a hierarchical knowledge graph for cockpit human-computer interaction, wherein the method includes: first, determining the type of knowledge acquired in the cockpit human-computer interaction scene, if the knowledge type is determined to be basic static knowledge, performing triple knowledge modeling on the basic static knowledge to construct a basic static knowledge graph; if the knowledge type is determined to be spatiotemporal dynamic knowledge, performing spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge to construct a spatiotemporal dynamic knowledge graph; if the knowledge type is determined to be multimodal knowledge, performing multimodal knowledge modeling on the multimodal knowledge to construct a multimodal knowledge graph; according to the cockpit human-computer interaction scene, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cut and merged, and finally an airborne knowledge graph is generated. This solves the problem that the prior art cannot fully utilize the combination of multiple modal knowledge, and thus cannot realize the airborne deployment of large-scale knowledge graphs.

[0131] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0133] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for constructing a hierarchical knowledge graph for cockpit human-computer interaction, characterized in that: The method comprises: Determine the types of knowledge acquired in the cockpit human-computer interaction scenario, the types of knowledge including: basic static knowledge, spatiotemporal dynamic knowledge, and multimodal knowledge; If the knowledge type is determined to be basic static knowledge, the basic static knowledge is modeled as a triplet knowledge to construct a basic static knowledge graph; If it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct a spatiotemporal dynamic knowledge graph, including: determining the type of the cockpit human-computer interaction scene, the type of the cockpit human-computer interaction scene includes: a time topic model, a space topic model and a spatiotemporal topic model; If the type of the cockpit human-computer interaction scene is the time theme model, a recurrent neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a time theme spatiotemporal dynamic knowledge graph; If the type of the cockpit human-computer interaction scene is the spatial theme model, a convolutional neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatial theme spatiotemporal dynamic knowledge graph; If the type of the cockpit human-computer interaction scene is the spatiotemporal topic model, a convolutional neural network and a recurrent neural network are combined as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatiotemporal topic spatiotemporal dynamic knowledge graph; If it is determined that the knowledge type is multimodal knowledge, multimodal knowledge modeling is performed on the multimodal knowledge to construct a multimodal knowledge graph; According to the cockpit human-computer interaction scenario, the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph are cut and fused to generate an airborne knowledge graph.

2. A method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 1, characterized in that: If it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct the spatiotemporal dynamic knowledge graph, including: According to the cockpit human-machine interaction scenario, establishing spatiotemporal coding rules; The spatiotemporal coding rule divides the geographical locations involved in the cockpit human-machine interaction scenario according to time on the basis of a global grid model; According to the spatiotemporal coding rules, the spatiotemporal dynamic knowledge is converted into information indexes.

3. A method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 2, characterized in that: If it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct the spatiotemporal dynamic knowledge graph, including: If the spatiotemporal dynamic knowledge is unstructured spatiotemporal information, extracting information from the unstructured spatiotemporal information to obtain key spatiotemporal information, wherein the key spatiotemporal information includes position information and time information; According to the key spatiotemporal information obtained by information extraction and in combination with the spatiotemporal coding rules, the unstructured spatiotemporal information is converted into the information index.

4. The method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 3 is characterized in that: If it is determined that the knowledge type is spatiotemporal dynamic knowledge, the spatiotemporal dynamic knowledge is modeled to construct the spatiotemporal dynamic knowledge graph, including: In combination with the information index, the spatiotemporal dynamic knowledge is represented and modeled to obtain the spatiotemporal dynamic knowledge graph, including: Determining the categories of the spatiotemporal dynamic knowledge, wherein the categories of the spatiotemporal dynamic knowledge include time concepts, space concepts, and spatiotemporal concepts; Determine the time concept, space concept and the relationship between the time and space concepts in the time and space dynamic knowledge; The time concept includes time point, time period and time series; The spatial concepts include points of interest, areas of interest, and paths; The space-time concept includes space-time sequences.

5. The method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 1, characterized in that: If it is determined that the knowledge type is multimodal knowledge, the multimodal knowledge is modeled to construct the multimodal knowledge graph, including: By using a multimodal knowledge representation learning method, the multimodal knowledge is represented as an entity, and the association relationship between the multimodal knowledge is obtained; That is, through the multimodal knowledge representation learning method, the information of all modalities in the multimodal knowledge is mapped to a unified language-oriented embedding space to align the semantics of the multimodal knowledge.

6. The method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 5, characterized in that: If it is determined that the knowledge type is multimodal knowledge, the multimodal knowledge is modeled to construct the multimodal knowledge graph, including: According to the logical architecture of the multimodal knowledge graph, the multimodal knowledge graph is constructed in combination with the multimodal knowledge learned by the multimodal knowledge representation; The logical architecture of the multimodal knowledge graph includes: ontology elements, entity classes, event classes, multimodal classes, label classes and rule classes.

7. The method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 6, characterized in that: If it is determined that the knowledge type is multimodal knowledge, the multimodal knowledge is modeled to construct the multimodal knowledge graph, including: The relationship between the entity class and the multimodal class is obtained by extracting the multimodal knowledge; The relationship between the multimodal classes is obtained by processing and aligning the multimodal knowledge; The relationship between the multimodal class and the tag class is obtained by extracting and semantically fusing the multimodal knowledge to obtain the business features of the multimodal knowledge.

8. The method for constructing a hierarchical knowledge graph for cockpit human-computer interaction according to claim 1, characterized in that: According to the cockpit human-machine interaction scenario, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cut and merged to generate the airborne knowledge graph, including: After obtaining the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph, obtaining the display requirements of the onboard knowledge graph in the cockpit human-computer interaction scene; According to the display requirements, the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph are cropped and fused through the pattern references of the basic static knowledge graph, the spatiotemporal dynamic knowledge graph and the multimodal knowledge graph to obtain the airborne knowledge graph.

9. A hierarchical knowledge graph construction system for cockpit human-computer interaction, characterized in that: The system is used to implement a hierarchical knowledge graph construction method for cockpit human-computer interaction as described in any one of claims 1 to 8, and the system includes: A knowledge type determination module is used to determine the type of knowledge acquired in the cockpit human-computer interaction scenario, wherein the knowledge types include: basic static knowledge, spatiotemporal dynamic knowledge, and multimodal knowledge; A basic static knowledge graph construction module is used to perform triple knowledge modeling on the basic static knowledge and construct a basic static knowledge graph if the knowledge type is determined to be basic static knowledge; A spatiotemporal dynamic knowledge graph construction module is used to, if it is determined that the knowledge type is spatiotemporal dynamic knowledge, perform spatiotemporal dynamic knowledge modeling on the spatiotemporal dynamic knowledge and construct a spatiotemporal dynamic knowledge graph, including determining the type of the cockpit human-computer interaction scene, the type of the cockpit human-computer interaction scene includes: a time topic model, a space topic model and a spatiotemporal topic model; If the type of the cockpit human-computer interaction scene is the time theme model, a recurrent neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a time theme spatiotemporal dynamic knowledge graph; If the type of the cockpit human-computer interaction scene is the spatial theme model, a convolutional neural network is used as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatial theme spatiotemporal dynamic knowledge graph; If the type of the cockpit human-computer interaction scene is the spatiotemporal topic model, a convolutional neural network and a recurrent neural network are combined as the model structure, and the information index corresponding to the spatiotemporal dynamic knowledge and the category of the spatiotemporal dynamic knowledge are combined to generate a spatiotemporal topic spatiotemporal dynamic knowledge graph; A multimodal knowledge graph construction module, for, if it is determined that the knowledge type is multimodal knowledge, performing multimodal knowledge modeling on the multimodal knowledge and constructing a multimodal knowledge graph; The airborne knowledge graph generation module is used to cut and fuse the basic static knowledge graph, spatiotemporal dynamic knowledge graph and multimodal knowledge graph according to the cockpit human-computer interaction scenario to generate an airborne knowledge graph.

Images