Multimodal knowledge graph construction method, device, equipment and medium
Through the multimodal knowledge graph construction method, combined with image information and text knowledge graph, the problem that traditional methods cannot use unstructured data is solved, efficient information extraction and knowledge system construction of combat scenarios are achieved, and the level of intelligence in combat decision-making is improved.
Patent Information
- Application Number
- CN202411839159.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-08
AI Technical Summary
Traditional knowledge graph construction methods mostly rely on structured data and cannot make full use of information in unstructured data. Especially in combat scenarios, real-time acquisition and analysis of visual information is of great significance to understanding complex environments, assessing threats, and making tactical decisions.
A multimodal knowledge graph construction method is provided, which preprocesses data by obtaining target multimodal data, extracts image information and constructs text knowledge graphs, and combines the two with correlation processing, and finally completes the relationship to form the final multimodal knowledge graph.
It has achieved efficient extraction of key entities and relationships in combat scenarios, improved the acquisition and processing capabilities of combat scenario information, and formed a more comprehensive and accurate knowledge system, thereby improving the intelligence level of combat command and decision-making.
Smart Images
Figure CN119783790A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the application field of knowledge graph technology, and in particular to a method, device, equipment and medium for constructing a multimodal knowledge graph. Background Art
[0002] With the continuous evolution of modern warfare, the complexity and dynamics of combat scenarios are increasing. In order to improve the effectiveness and real-time nature of combat decisions, the innovation of information acquisition and analysis technology is particularly important. In this context, knowledge graphs, as an effective way of organizing and expressing information, have gradually become an important research direction in the combat field.
[0003] Traditional knowledge graph construction methods mostly rely on structured data and often fail to fully utilize information in unstructured data (such as images, videos, etc.). In combat scenarios, real-time acquisition and analysis of visual information is important for understanding complex environments, assessing threats, and making tactical decisions. Therefore, there is an urgent need for a knowledge graph construction method that can effectively integrate multimodal information (such as images, videos, texts, etc.) to enhance the understanding and analysis capabilities of combat scenarios. Summary of the invention
[0004] In order to solve the above technical problems, the present disclosure provides a method, device, equipment and medium for constructing a multimodal knowledge graph.
[0005] In a first aspect, the present disclosure provides a method for constructing a multimodal knowledge graph, comprising:
[0006] Acquire target multimodal data, and perform data preprocessing on the target multimodal data;
[0007] Performing image information extraction processing on the target multimodal data to obtain corresponding image information, and performing text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph;
[0008] Combining and associating the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph;
[0009] Perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
[0010] In a second aspect, the present disclosure provides a multimodal knowledge graph construction device, comprising:
[0011] A data acquisition module, used to acquire target multimodal data and perform data preprocessing on the target multimodal data;
[0012] A first processing module is used to perform image information extraction processing on the target multimodal data to obtain corresponding image information, and to perform text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph;
[0013] A second processing module is used to combine and associate the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph;
[0014] The third processing module is used to perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
[0015] In a third aspect, the present disclosure provides a multimodal knowledge graph construction device, including:
[0016] processor;
[0017] A memory for storing executable instructions;
[0018] Among them, the processor is used to read executable instructions from the memory and execute the executable instructions to implement the multimodal knowledge graph construction method of the first aspect.
[0019] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements the multimodal knowledge graph construction method of the first aspect.
[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0021] The multimodal knowledge graph construction method, device, equipment and medium of the disclosed embodiment can obtain target multimodal data, and perform data preprocessing on the target multimodal data, then perform image information extraction processing on the target multimodal data to obtain the corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain the corresponding text knowledge graph, and then combine and associate the image information and the text knowledge graph to obtain a preliminary multimodal knowledge graph, and finally perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph. Thus, combining image information and text knowledge graphs can realize efficient extraction of key entities and relationships in combat scenarios, which can not only improve the acquisition and processing capabilities of combat scenario information, but also help to form a more comprehensive and accurate knowledge system, thereby improving the intelligence level of combat command and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0023] Figure 1 A schematic diagram of a process for constructing a multimodal knowledge graph provided in an embodiment of the present disclosure;
[0024] Figure 2 A flowchart of a data preprocessing method provided by an embodiment of the present disclosure;
[0025] Figure 3 A schematic diagram of the structure of a C2F module provided in an embodiment of the present disclosure;
[0026] Figure 4 A schematic diagram of a scene of an image information extraction and processing method provided by an embodiment of the present disclosure;
[0027] Figure 5 A schematic diagram of the structure of a Bi-LSTM model provided in an embodiment of the present disclosure;
[0028] Figure 6 A schematic diagram of the structure of a text knowledge graph provided in an embodiment of the present disclosure;
[0029] Figure 7 A schematic diagram of the structure of a multimodal knowledge graph provided in an embodiment of the present disclosure;
[0030] Figure 8 A schematic diagram of a process of constructing another multimodal knowledge graph provided in an embodiment of the present disclosure;
[0031] Fig. 9 A schematic diagram of the structure of a multimodal knowledge graph construction device provided in an embodiment of the present disclosure;
[0032] Fig.10 A schematic diagram of the structure of a multimodal knowledge graph construction device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0033] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0034] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0035] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0036] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0037] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0038] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0039] In order to solve the above problems, the present disclosure provides a method, device, equipment and medium for constructing a multimodal knowledge graph. Figures 1 to 8 The multimodal knowledge graph construction method provided in the embodiment of the present disclosure is described in detail.
[0040] Figure 1 A flow chart of a multimodal knowledge graph construction method provided by an embodiment of the present disclosure is shown.
[0041] In the disclosed embodiment, the multimodal knowledge graph construction method may be executed by an electronic device, wherein the electronic device may include but is not limited to devices such as computer devices, cloud servers or cloud server clusters.
[0042] like Figure 1 As shown, the multimodal knowledge graph construction method may include the following steps.
[0043] S110: Acquire target multimodal data, and perform data preprocessing on the target multimodal data.
[0044] In an embodiment of the present disclosure, the electronic device may acquire target multimodal data and perform data preprocessing on the target multimodal data.
[0045] Optionally, the target multimodal data may be multimodal data related to the combat scenario, wherein the target multimodal data may include associated target text data and target image data.
[0046] Specifically, the electronic device can obtain target multimodal data, including associated target text data and target image data. For example, it can obtain real-time image data of combat scenes by means of satellite images, drone aerial photography, on-site monitoring videos, etc., and collect text data such as combat exercises, battle reports, and intelligence analysis at the same time; collect text data related to the project through various public databases, online forums, social media platforms and other channels, and collect images related to text data from image libraries, online galleries, user uploads and other sources, and then perform data preprocessing on the target multimodal data.
[0047] S120. Perform image information extraction processing on the target multimodal data to obtain corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph.
[0048] In an embodiment of the present disclosure, the electronic device may perform image information extraction processing on the target multimodal data to obtain corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph.
[0049] Optionally, the text knowledge graph may be an image showing the structural relationship between different texts and images.
[0050] Specifically, the electronic device can extract image information from the target multimodal data to obtain corresponding image information, and extract entities from the target multimodal data to construct a corresponding text knowledge graph.
[0051] S130: Combine and associate the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph.
[0052] In the disclosed embodiment, the electronic device combines and associates the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph.
[0053] Optionally, the multimodal knowledge graph may be in the form of a graph that enhances knowledge representation and understanding by integrating multiple modal data (such as language, vision, hearing, etc.).
[0054] Specifically, the electronic device can combine and associate the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph.
[0055] S140. Perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
[0056] In an embodiment of the present disclosure, the electronic device may perform relationship completion processing on a preliminary multimodal knowledge graph to obtain a final multimodal knowledge graph.
[0057] Therefore, in the disclosed embodiment, it is possible to obtain target multimodal data, and perform data preprocessing on the target multimodal data, then perform image information extraction processing on the target multimodal data to obtain corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain the corresponding text knowledge graph, and then perform combined association processing on the image information and the text knowledge graph to obtain a preliminary multimodal knowledge graph, and finally perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph. Therefore, combining image information and text knowledge graphs can achieve efficient extraction of key entities and relationships in combat scenarios, which can not only improve the ability to obtain and process combat scenario information, but also help to form a more comprehensive and accurate knowledge system, thereby improving the level of intelligence in combat command and decision-making.
[0058] Optionally, S110 may specifically include: performing data cleaning processing on the target text data to obtain cleaned target text data; performing data annotation processing on the target image data to obtain annotated target image data; performing association matching processing on the cleaned target text data and the annotated target image data to obtain the target multimodal data.
[0059] In the disclosed embodiment, the electronic device may perform data cleaning on the target text data to obtain cleaned target text data.
[0060] For example, electronic devices can collect text data related to the project through various public databases, online forums, social media platforms and other channels, and clean the collected text data, including removing useless formats, special characters, stop words, and correcting spelling errors, to obtain cleaned target text data.
[0061] Furthermore, the electronic device may perform data annotation processing on the target image data to obtain annotated target image data.
[0062] For example, electronic devices can collect images related to text data from image libraries, online galleries, user uploads and other sources, and annotate the collected image data, including identifying key objects, scenes, actions, etc. in the image, and assigning corresponding labels to these elements to obtain annotated target image data.
[0063] Furthermore, the electronic device may perform association matching processing on the cleaned target text data and the annotated target image data to obtain the target multimodal data.
[0064] Figure 2 A flow chart of a data preprocessing method provided by an embodiment of the present disclosure is shown.
[0065] like Figure 2 As shown, the electronic device can perform data cleaning processing on the target text data to obtain cleaned target text data, and perform data annotation processing on the target image data to obtain annotated target image data. For example, firstly, satellite images, drone aerial photography, on-site monitoring videos and other means are used to obtain real-time image data of combat scenes, and text data such as combat exercises, combat reports, and intelligence analysis are collected at the same time; then, a combination of manual annotation and automatic annotation is used to annotate the targets in the image and video data, including combat equipment, personnel, terrain and other key entities; finally, the collected data is cleaned to remove irrelevant data, and the generalization ability of the model is improved through data enhancement technology (such as image flipping, rotation, noise addition, etc.). Finally, the cleaned target text data is matched with the annotated target image data, and the associated matching process is performed to obtain the target multimodal data (i.e., text-image data set).
[0066] Optionally, S120 may specifically include: performing model training based on the optimized primary target detection algorithm and the preprocessed training data set to obtain a trained primary target detection model; inputting the target multimodal data into the pre-trained primary target detection model to perform image information extraction processing to obtain corresponding image information.
[0067] In an embodiment of the present disclosure, the electronic device may perform model training based on the optimized primary target detection algorithm and the preprocessed training data set to obtain a trained primary target detection model.
[0068] Alternatively, the You Only Look Once (YOLO) algorithm can quickly locate and identify multiple targets in an image by converting the target detection problem into a regression problem, which is suitable for application in dynamic and complex combat environments.
[0069] Specifically, the electronic device can improve the primary target detection algorithm by adding a C2F (Faster Implementation of CSP Bottleneck with 2convolutions) module and improving the loss function of the network.
[0070] Figure 3 A schematic structural diagram of a C2F module provided in an embodiment of the present disclosure is shown.
[0071] like Figure 3 As shown in the figure, the C2F module can make low-level features interact with high-level features, so that the algorithm has better global perception ability, which is conducive to small target detection. In addition, the module can greatly reduce the number of parameters and improve the model detection speed.
[0072] Furthermore, the loss function of the network is improved and NWD loss is introduced. This loss function uses a new measurement method: the box is modeled as a Gaussian distribution, and then the Wasserstein distance is used to measure the similarity between the two distributions, as follows:
[0073]
[0074] In NNW, the similarity can be measured even if the two target boxes do not overlap at all. Therefore, NWD is insensitive to the scale of the target and is more stable for detecting small targets.
[0075] Specifically, electronic devices can perform model training based on the optimized primary target detection algorithm and the preprocessed training data set. For example, according to the characteristics of the combat scenario, the network structure of the YOLO algorithm can be adjusted, the loss function can be optimized, and the attention mechanism can be introduced to enhance the detection capabilities of small targets and dense scenes. The preprocessed data set is then used to train the algorithm, and the model parameters are optimized through multiple rounds of iterations to ensure that the detection accuracy meets the expected standards, thereby obtaining a trained primary target detection model.
[0076] Furthermore, the electronic device may input the target multimodal data into a pre-trained primary target detection model to perform image information extraction processing to obtain corresponding image information.
[0077] Figure 4 A scene schematic diagram of an image information extraction and processing method provided by an embodiment of the present disclosure is shown.
[0078] like Figure 4As shown, the electronic device can input the target multimodal data into a pre-trained target detection model to perform image information extraction and processing, such as inputting the real-time image data of the combat scene into the YOLO model to perform image information extraction and processing to obtain the corresponding image information, such as naval equipment, soldiers, army equipment, etc.
[0079] Optionally, S120 may also specifically include: performing model training based on a deep learning model and a preprocessed training data set to obtain a trained named entity recognition model; inputting the target multimodal data into the trained named entity recognition model to obtain corresponding named entity recognition results; and constructing the text knowledge graph based on the named entity recognition results.
[0080] In an embodiment of the present disclosure, the electronic device can perform model training based on a deep learning model and a preprocessed training data set to obtain a trained named entity recognition model.
[0081] Specifically, the electronic device can select a suitable deep learning model (such as time series prediction based on long short-term memory network (LSTM) (Bidirectional Long Short Term Memory, Bi-LSTM), conditional random field (conditional random field, CRF), etc.), and perform model training in combination with the preprocessed training data set to obtain a trained named entity recognition model (Named Entity Recognition, NER).
[0082] Figure 5 A schematic diagram of the structure of a Bi-LSTM model provided by an embodiment of the present disclosure is shown.
[0083] like Figure 5 As shown in the figure, the Bi-LSTM (Bidirectional Long Short Term Memory) network is a time series prediction method based on the long short-term memory network (LSTM); it combines the bidirectional model and the gating mechanism of LSTM, and is composed of two independent LSTM networks. When Bi-LSTM processes sequence data, the input sequence is input into two LSTM networks in positive and reverse order for feature extraction, and the output vector formed by concatenating the two output vectors is used as the final output of the time step. The model design concept of Bi-LSTM is to make the feature data obtained at time t have information between the past and the future at the same time; the parameters of the two LSTM networks in Bi-LSTM are independent of each other, and they only share the same batch of sequence data.
[0084] Furthermore, the electronic device may input the target multimodal data into a trained named entity recognition model to obtain corresponding named entity recognition results, and construct the text knowledge graph based on the named entity recognition results.
[0085] Specifically, the electronic device inputs the target multimodal data into a trained named entity recognition model. The named entity recognition model (NER) can output corresponding entities and identify the relationships between entities. It then uses RDF or other forms to represent the extracted knowledge, including entities, attributes, and relationships, to form a preliminary text knowledge graph. Finally, the constructed text knowledge graph is cleaned to remove duplicate and irrelevant information to ensure the accuracy and completeness of the graph.
[0086] Figure 6 A structural schematic diagram of a text knowledge graph provided by an embodiment of the present disclosure is shown.
[0087] like Figure 6 As shown, the electronic device identifies the relationship between entities in the text based on the output of the NER model, and then uses RDF to represent the extracted knowledge, including entities, attributes and relationships, to form a preliminary knowledge graph. For example, "soldiers land and sea integrated landing" identifies the entities "soldier", "sea", "land", "integration", and "landing", and then uses RDF to represent the extracted knowledge, including entities, attributes and relationships, to form a preliminary knowledge graph, such as "soldier" is: personnel, "naval equipment" is: equipment, "army equipment" is: equipment, "integration" is: method, "landing" is: behavior, etc.
[0088] Optionally, S130 may specifically include: associating entities in the text knowledge graph with results in the image information to obtain corresponding association results; and expanding the relationship types of the association results to obtain a preliminary multimodal knowledge graph.
[0089] In an embodiment of the present disclosure, the electronic device may associate entities in the text knowledge graph with results in the image information to obtain corresponding association results, and expand the relationship types of the association results to obtain a preliminary multimodal knowledge graph.
[0090] Specifically, the electronic device first associates the entities in the text knowledge graph with the detection results in the image to form a unified knowledge representation; then, based on the image information and text relationship, it expands the relationship types in the knowledge graph to ensure that the information in the multimodal knowledge graph is semantically consistent and establishes a multi-dimensional knowledge structure.
[0091] Figure 7 A schematic diagram of the structure of a multimodal knowledge graph provided by an embodiment of the present disclosure is shown.
[0092] like Figure 7 As shown, the electronic device can associate entities in the text knowledge graph with detection results in the image, such as associating "soldier" in the text knowledge graph with personnel identification pictures in the image information, associating "naval equipment" in the text knowledge graph with equipment identification pictures in the image information, and associating "army equipment" in the text knowledge graph with equipment identification pictures in the image information, and based on the relationship between image information and text, expand the relationship type in the knowledge graph, such as the behavior is "landing", the usage method is "integration", etc., so as to obtain a preliminary multimodal knowledge graph.
[0093] Optionally, S140 may specifically include: reasoning the known association results in the preliminary multimodal knowledge graph according to the graph reasoning algorithm to obtain implicit multimodal relationships; completing the implicit multimodal relationships through preset rule reasoning to obtain the final multimodal knowledge graph.
[0094] In the disclosed embodiment, the electronic device can infer the known association results in the preliminary multimodal knowledge graph according to the graph reasoning algorithm to obtain the implicit multimodal relationship, and then complete the implicit multimodal relationship through preset rule reasoning to obtain the final multimodal knowledge graph.
[0095] Figure 7 A schematic diagram of the structure of another multimodal knowledge graph provided by an embodiment of the present disclosure is shown.
[0096] like Figure 7 As shown, the electronic device can infer the known association results in the preliminary multimodal knowledge graph according to the graph reasoning algorithm to obtain implicit multimodal relationships. For example, there is an implicit "user" relationship between "soldier" and "army equipment" and "navy equipment". Then, the implicit multimodal relationship is completed through preset rule reasoning to obtain the final multimodal knowledge graph, that is, the "user" relationship is completed between "soldier" and "army equipment" and "navy equipment".
[0097] In some embodiments, the electronic device uses Neo4j to store the constructed multimodal knowledge graph. Neo4j can quickly execute complex graph queries. Through graph traversal algorithms, it can run efficiently on large-scale data sets and is suitable for real-time applications. In addition, Neo4j can be horizontally expanded to support large-scale graph data storage, which is suitable for growing data needs.
[0098] Figure 8 A flowchart of another multimodal knowledge graph construction method provided by an embodiment of the present disclosure is shown.
[0099] like Figure 8As shown, the electronic device can collect text data related to the project through various public databases, online forums, social media platforms and other channels, and clean the collected text data, including removing useless formats, special characters, stop words, and correcting spelling errors, etc. Images related to the text data are collected from image libraries, online galleries, user uploads and other sources, and the collected image data are annotated, including identifying key objects, scenes, actions, etc. in the image, and assigning corresponding labels to these elements.
[0100] Next, the electronic device can improve the primary target detection algorithm by adding a C2F module, improving the network loss function, and performing model training based on the optimized primary target detection algorithm and the preprocessed training data set. For example, according to the characteristics of the combat scene, the network structure of the YOLO algorithm is adjusted, the loss function is optimized, and the attention mechanism is introduced to enhance the detection capabilities of small targets and dense scenes. Then, the algorithm is trained using the preprocessed data set, and the model parameters are optimized through multiple rounds of iterations to ensure that the detection accuracy reaches the expected standard, and a trained primary target detection model is obtained. Furthermore, the electronic device can input the target multimodal data into the pre-trained primary target detection model for image information extraction and processing to obtain the corresponding image information.
[0101] Then, the electronic device can select a suitable deep learning model and combine it with the preprocessed training data set to perform model training to obtain a trained named entity recognition model (NER). Furthermore, the electronic device can input the target multimodal data into the trained named entity recognition model to obtain the corresponding named entity recognition result, and construct the text knowledge graph based on the named entity recognition result.
[0102] Finally, the entities in the text knowledge graph are associated with the detection results in the image to form a unified knowledge representation; then, based on the image information and text relationship, the relationship types in the knowledge graph are expanded to ensure that the information in the multimodal knowledge graph is semantically consistent and a multi-dimensional knowledge structure is established. Furthermore, the electronic device can infer the known association results in the preliminary multimodal knowledge graph based on the graph reasoning algorithm to obtain the implicit multimodal relationship, and then complete the implicit multimodal relationship through the preset rule reasoning to obtain the final multimodal knowledge graph.
[0103] In some embodiments, the electronic device uses Neo4j to store the constructed multimodal knowledge graph. Neo4j can quickly execute complex graph queries. Through graph traversal algorithms, it can run efficiently on large-scale data sets and is suitable for real-time applications. In addition, Neo4j can be horizontally expanded to support large-scale graph data storage, which is suitable for growing data needs.
[0104] Therefore, in the disclosed embodiment, the YOLO algorithm is combined with the knowledge graph construction to achieve efficient extraction of key entities and relationships in combat scenarios, providing strong support for combat decision-making. It can not only improve the ability to obtain and process combat scenario information, but also help to form a more comprehensive and accurate knowledge system, thereby improving the intelligence level of combat command and decision-making.
[0105] Fig. 9 A schematic diagram of the structure of a multimodal knowledge graph construction device provided by an embodiment of the present disclosure is shown.
[0106] like Fig. 9 As shown, the multimodal knowledge graph construction device 900 may include a data acquisition module 910, a first processing module 920, a second processing module 930 and a third processing module 940.
[0107] The data acquisition module 910 may be used to acquire target multimodal data and perform data preprocessing on the target multimodal data.
[0108] The first processing module 920 can be used to perform image information extraction processing on the target multimodal data to obtain corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph.
[0109] The second processing module 930 can be used to combine and associate the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph.
[0110] The third processing module 940 can be used to perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
[0111] Therefore, in the disclosed embodiment, it is possible to obtain target multimodal data, and perform data preprocessing on the target multimodal data, then perform image information extraction processing on the target multimodal data to obtain corresponding image information, and perform text knowledge graph construction processing on the target multimodal data to obtain the corresponding text knowledge graph, and then perform combined association processing on the image information and the text knowledge graph to obtain a preliminary multimodal knowledge graph, and finally perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph. Therefore, combining image information and text knowledge graphs can achieve efficient extraction of key entities and relationships in combat scenarios, which can not only improve the ability to obtain and process combat scenario information, but also help to form a more comprehensive and accurate knowledge system, thereby improving the level of intelligence in combat command and decision-making.
[0112] In some embodiments of the present disclosure, the target multimodal data includes associated target text data and target image data.
[0113] In some embodiments of the present disclosure, the data acquisition module 910 may specifically include a first processing unit, a second processing unit, and a third processing unit.
[0114] The first processing unit can be used to perform data cleaning processing on the target text data to obtain cleaned target text data.
[0115] The second processing unit can be used to perform data annotation processing on the target image data to obtain annotated target image data.
[0116] The third processing unit can be used to perform association matching processing on the cleaned target text data and the annotated target image data to obtain the target multimodal data.
[0117] In some embodiments of the present disclosure, the first processing module 920 may specifically include a first training unit and a fourth processing unit.
[0118] The first training unit can be used to perform model training based on the optimized primary target detection algorithm and the preprocessed training data set to obtain a trained primary target detection model.
[0119] The fourth processing unit can be used to input the target multimodal data into a pre-trained primary target detection model to perform image information extraction processing to obtain corresponding image information.
[0120] In some embodiments of the present disclosure, the first processing module 920 may further specifically include a second training unit, a fifth processing unit and a graph construction unit.
[0121] The second training unit can be used to perform model training based on the deep learning model and the preprocessed training data set to obtain a trained named entity recognition model.
[0122] The fifth processing unit can be used to input the target multimodal data into a trained named entity recognition model to obtain a corresponding named entity recognition result.
[0123] The graph construction unit can be used to construct the text knowledge graph based on the named entity recognition result.
[0124] In some embodiments of the present disclosure, the second processing module 930 may specifically include a result association unit and a relationship expansion unit.
[0125] The result association unit can be used to associate the entities in the text knowledge graph with the results in the image information to obtain corresponding association results.
[0126] The relationship expansion unit can be used to expand the relationship type of the association result to obtain a preliminary multimodal knowledge graph.
[0127] In some embodiments of the present disclosure, the third processing module 940 may specifically include a relational reasoning unit and a sixth processing unit.
[0128] The relational reasoning unit can be used to infer known association results in the preliminary multimodal knowledge graph according to the graph reasoning algorithm to obtain implicit multimodal relationships.
[0129] The sixth processing unit can be used to complete the implicit multimodal relationship through preset rule reasoning to obtain the final multimodal knowledge graph.
[0130] It should be noted that Fig. 9 The multimodal knowledge graph construction device 900 shown can execute Figures 1 to 8 The various steps in the method embodiment shown in the figure are implemented Figures 1 to 8 The various processes and effects in the method embodiment shown are not described in detail here.
[0131] Fig.10 A structural schematic diagram of a multimodal knowledge graph construction device provided by an embodiment of the present disclosure is shown.
[0132] In some embodiments of the present disclosure, Fig.10 The multimodal knowledge graph construction device shown may be an electronic device. Specifically, the electronic device may include but is not limited to devices such as computer devices, cloud servers or cloud server clusters.
[0133] like Fig.10 As shown, the multimodal knowledge graph construction device may include a processor 1001 and a memory 1002 storing computer program instructions.
[0134] Specifically, the processor 1001 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0135] The memory 1002 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 1002 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 1002 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 1002 may be inside or outside the integrated gateway device. In a specific embodiment, the memory 1002 is a non-volatile solid-state memory. In a specific embodiment, the memory 1002 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically alterable ROM (EAROM) or flash memory, or a combination of two or more of these.
[0136] The processor 1001 reads and executes the computer program instructions stored in the memory 1002 to perform the steps of the multimodal knowledge graph construction method provided in the embodiment of the present disclosure.
[0137] In one example, the multimodal knowledge graph construction device may further include a transceiver 1003 and a bus 1004. Fig.10 As shown, the processor 1001, the memory 1002 and the transceiver 1003 are connected via a bus 1004 and communicate with each other.
[0138] The bus 1004 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 1004 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.
[0139] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the multimodal knowledge graph construction method provided by the embodiments of the present disclosure.
[0140] The above-mentioned storage medium may, for example, include a memory 1002 of computer program instructions, and the above-mentioned instructions may be executed by the processor 1001 of the multimodal knowledge graph construction device to complete the multimodal knowledge graph construction method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0141] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0142] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for constructing a multimodal knowledge graph, characterized in that: include: Acquire target multimodal data, and perform data preprocessing on the target multimodal data; Performing image information extraction processing on the target multimodal data to obtain corresponding image information, and performing text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph; Combining and associating the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph; Perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
2. The method according to claim 1, characterized in that The target multimodal data includes associated target text data and target image data.
3. The method according to claim 2, characterized in that The performing data preprocessing on the target multimodal data includes: Performing data cleaning on the target text data to obtain cleaned target text data; Performing data annotation processing on the target image data to obtain annotated target image data; The cleaned target text data and the annotated target image data are subjected to association matching processing to obtain the target multimodal data.
4. The method according to claim 1, characterized in that: The extracting and processing the target multimodal data to obtain corresponding image information includes: Model training is performed based on the optimized primary target detection algorithm and the preprocessed training data set to obtain a trained primary target detection model; The target multimodal data is input into a pre-trained primary target detection model to perform image information extraction processing to obtain corresponding image information.
5. The method according to claim 1, characterized in that The step of constructing a text knowledge graph for the target multimodal data to obtain a corresponding text knowledge graph includes: Perform model training based on the deep learning model and the preprocessed training data set to obtain a trained named entity recognition model; Inputting the target multimodal data into a trained named entity recognition model to obtain corresponding named entity recognition results; The text knowledge graph is constructed based on the named entity recognition results.
6. The method according to claim 1, characterized in that The combining and associating the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph includes: Associating entities in the text knowledge graph with results in the image information to obtain corresponding association results; The relationship types of the association results are expanded to obtain a preliminary multimodal knowledge graph.
7. The method according to claim 1, characterized in that The relationship completion process is performed on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph, including: According to the graph reasoning algorithm, the known association results in the preliminary multimodal knowledge graph are inferred to obtain the implicit multimodal relationship; The implicit multimodal relationships are completed through reasoning based on preset rules to obtain the final multimodal knowledge graph.
8. A multimodal knowledge graph construction device, characterized in that: include: A data acquisition module, used to acquire target multimodal data and perform data preprocessing on the target multimodal data; A first processing module is used to perform image information extraction processing on the target multimodal data to obtain corresponding image information, and to perform text knowledge graph construction processing on the target multimodal data to obtain a corresponding text knowledge graph; A second processing module is used to combine and associate the image information with the text knowledge graph to obtain a preliminary multimodal knowledge graph; The third processing module is used to perform relationship completion processing on the preliminary multimodal knowledge graph to obtain the final multimodal knowledge graph.
9. A multimodal knowledge graph construction device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the multimodal knowledge graph construction method described in any one of claims 1-7.
10. A non-volatile computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the multimodal knowledge graph construction method described in any one of claims 1 to 7.