Unmanned aerial vehicle knowledge graph construction method based on multi-modal large model recognition

By adopting multimodal large model recognition technology in the construction of drone knowledge graphs, integrating text, images, videos and sensor data, the problems of single data sources, weak semantic understanding capabilities and low update efficiency in the existing technology are solved, and a higher quality and smarter knowledge graph construction is achieved.

CN119623593APending Publication Date: 2025-03-14CHINA ACAD OF AEROSPACE SCI & TECH INNOVATION
View PDF 0 Cites 10 Cited by

Patent Information

Application Number
CN202411576617.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-06
Publication Date
2025-03-14

AI Technical Summary

Technical Problem

The existing drone knowledge graph construction methods have problems such as single data source, weak semantic understanding ability and low update efficiency, resulting in incomplete knowledge graph construction, low information coverage and difficulty in real-time reflecting the latest situation.

Method used

Using a method based on multimodal large model recognition, data fusion of text, images, videos and sensor data related to drones is carried out to build a multimodal data set, and a multimodal large model integrating Transformer architecture and self-attention mechanism is used for training, structured information is extracted to build a knowledge graph, data is updated regularly through online learning mechanisms, and automated verification and error correction mechanisms are used to ensure the quality of the graph.

Benefits of technology

It improves the construction quality of knowledge graphs and the intelligence level of the system, enhances the semantic understanding and scene recognition capabilities of drone data, making the knowledge graph more comprehensive, timely and practical.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119623593A_ABST
    Figure CN119623593A_ABST
Patent Text Reader

Abstract

The invention relates to an unmanned aerial vehicle knowledge graph construction method based on multi-modal large model recognition, and the method comprises the steps: carrying out the data fusion of texts, images, videos and sensor data related to an unmanned aerial vehicle, and constructing an unmanned aerial vehicle multi-modal data set; constructing a multi-modal large model integrated with a Transform architecture and a self-attention mechanism for extracting structured information from input data, and training the multi-modal large model by using a multi-modal data set of the unmanned aerial vehicle; structured information is extracted from unmanned aerial vehicle data input in real time by using the trained multi-modal large model to construct a knowledge graph, knowledge graph data is updated regularly through an online learning mechanism, and the quality and consistency of the knowledge graph are ensured by adopting an automatic verification and error correction mechanism. According to the method, the defects of low accuracy and low efficiency in unmanned aerial vehicle knowledge graph construction in the prior art are overcome, and the construction quality of the knowledge graph is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and unmanned aerial vehicle technology, and relates to a method for constructing an unmanned aerial vehicle knowledge graph based on multimodal large model recognition. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in many fields such as agriculture, logistics, monitoring, search and rescue, etc. However, the effective management and intelligent operation of drones face huge challenges, especially in autonomous decision-making and mission execution in complex environments.

[0003] As a bridge between drone operation and big data analysis, drone knowledge graph is crucial to improving the intelligence and adaptability of drone systems. However, in the existing technology, the traditional drone knowledge graph construction method has the following limitations:

[0004] Single data source: Most of the current knowledge extraction methods rely only on structured data of text type, ignoring the rich information of unstructured data (such as images and video streams), resulting in incomplete knowledge graph construction and low information coverage.

[0005] Weak semantic understanding ability: Even if there are some model methods that can identify images, they often lack deep semantic understanding and overall perception, making it difficult to automatically build the association relationship of the knowledge graph. Low efficiency: The current knowledge graph construction often requires the participation of professionals. Faced with massive amounts of new self-media data such as TikTok, the existing knowledge graph update method is slow, especially for the ever-changing drone field, it is even more difficult to reflect the latest situation in real time. Summary of the invention

[0006] The technical problem solved by the present invention is to overcome the shortcomings of the prior art in terms of low accuracy and efficiency in constructing a UAV knowledge graph, and propose a method for constructing a UAV knowledge graph based on multimodal large model recognition to improve the construction quality of the knowledge graph and enhance the intelligence level of the system.

[0007] The technical solution of the present invention is: a method for constructing a UAV knowledge graph based on multimodal large model recognition, comprising the following steps:

[0008] Perform data fusion on drone-related text, images, videos, and sensor data to build a multimodal drone dataset;

[0009] Build a large multimodal model that integrates the Transformer architecture and self-attention mechanism to extract structured information from input data, and use the multimodal dataset of drones to train the large multimodal model;

[0010] Use the trained multimodal large model to extract structured information from real-time input drone data to build a knowledge graph. Update the knowledge graph data regularly through online learning mechanisms, and use automated verification and error correction mechanisms to ensure the quality and consistency of the knowledge graph.

[0011] Further, the method for data fusion is:

[0012] Data preprocessing: Preprocess the text, images, videos, and sensor data related to drones. The text data is standardized through natural language processing methods, the image and video data are feature extracted through image processing technology, and the sensor data is filtered and normalized to remove noise;

[0013] Feature alignment: Temporally and spatially align data from different modalities to ensure the relevance of data from each modality. For video, image, and sensor data, a timestamp-based synchronization mechanism is used to achieve temporal alignment between modalities. For text data, content alignment is achieved through the description information associated with the corresponding image or video content. After feature alignment, the feature vector is obtained.

[0014] Semantic enhancement: The multimodal data is semantically enhanced through the knowledge graph, and the feature vector is supplemented with the pre-built knowledge graph so that the fused feature vector has richer semantic information.

[0015] Furthermore, the method of using the drone multimodal dataset to train the multimodal large model includes:

[0016] Data annotation: Annotate the multimodal data set of drones and generate structured annotation information to ensure the accuracy and consistency of training data;

[0017] Model initialization: Use a pre-trained model to initialize the multimodal large model. Pre-train the parameters of the multimodal large model using a large external dataset to improve the initial performance of the model.

[0018] Supervised learning: Use the labeled drone multimodal dataset to supervise the multimodal large model, and use the cross entropy loss function to compare the predicted output with the true label to adjust the model parameters;

[0019] Transfer learning: Introducing transfer learning methods during the training process to transfer existing knowledge to data of specific drone scenarios to accelerate training and improve the generalization ability of the model;

[0020] Model optimization: Use Adam or SGD optimization algorithms to optimize large multimodal models to improve model training efficiency and accuracy, and evaluate model performance through validation sets to ensure that the model does not overfit;

[0021] Model evaluation: After training is completed, the multimodal large model is evaluated using the test set to verify its ability to identify and extract knowledge from drone data, to ensure the effectiveness and accuracy of the model under different modal inputs.

[0022] Furthermore, extracting structured information from the real-time input drone data includes:

[0023] A multimodal large model is used to extract entities, relations, and attributes from drone data and structure them into "entity-relationship-entity" triplets or "entity-attribute-value" triplets to build a comprehensive and dynamic knowledge graph.

[0024] Furthermore, the method of constructing a knowledge graph using the extracted structured information is as follows:

[0025] After multimodal data fusion, the multimodal large model is used to identify and extract entities, relationships and attributes from the data, and structure them into triplets in the form of "entity-relationship-entity" or "entity-attribute-value". The triplets are then deduplicated and normalized to ensure consistency and accuracy. Next, the triplets are stored in a pre-established knowledge graph database to construct a queryable and dynamically updated graph structure. At the same time, the knowledge graph data is regularly updated through an online learning mechanism to generate new knowledge from existing triplets to support real-time intelligent decision-making of drones.

[0026] Furthermore, the periodic updating of the knowledge graph data through the online learning mechanism includes:

[0027] New drone data is collected and uploaded to the knowledge graph database, which preprocesses and extracts features from the new drone data. Then, the multimodal big model is used to identify new entities, relationships, and attributes, and the extracted triples are compared with the existing graph. If new or updated information is detected, it is integrated into the graph, and outdated or duplicate content is removed. At the same time, the multimodal big model is fine-tuned at each update to optimize subsequent data extraction and recognition capabilities.

[0028] Furthermore, the automated verification and error correction mechanism is used to ensure the quality and consistency of the knowledge graph, including:

[0029] Check the validity of newly added triples through predefined rules and logical reasoning to identify potential conflicts or outliers;

[0030] Automatically detect and correct duplicate, redundant or contradictory information using statistical methods and pre-set knowledge graph constraints;

[0031] Set up a continuous monitoring mechanism to regularly scan the data in the map to detect possible errors and trigger an automatic error correction process, including correcting deviations through historical data comparison and statistical analysis to ensure the accuracy and consistency of the map information.

[0032] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a method for constructing a knowledge graph of a drone based on multimodal large model recognition are implemented.

[0033] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a method for constructing a knowledge graph of a drone based on multimodal large model recognition when executing the computer program.

[0034] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a method for constructing a knowledge graph of a drone based on multimodal large model recognition.

[0035] The beneficial effects of the present invention compared with the prior art are:

[0036] The present invention adopts advanced data fusion technology to integrate drone-related text, images, videos, sensors and other multimodal data to construct a data set; the constructed data set is used to train a multimodal large model that integrates the Transformer architecture and self-attention mechanism, so that the multimodal large model has powerful semantic understanding and scene recognition capabilities; the trained multimodal large model is used to extract structured information from the input drone data to construct a knowledge graph, and the latest multimodal data is regularly obtained and processed through an online learning mechanism, and the new information is compared and integrated with the existing knowledge graph to maintain the timeliness and practicality of the knowledge graph; finally, an automated verification and error correction mechanism is used to ensure the quality and consistency of the graph. The present invention overcomes the shortcomings of the prior art in the construction of drone knowledge graphs, such as low accuracy and efficiency, improves the construction quality of the knowledge graph, and enhances the intelligence level of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a method for constructing a UAV knowledge graph based on multimodal large model recognition according to the present invention. DETAILED DESCRIPTION

[0038] like Figure 1 As shown, the present invention proposes a method for constructing a UAV knowledge graph based on multimodal large model recognition, comprising the following steps:

[0039] Step 1: Perform data fusion on drone-related text, images, videos, and sensor data to build a drone multimodal dataset;

[0040] Step 2: Build a large multimodal model that integrates the Transformer architecture and the self-attention mechanism to extract structured information from the input data, and use the drone multimodal dataset to train the large multimodal model;

[0041] Step 3: Use the trained multimodal large model to extract structured information from real-time input drone data to build a knowledge graph, regularly update the knowledge graph data through an online learning mechanism, and use automated verification and error correction mechanisms to ensure the quality and consistency of the knowledge graph.

[0042] In step 1, the method for data fusion is:

[0043] Data preprocessing: Preprocess the text, images, videos, and sensor data related to drones. The text data is standardized through natural language processing methods, the image and video data are feature extracted through image processing technology, and the sensor data is filtered and normalized to remove noise;

[0044] Feature alignment: Temporally and spatially align data from different modalities to ensure the relevance of data from each modality. For video, image, and sensor data, a timestamp-based synchronization mechanism is used to achieve temporal alignment between modalities. For text data, content alignment is achieved through the description information associated with the corresponding image or video content. After feature alignment, the feature vector is obtained.

[0045] Semantic enhancement: The multimodal data is semantically enhanced through the knowledge graph, and the feature vector is supplemented with the pre-built knowledge graph so that the fused feature vector has richer semantic information.

[0046] In step 2, the multimodal large model is trained using the drone multimodal dataset, including:

[0047] Data annotation: Annotate the multimodal data set of drones and generate structured annotation information to ensure the accuracy and consistency of training data;

[0048] Model initialization: Use a pre-trained model to initialize the multimodal large model. Pre-train the parameters of the multimodal large model using a large external dataset to improve the initial performance of the model.

[0049] Supervised learning: Use the labeled drone multimodal dataset to supervise the multimodal large model, and use the cross entropy loss function to compare the predicted output with the true label to adjust the model parameters;

[0050] Transfer learning: Introducing transfer learning methods during the training process to transfer existing knowledge to data of specific drone scenarios to accelerate training and improve the generalization ability of the model;

[0051] Model optimization: Use optimization algorithms (such as Adam, SGD, etc.) to optimize large multimodal models to improve model training efficiency and accuracy, and evaluate model performance through validation sets to ensure that the model does not overfit;

[0052] Model evaluation: After training is completed, the multimodal large model is evaluated using the test set to verify its ability to identify and extract knowledge from drone data, to ensure the effectiveness and accuracy of the model under different modal inputs.

[0053] In step 3, structured information is extracted from the real-time input drone data, including:

[0054] A multimodal large model is used to extract entities, relations, and attributes from drone data and structure them into "entity-relationship-entity" triplets or "entity-attribute-value" triplets to build a comprehensive and dynamic knowledge graph.

[0055] In step 3, the method of constructing a knowledge graph using the extracted structured information is:

[0056] After multimodal data fusion, the multimodal large model is used to identify and extract entities (such as buildings, vehicles), relationships (such as proximity, inclusion) and attributes (such as color, size) from the data, and structure them into triplets in the form of "entity-relationship-entity" or "entity-attribute-value". The triplets are then deduplicated and normalized to ensure consistency and accuracy. Next, the triplets are stored in a pre-established knowledge graph database to construct a queryable and dynamically updated graph structure. At the same time, the knowledge graph data is regularly updated through an online learning mechanism to generate new knowledge from existing triplets to support real-time intelligent decision-making of drones.

[0057] In step 3, the knowledge graph data is regularly updated through online learning mechanisms, including:

[0058] New drone data is collected and uploaded to the knowledge graph database, which preprocesses and extracts features from the new drone data. Then, the multimodal big model is used to identify new entities, relationships, and attributes, and the extracted triples are compared with the existing graph. If new or updated information is detected, it is integrated into the graph, and outdated or duplicate content is removed. At the same time, the multimodal big model is fine-tuned at each update to optimize subsequent data extraction and recognition capabilities.

[0059] In step 3, an automated verification and error correction mechanism is used to ensure the quality and consistency of the knowledge graph, including:

[0060] First, the validity of newly added triples is checked through predefined rules and logical reasoning to identify potential conflicts or outliers. Then, statistical methods and preset knowledge graph constraints are used to automatically detect and correct repeated, redundant or contradictory information. Finally, a continuous monitoring mechanism is set up to regularly scan the data in the graph to detect possible errors, triggering the automatic error correction process to ensure the accuracy and consistency of the graph information.

[0061] The present invention will be further described below in conjunction with the embodiments.

[0062] Example 1

[0063] A method for constructing a UAV knowledge graph based on multimodal large model recognition includes the following steps:

[0064] Step 1: Perform data fusion on drone-related text, images, videos, and sensor data to build a drone multimodal dataset.

[0065] In this embodiment, the method for performing data fusion is specifically as follows:

[0066] First, image data and video data are obtained from the pictures and videos taken by the drone, and sensor data including GPS coordinates, altitude, speed, etc. are obtained from the drone sensors. At the same time, text data such as mission descriptions, location names, etc. are collected from drone mission records, reports, or geographic information files to obtain supplementary information.

[0067] In the data preprocessing stage, the image and video data are first denoised, using algorithms such as Gaussian filtering or median filtering to reduce the interference of environmental noise on the image and video; then the resolution and frame rate are unified so that the image and video data remain consistent during subsequent analysis. In addition, the image and video data are converted into a format suitable for model input (such as images in JPEG or PNG format, H.264 encoded videos) to ensure compatibility. The text data undergoes steps such as word segmentation, stop word removal, and named entity recognition to extract the effective information from it, ensuring that the text information can be processed in coordination with other data types.

[0068] Sensor data is also standardized, for example, GPS coordinates are converted to a unified format (such as WGS84) to ensure consistency of location data, and speed and altitude data are converted to unified units (such as meters per second and meters) to facilitate comparison and analysis with other data types.

[0069] Then, the images and videos are aligned with the sensor data and text information at the corresponding moment to ensure that all types of data are fused and analyzed at the same time or in the same event context. This step is achieved through precise timestamp matching or event tagging, so that the multimodal information forms a complete time clue after fusion.

[0070] Finally, the preprocessed and aligned images, videos, sensor data, and text information are integrated into a multimodal dataset, laying the data foundation for the subsequent use of multimodal large models for feature extraction and entity recognition.

[0071] Step 2: Build a large multimodal model that integrates the Transformer architecture and self-attention mechanism to extract structured information from input data, and use the drone multimodal dataset to train the large multimodal model.

[0072] In this embodiment, the multimodal large model is trained using the drone multimodal dataset, specifically:

[0073] During the training process, a large multimodal model architecture such as the cross-modal Transformer is used. By inputting labeled data samples for supervised learning, the model learns how to extract key entities from text, how to detect the location and category of objects from images and videos, and combines sensor information to understand the relationship between entities to achieve a unified cross-modal representation. At the same time, multi-task learning goals are set to train knowledge extraction and disambiguation tasks. Specifically, the model learns to identify entities in text in the named entity recognition (NER) task and to identify image targets in the object detection task; it learns how to match objects in images with entities in text through multimodal alignment. In order to achieve entity disambiguation and reference resolution, the model learns to use contextual information to identify the same entity between different modalities through training, and solves the reference problem of synonyms and pronouns.

[0074] Step 3: Use the trained multimodal large model to extract structured information from real-time input drone data to build a knowledge graph, regularly update the knowledge graph data through an online learning mechanism, and use automated verification and error correction mechanisms to ensure the quality and consistency of the knowledge graph.

[0075] Among them, the structured information is extracted from the real-time input drone data, specifically:

[0076] A multimodal large model is used to extract entities, relations, and attributes from drone data and structure them into (entity-relationship-entity) triples or (entity-attribute-value) triples to build a comprehensive and dynamic knowledge graph.

[0077] In this embodiment, the method of constructing a knowledge graph using the extracted structured information is as follows:

[0078] The extracted information is structured into (entity-relationship-entity) or (entity-attribute-value) triples, such as "drone-located-in front of a building" or "building-height-50 meters". Normalization ensures that the triples conform to the graph naming rules and improves the integration and consistency of the graph. The normalized triples are stored in the knowledge graph database to form a query-friendly knowledge graph structure. A graph database (such as Neo4j) is used to manage graph data to support efficient data storage and indexing.

[0079] The knowledge graph data is updated regularly through the online learning mechanism. The specific method is as follows:

[0080] As drones continue to collect new data, the data is transmitted to the atlas system in real time, and the data is cleaned and standardized to ensure consistency. After processing the new data, the multimodal large model extracts new entities, relationships, and attributes, converts them into triples, and compares them with the existing data in the atlas; if new information or content that needs to be updated is found, it is integrated into the atlas, and duplicates and outdated content are automatically removed. After each update, the model is fine-tuned to enhance its adaptability and data processing accuracy to improve the accuracy of subsequent extractions.

[0081] The automated verification and error correction mechanisms described above ensure the quality and consistency of the knowledge graph, including:

[0082] Use predefined rules and logical reasoning to verify the rationality of triples, automatically identify and mark conflicting or abnormal data, and prevent invalid information from being written. Use the constraints of the knowledge graph (for example, geographic coordinate range, data type consistency, etc.) to detect and correct redundant or contradictory information, and verify data integrity through statistical methods. Set up a continuous monitoring program to regularly scan the graph data, identify potential errors and trigger automatic error correction processes, for example, correct deviations through historical data comparison and statistical analysis to ensure graph quality and consistency.

[0083] The present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes Figure 1 The method described.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0089] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A method for constructing a UAV knowledge graph based on multimodal large model recognition, characterized in that: The following steps are involved: Perform data fusion on drone-related text, images, videos, and sensor data to build a multimodal drone dataset; Build a large multimodal model that integrates the Transformer architecture and self-attention mechanism to extract structured information from input data, and use the multimodal dataset of drones to train the large multimodal model; Use the trained multimodal large model to extract structured information from real-time input drone data to build a knowledge graph. Update the knowledge graph data regularly through online learning mechanisms, and use automated verification and error correction mechanisms to ensure the quality and consistency of the knowledge graph.

2. According to claim 1, a method for constructing a UAV knowledge graph based on multimodal large model recognition is characterized in that: The method for data fusion is: Data preprocessing: Preprocess the text, images, videos, and sensor data related to drones. The text data is standardized through natural language processing methods, the image and video data are feature extracted through image processing technology, and the sensor data is filtered and normalized to remove noise; Feature alignment: Temporally and spatially align data from different modalities to ensure the relevance of data from each modality. For video, image, and sensor data, a timestamp-based synchronization mechanism is used to achieve temporal alignment between modalities. For text data, content alignment is achieved through the description information associated with the corresponding image or video content. After feature alignment, the feature vector is obtained. Semantic enhancement: The multimodal data is semantically enhanced through the knowledge graph, and the feature vector is supplemented with the pre-built knowledge graph so that the fused feature vector has richer semantic information.

3. The method for constructing a UAV knowledge graph based on multimodal large model recognition according to claim 1 is characterized in that: The method of using the drone multimodal data set to train the multimodal large model includes: Data annotation: Annotate the multimodal data set of drones and generate structured annotation information to ensure the accuracy and consistency of training data; Model initialization: Use a pre-trained model to initialize the multimodal large model. Pre-train the parameters of the multimodal large model using a large external dataset to improve the initial performance of the model. Supervised learning: Use the labeled drone multimodal dataset to supervise the multimodal large model, and use the cross entropy loss function to compare the predicted output with the true label to adjust the model parameters; Transfer learning: Introducing transfer learning methods during the training process to transfer existing knowledge to data of specific drone scenarios to accelerate training and improve the generalization ability of the model; Model optimization: Use Adam or SGD optimization algorithms to optimize large multimodal models to improve model training efficiency and accuracy, and evaluate model performance through validation sets to ensure that the model does not overfit; Model evaluation: After training is completed, the multimodal large model is evaluated using the test set to verify its ability to identify and extract knowledge from drone data, to ensure the effectiveness and accuracy of the model under different modal inputs.

4. The method for constructing a UAV knowledge graph based on multimodal large model recognition according to claim 1 is characterized in that: The extracting structured information from the real-time input drone data includes: A multimodal large model is used to extract entities, relations, and attributes from drone data and structure them into "entity-relationship-entity" triplets or "entity-attribute-value" triplets to build a comprehensive and dynamic knowledge graph.

5. The method for constructing a UAV knowledge graph based on multimodal large model recognition according to claim 4 is characterized in that: The method of constructing a knowledge graph using the extracted structured information is: After multimodal data fusion, the multimodal large model is used to identify and extract entities, relationships and attributes from the data, and structure them into triples in the form of "entity-relationship-entity" or "entity-attribute-value". The triples are then deduplicated and normalized to ensure consistency and accuracy. Then, the triples are stored in a pre-established knowledge graph database to construct a queryable and dynamically updated graph structure. At the same time, the knowledge graph data is regularly updated through an online learning mechanism to generate new knowledge from existing triples to support real-time intelligent decision-making of drones.

6. The method for constructing a UAV knowledge graph based on multimodal large model recognition according to claim 5 is characterized in that: The knowledge graph data is regularly updated through the online learning mechanism, including: New drone data is collected and uploaded to the knowledge graph database, which preprocesses and extracts features from the new drone data. Then, the multimodal big model is used to identify new entities, relationships, and attributes, and the extracted triples are compared with the existing graph. If new or updated information is detected, it is integrated into the graph, and outdated or duplicate content is removed. At the same time, the multimodal big model is fine-tuned at each update to optimize subsequent data extraction and recognition capabilities.

7. The method for constructing a UAV knowledge graph based on multimodal large model recognition according to claim 6 is characterized in that: The automated verification and error correction mechanisms described above ensure the quality and consistency of the knowledge graph, including: Check the validity of newly added triples through predefined rules and logical reasoning to identify potential conflicts or outliers; Automatically detect and correct duplicate, redundant or contradictory information using statistical methods and pre-set knowledge graph constraints; Set up a continuous monitoring mechanism to regularly scan the data in the map to detect possible errors and trigger an automatic error correction process, including correcting deviations through historical data comparison and statistical analysis to ensure the accuracy and consistency of the map information.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Marine noise online learning magnetic anomaly detection method

    CN119882077A

  • Inspection method, device and equipment based on digital airspace system and medium

    CN119989283A

  • Inspection methods, devices, equipment, and media based on digital airspace systems

    CN119989283B

  • Intelligent construction method, system and device of underground pipe network dynamic information model

    CN120145605A

  • Knowledge graph optimization method based on large model and multi-modal data fusion

    CN120338067A