Dynamic situation updating and obtaining method based on multi-modal knowledge graph

Abstract

The invention discloses a dynamic situation updating and obtaining method based on a multi-modal knowledge graph, and belongs to the technical field of natural language processing, and the method comprises the steps: collecting multi-source and multi-modal data; preprocessing the multi-source and multi-modal data to generate standardized data with a timestamp and a position label; a body is defined, cross-modal feature alignment and semantic mapping are realized, a multi-modal knowledge triple is generated, and a multi-modal knowledge graph is constructed; and in combination with a data increment threshold and a time period, realizing automatic increment updating and timing full-quantity verification of the multi-modal knowledge graph, and based on data credibility weighting and multi-source modal verification, outputting a high-credibility updating result to realize minute-level updating of the situation. According to the method, the problems of weak multi-modal data fusion capability, insufficient dynamic updating efficiency and accuracy and lack of uncertain data processing in the prior art are solved. According to the method, the utilization rate of multi-modal data can be improved, the situation updating delay is reduced, the situation credibility is improved, and the situation interpretability is enhanced.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method for acquiring dynamic situation updates based on multimodal knowledge graphs. Background Technology

[0002] Dynamic situation updates are the core support for command and decision-making. The core requirement is to integrate multi-source information in real time and accurately and generate an interpretable situation view. Current information exhibits typical multimodal characteristics, including image data collected by satellites / UAVs, signal data generated by radar / sensors, text command data transmitted by communication equipment, and voice data fed back by soldiers' terminals. These data types have large differences in format and weak semantic correlation, making them difficult to use directly for situation analysis.

[0003] Knowledge graph technology, as a semantic data integration tool, has been initially applied in the field of intelligence analysis. It achieves structured knowledge representation by defining entities (such as combat units, equipment, and terrain), relationships (such as affiliation, deployment, and attack), and attributes (such as location, status, and combat power). However, the environment is highly dynamic (equipment movement, real-time status changes), data is uncertain (signal interference, intelligence errors), and there is a need for multimodal collaboration. Traditional static knowledge graphs cannot meet the operational requirements of real-time situation updates. Therefore, it is urgent to build a situation acquisition technology system that supports multimodal data fusion and dynamic updates.

[0004] Current situational awareness and knowledge graph-related technologies suffer from the following three core shortcomings, making it difficult to meet practical application needs:

[0005] 1) Weak multimodal data fusion capability: Existing technologies mostly construct knowledge graphs for single-modal data (such as only processing text intelligence or image targets), lacking cross-modal association mechanisms. For example, the "tank cluster" identified in the image cannot be associated with the "red force armored brigade" in the text command, resulting in fragmented multi-source information and insufficient situational integrity.

[0006] 2) Insufficient efficiency and accuracy of dynamic updates: Traditional knowledge graph updates rely on manual annotation or batch data retraining, with update cycles lasting for hours or even days, which cannot meet the needs of "minute-level" situational changes. At the same time, data conflicts (such as positional deviations of the same target reported by different sensors) are not considered during the update process, which can easily lead to misjudgment of the situation.

[0007] 3) Lack of processing for uncertain data: Data is often accompanied by noise (such as radar clutter), missing data (such as data gaps in some areas due to communication interruptions) or ambiguity (such as blurry images that cannot determine the equipment model). Existing technologies often directly filter such data, resulting in incomplete situation information; or simply fill in default values, reducing the credibility of the situation. Summary of the Invention

[0008] The purpose of this invention is to provide a dynamic situation update acquisition method based on multimodal knowledge graphs, which can improve the utilization rate of multimodal data, reduce the situation update delay, improve the credibility of the situation and enhance the interpretability of the situation, thus solving the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] Dynamic situation update acquisition methods based on multimodal knowledge graphs include:

[0011] Based on standardized interfaces, image, signal, text and voice data are collected synchronously to identify multi-source and multimodal data. Timestamps and location tags are embedded during the collection process to trace each data segment to a specific spatiotemporal dimension.

[0012] Based on the characteristics of different modal data, multi-source multimodal data is preprocessed, and standardized data with timestamps and location labels are generated through target feature extraction, signal noise reduction, entity recognition and speech-to-text operations.

[0013] Define an ontology that includes combat entities, relationships, and attributes; achieve cross-modal feature alignment through an attention mechanism; establish semantic mapping based on a multimodal dictionary; generate multimodal knowledge triples with credibility labels; and construct a multimodal knowledge graph.

[0014] By combining data increment thresholds and time periods, automatic incremental updates and timed full verification of multimodal knowledge graphs are achieved. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation.

[0015] Preferably, based on a standardized interface, image, signal, text, and voice data are acquired synchronously to determine multi-source, multimodal data, and the following operations are performed:

[0016] Based on image data collected by satellites, drones and ground cameras, it covers visual information on terrain, equipment appearance and personnel deployment;

[0017] Signal data collected based on radar and electronic reconnaissance equipment, including signal information such as target speed, azimuth, and electromagnetic characteristics;

[0018] Text data collected based on command systems and intelligence terminals, including structured or unstructured text of combat instructions, intelligence reports, and equipment parameters;

[0019] Voice data collected from soldiers' walkie-talkies and command radios, including voice information for real-time tactical communication and emergency response;

[0020] Multi-source, multi-modal data is generated based on the collected image data, signal data, text data, and voice data.

[0021] Preferably, preprocessing of multi-source multimodal data based on the characteristics of different modal data involves the following operations:

[0022] Image data is preprocessed based on convolutional neural networks to extract target features and generate feature vectors. Image data is also preprocessed based on image enhancement algorithms to reduce the impact of smoke and dust on image quality.

[0023] Preprocessing of signal data based on wavelet transform filters radar clutter, extracts target motion features including velocity and acceleration, and performs spectral analysis on electromagnetic signals to generate signal fingerprints;

[0024] Based on the BERT model, text data is preprocessed, and deep semantic representation of text is achieved through bidirectional contextual understanding. Word segmentation and entity recognition are performed, and unstructured text is transformed into structured triples.

[0025] The speech data is preprocessed using automatic speech recognition technology, converting the speech into text, and then the text preprocessing process is performed to preserve the emotional features of the speech, which helps to determine the degree of urgency.

[0026] Preferably, an ontology is defined that includes combat entities, relationships, and attributes. Cross-modal feature alignment is achieved through an attention mechanism. Semantic mapping is established based on a multimodal dictionary. Multimodal knowledge triples with credibility labels are generated to construct a multimodal knowledge graph and perform the following operations:

[0027] Define the ontology structure and design a unique ontology, which includes three core elements: entities, relationships, and attributes. Entities include combat units, equipment, terrain, and personnel; relationships include affiliation, deployment, and attack; and attributes include entity attributes, time attributes, and credibility attributes.

[0028] A two-stage fusion scheme based on feature alignment and semantic mapping is used for multimodal data fusion. In this scheme, image feature vectors, signal fingerprints and text entity features are mapped to the same semantic space through an attention mechanism. Furthermore, based on a multimodal dictionary generated by a pre-trained large model, cross-modal entity associations are established to generate multimodal knowledge triples.

[0029] A graph database is used to store the multimodal knowledge graph, which supports fast querying and modification of entities, relations and attributes. A timestamp and credibility tag are attached to each triple for subsequent updates and situation assessment.

[0030] Preferably, a confidence label is attached to each triple, including:

[0031] Define a baseline score for the credibility of data sources and assign weights to data sources according to the level of military intelligence.

[0032] Among them, the primary source of satellite or early warning aircraft data is 90-100, the secondary source of radar or UAV data is 70-89, the tertiary source of human reconnaissance or open source intelligence is 40-69, and the quaternary source of unverified communication interception is 0-39.

[0033] If a triplet is supported by n data sources, the weighted average score is taken. If there is a conflict source, a conflict penalty score is deducted.

[0034] Preferably, by combining data increment thresholds and time periods, automatic incremental updates and timed full verification of the multimodal knowledge graph are achieved. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation. The following operations are performed:

[0035] Configure an incremental triggering mechanism, including two triggering conditions: incremental data triggering and time-based triggering;

[0036] Specifically, when data increment is triggered, an update is automatically triggered during system idle time when the amount of newly collected multimodal data reaches a threshold; when time-based trigger is triggered, a full-scale situational verification is triggered at a fixed period to ensure that the situational verification remains accurate even when there is no incremental data.

[0037] Set up conflict detection and solutions, and adopt a confidence weighting and multi-source verification strategy to address conflicts between new data and existing map information;

[0038] Among them, credibility weighting calculates the credibility of conflicting data and prioritizes the retention of high-credibility data; multi-source verification automatically associates other modal data for verification if the credibility of a single data is low, and marks it as pending confirmation and feeds it back to the command end when it cannot be verified.

[0039] Set up an update execution plan and perform three types of operations on the map based on the conflict resolution results: adding, modifying, and deleting.

[0040] Among them, "Add" means adding new entities or relationships; "Modify" means updating entity attributes or relationship status; and "Delete" means removing invalid information.

[0041] Preferably, when data increment is triggered, if the amount of newly collected multimodal data reaches a threshold, an update is automatically triggered during system idle time, and the following operations are performed:

[0042] All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping.

[0043] When data increment is triggered, the amount of unprocessed data of each information source or key entity type in the monitoring message queue is monitored. When the accumulated data amount reaches a preset threshold and the system load is lower than the idle time threshold, the incremental update process is immediately triggered. The system idle time is determined by obtaining the system CPU, memory and I / O load in real time through the resource monitor to ensure that the update task does not affect online high-priority transactions. Cross-modal association is performed on the newly added data to identify new entities or new attributes or relationships of existing entities. Real-time conflict detection is performed and compared with the existing data in the graph to find contradictions in position and status. The fused result is updated to the graph in the form of incremental transactions. After the graph is updated, the graph update event is published through the event bus.

[0044] Preferably, when the timer is triggered, a full situational awareness check is triggered at a fixed period to ensure the situational awareness remains accurate even without incremental data, and the following operations are performed:

[0045] All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping.

[0046] When triggered by a timer, a timer independent of the data stream triggers once per periodic interval. After triggering, the system load is checked. If the system is busy, a short delay is made or the granularity of this verification is reduced, but it must be executed. The entire knowledge graph and snapshots of all external data sources are checked. Hard constraints are checked based on rules, and logical contradictions are detected based on reasoning and the integrity of the graph. For data that has not been updated for a long time, its credibility is decayed according to a preset model. The credibility of key entities is recalculated by combining multi-source information. For entities that should have dynamics but no new data, their status is inferred by combining their last known speed, path, and task, and they are marked with low credibility. Active probe requests may be triggered. After updating the graph, the graph update event is published through the event bus.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention improves the utilization rate of multimodal data. Through a cross-modal fusion mechanism, it links fragmented image, signal, text, and voice data into a unified knowledge graph, significantly improving data utilization, enhancing situational integrity, and reducing situational update latency. By employing incremental triggering and lightweight update algorithms, the situational update cycle is shortened from several hours in existing technologies to less than 5 minutes, meeting real-time decision-making needs, providing command with tactical response time, and improving situational credibility. Through uncertainty handling and conflict resolution mechanisms, it effectively filters noisy data and corrects contradictory information, improving the accuracy of situational assessment, reducing decision-making errors caused by data errors, and enhancing situational interpretability. The multimodal knowledge graph, through the structured representation of entities, relationships, and attributes, can be intuitively displayed, allowing commanders to understand the situation more efficiently compared to traditional unstructured data. Attached Figure Description

[0049] Figure 1 This is a flowchart of the dynamic situation update acquisition method based on multimodal knowledge graph of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] To address the existing problems of weak multimodal data fusion capabilities, insufficient efficiency and accuracy in dynamic updates, and lack of handling for uncertain data, please refer to [link / reference needed]. Figure 1 This embodiment provides the following technical solution:

[0052] Dynamic situation update acquisition methods based on multimodal knowledge graphs include:

[0053] Based on standardized interfaces, image, signal, text and voice data are collected synchronously to identify multi-source and multimodal data. Timestamps and location tags are embedded during the collection process, so that each data segment can be traced back to a specific spatiotemporal dimension.

[0054] In this embodiment, image, signal, text, and voice data are simultaneously acquired based on a standardized interface to determine multi-source, multimodal data, and the following operations are performed:

[0055] Based on image data collected by satellites, drones and ground cameras, it covers visual information on terrain, equipment appearance and personnel deployment;

[0056] Signal data collected based on radar and electronic reconnaissance equipment, including signal information such as target speed, azimuth, and electromagnetic characteristics;

[0057] Text data collected based on command systems and intelligence terminals, including structured or unstructured text of combat instructions, intelligence reports, and equipment parameters;

[0058] Voice data collected from soldiers' walkie-talkies and command radios, including voice information for real-time tactical communication and emergency response;

[0059] Multi-source, multi-modal data is generated based on the collected image data, signal data, text data, and voice data.

[0060] Based on the characteristics of different modal data, multi-source multimodal data is preprocessed, and standardized data with timestamps and location labels are generated through target feature extraction, signal noise reduction, entity recognition and speech-to-text operations.

[0061] In this embodiment, multi-source multimodal data is preprocessed based on the characteristics of different modal data by performing the following operations:

[0062] Image data is preprocessed using convolutional neural networks (such as YOLOv8) to extract target features (such as tank outlines and fighter jet exhaust flames) and generate feature vectors. Image data is also preprocessed using image enhancement algorithms (such as histogram equalization) to reduce the impact of smoke and dust on image quality.

[0063] Among them, a pre-trained video image big model is used to preprocess the image data. By leveraging the model's powerful cross-modal understanding and general visual feature extraction capabilities, the original image data is transformed into a structured representation rich in semantic information, thereby optimizing subsequent tasks. This video image big model has undergone joint pre-training with massive image-text pairs. It can not only understand the low-level pixel features of images, but also understand the high-level semantic information contained therein, and encode it into an intermediate representation that is convenient for subsequent dynamic situation update processing.

[0064] Specifically, raw image data is input into a pre-trained video image large model for basic visual encoding. The large model's visual encoder is used to transform the images into high-dimensional feature vectors. The large model is then used to automatically generate high-quality text descriptions or question-and-answer pairs for unlabeled images, constructing an instruction fine-tuning dataset. The large model is also used to evaluate image quality, content relevance, or information content, thereby selecting the most valuable data subset for subsequent annotation or training, significantly improving data efficiency.

[0065] Preprocessing of signal data based on wavelet transform filters radar clutter, extracts target motion features including velocity and acceleration, and performs spectral analysis on electromagnetic signals to generate signal fingerprints;

[0066] Among them, a pre-trained multimodal large model is used to preprocess the signal data. By leveraging the powerful representation and cross-modal alignment capabilities learned by the large model in joint training with multiple modal data, the low-level, unstructured raw signal is transformed into a high-level, semantically rich feature representation that is aligned with other modalities, thereby greatly improving the performance of dynamic situation update.

[0067] Specifically, raw signal data is input into a pre-trained multimodal large model to perform signal type and target analysis, transform it into a visual modality, generate time-frequency graphs, spectrograms or feature images, use a visual encoder to extract image features, obtain vector representations with visual semantics, and can also be transformed into text descriptions. A domain model or rules are used to generate preliminary text descriptions, and a large language model is used for polishing and structuring to obtain high-quality, semantically rich text descriptions. Direct encoding and alignment are also possible, and the decoder is fine-tuned to map the signal to a shared space to obtain vector representations directly aligned with the text or image.

[0068] Based on the BERT model, text data is preprocessed, and deep semantic representation of text is achieved through bidirectional contextual understanding. Word segmentation and entity recognition are performed, such as extracting key information such as Red Team, Armored Brigade and 10:30. Unstructured text is transformed into structured triples.

[0069] Among them, using a pre-trained text-based large model to preprocess text data and transform the original text into a structured representation rich in semantic information that can serve downstream tasks more efficiently and accurately can greatly improve the performance of natural language processing tasks.

[0070] Specifically, raw text data is input into a pre-trained large text model, basic word segmentation and encoding are performed to obtain general semantic vectors and dynamic context-related high-dimensional vector representations, instruction fine-tuning data is synthesized to obtain high-quality instruction-response pairing data, and data cleaning and quality assessment are performed to obtain a text subset with lower noise and higher quality, and complex text structure parsing is performed to obtain structured data containing logical relationships.

[0071] The speech data is preprocessed using automatic speech recognition technology, converting the speech into text, and then the text preprocessing process is performed to retain the emotional features of the speech (such as tension and urgency) to help determine the degree of urgency.

[0072] Among them, a pre-trained large speech model is used to preprocess the speech data. By utilizing the general acoustic and linguistic knowledge learned by the model from massive data, the original speech data is transformed into a high-level, compact and semantically rich representation that is more conducive to dynamic situational update processing, and the acoustic, speaker and linguistic semantic information in the speech content is understood and extracted.

[0073] Specifically, raw speech data is input into a pre-trained speech model, basic waveform / spectral coding is performed, and frame-level or sentence-level general speech representation vectors are obtained based on general speech feature extraction. Based on speech recognition and textification, high-accuracy transcribed text and timestamps are obtained. Based on speech enhancement and separation, clear and pure single-speaker speech information is obtained. Based on the structured parsing of speech content, structured information including speaker, emotion, and events is obtained.

[0074] Define an ontology that includes combat entities, relationships, and attributes. Achieve cross-modal feature alignment through an attention mechanism. Establish semantic mapping based on a multimodal dictionary. Generate multimodal knowledge triples with credibility labels and construct a multimodal knowledge graph.

[0075] In this embodiment, an ontology containing combat entities, relationships, and attributes is defined. Cross-modal feature alignment is achieved through an attention mechanism. Semantic mapping is established based on a multimodal dictionary. Multimodal knowledge triples with credibility labels are generated, and a multimodal knowledge graph is constructed. The following operations are performed:

[0076] Define the ontology structure and design a unique ontology, which includes three core elements: entities, relationships, and attributes. Among them, entities include combat units (such as brigades and battalions), equipment (such as tanks and fighter jets), terrain (such as mountains and bridges), and personnel (such as commanders and soldiers).

[0077] Relationships include affiliation (Red Force Armored Brigade - Affiliation - Red Force Army Group), deployment (Tank Company - Deployment - Hill A), and attack (Blue Force Fighter Jet - Attack - Red Force Radar Station).

[0078] The attributes include entity attributes (tank - speed - 60km / h), time attributes (deployment - time - 10 hours 30 minutes), and credibility attributes (intelligence - credibility - 90%).

[0079] A two-stage fusion scheme based on feature alignment and semantic mapping is used for multimodal data fusion. In this scheme, an attention mechanism is used to map image feature vectors, signal fingerprints and text entity features to the same semantic space. For example, tank image features are aligned with text, tank and entity. Based on a multimodal dictionary generated by a pre-trained large model, for example, the features of a fighter jet exhaust image are converted into the text "fighter jet", and then converted into the signal "fighter jet electromagnetic features". Cross-modal entity associations are established to generate multimodal knowledge triples, such as tank image, corresponding entity, text "red team tank company".

[0080] A graph database is used to store the multimodal knowledge graph, which supports fast querying and modification of entities, relations and attributes. A timestamp and credibility tag are attached to each triple for subsequent updates and situation assessment.

[0081] In this embodiment, a confidence label is attached to each triple, including:

[0082] Define a baseline score for the credibility of data sources and assign weights to data sources according to the level of military intelligence.

[0083] Among them, the primary source of satellite or early warning aircraft data is 90-100, the secondary source of radar or UAV data is 70-89, the tertiary source of human reconnaissance or open source intelligence is 40-69, and the quaternary source of unverified communication interception is 0-39.

[0084] If a triplet is supported by n data sources, the weighted average score is taken. If there is a conflict source, a conflict penalty score is deducted.

[0085] By combining data increment thresholds and time periods, automatic incremental updates and timed full verification of multimodal knowledge graphs are achieved. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation.

[0086] In this embodiment, automatic incremental updates and timed full verification of the multimodal knowledge graph are achieved by combining data increment thresholds and time periods. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation. The following operations are performed:

[0087] Configure an incremental triggering mechanism, including two triggering conditions: incremental data triggering and time-based triggering;

[0088] Specifically, when data increment is triggered, an update is automatically triggered during system idle time (such as at night) when the amount of newly collected multimodal data reaches a threshold (such as 10 new target data); when time-based trigger is triggered, a full situational verification is triggered at a fixed period (such as 5 minutes) to ensure that the situational verification is still accurate when there is no incremental data.

[0089] Set up conflict detection and solutions to address conflicts between new data and existing map information. For example, if the existing map shows the tank company is on high ground A, while the new radar data shows the tank company is on high ground B, a confidence weighting and multi-source verification strategy will be adopted.

[0090] Among them, credibility weighting is used to calculate the credibility of conflict data. For example, radar data has a credibility of 80% and image data has a credibility of 95%, so high credibility data is retained first. Multi-source verification is used to automatically associate other modal data for verification if the credibility of a single data is low, such as <50%. For example, if image data is used to confirm the location of a tank company, it is marked as pending confirmation and fed back to the command end if it cannot be verified.

[0091] Set up an update execution plan and perform three types of operations on the map based on the conflict resolution results: adding, modifying, and deleting.

[0092] Among them, the additions are new entities (such as discovering unknown equipment) and new relationships (such as new equipment belonging to the blue team).

[0093] Modify to update entity attributes (e.g., change the tank company's location from Highland A to Highland B) and relationship status (e.g., change the attack relationship from "never occurred" to "has occurred").

[0094] Delete removes invalid information (such as entity associations of destroyed equipment).

[0095] In this embodiment, when the data increment is triggered, an update is automatically triggered during system idle time when the amount of newly collected multimodal data reaches a threshold, and the following operations are performed:

[0096] All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping.

[0097] When data increment is triggered, the amount of unprocessed data of each information source or key entity type in the monitoring message queue is monitored. When the accumulated data amount reaches a preset threshold and the system load is lower than the idle time threshold, the incremental update process is immediately triggered. The system idle time is determined by obtaining the system CPU, memory and I / O load in real time through the resource monitor to ensure that the update task does not affect online high-priority transactions. Cross-modal association is performed on the newly added data to identify new entities or new attributes or relationships of existing entities. Real-time conflict detection is performed and compared with the existing data in the graph to find contradictions in position and status. The fused result is updated to the graph in the form of incremental transactions. After the graph is updated, the graph update event is published through the event bus.

[0098] In this embodiment, when the timer is triggered, a full situational awareness check is performed at fixed intervals to ensure the situational awareness remains accurate even without incremental data. The following operations are performed:

[0099] All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping.

[0100] When triggered by a timer, a timer independent of the data stream triggers once per periodic interval. After triggering, the system load is checked. If the system is busy, a short delay is made or the granularity of this verification is reduced, but it must be executed. The entire knowledge graph and snapshots of all external data sources are checked. Hard constraints are checked based on rules, and logical contradictions are detected based on reasoning and the integrity of the graph. For data that has not been updated for a long time, its credibility is decayed according to a preset model. The credibility of key entities is recalculated by combining multi-source information. For entities that should have dynamics but no new data, their status is inferred by combining their last known speed, path, and task, and they are marked with low credibility. Active probe requests may be triggered. After updating the graph, the graph update event is published through the event bus.

[0101] Specifically, addressing the issue of missing cross-modal semantic associations in multimodal data, existing technologies only construct knowledge graphs for single-modal data (such as text intelligence or image targets), lacking a semantic mapping mechanism between image, signal, text, and voice data. This results in the inability to associate the tank cluster identified by image recognition with the red force armored brigade in text commands and the armored target signals detected by radar, leading to fragmented multi-source information and an inability to form a complete situational view. In this embodiment, a cross-modal semantic association mechanism is established to achieve deep fusion of multimodal data such as images, signals, text, and voice, integrating fragmented information into a structured knowledge graph and improving the integrity of the situational view.

[0102] Specifically, addressing the issues of delayed dynamic situation updates and ineffective conflict handling, traditional knowledge graphs rely on manual annotation or batch retraining for updates, with update cycles lasting from hours to days, which cannot meet the needs of minute-level situation changes. Furthermore, when newly acquired data conflicts with existing graph information (such as deviations in the location of the same target reported by different sensors), there is a lack of effective quantitative assessment and resolution mechanisms, easily leading to misjudgments of the situation. In this embodiment, an efficient dynamic update triggering and conflict handling scheme is designed to compress the situation update cycle to the minute level. Simultaneously, data conflicts are resolved through reliable quantification and multi-source verification, ensuring the timeliness and accuracy of situation updates.

[0103] Specifically, addressing the problem of insufficient utilization of uncertain data, which is often accompanied by noise (such as radar clutter and blurred images), missing data (such as data gaps in local areas caused by communication interruptions) or ambiguity (such as visual information that cannot determine the equipment model), existing technologies often directly filter such data or fill in default values, resulting in incomplete situational information or reduced situational credibility due to data errors. In this embodiment, a dual-label knowledge graph model containing timestamps and credibility is constructed to achieve effective utilization of uncertain data (such as marking low-credibility data and triggering verification processes), thereby improving the completeness and credibility of situational information.

[0104] Specifically, addressing the issue of insufficient situational interpretability, existing technologies often present the situation using unstructured data (such as raw images and text reports) or single-dimensional indicators. Commanders need to manually integrate information from multiple sources to understand combat units, locations, states, and interactions, resulting in low situational interpretation efficiency and difficulty in quickly supporting decision-making. In contrast, this embodiment presents the situation in a structured form of entities, relationships, and attributes, intuitively displaying the spatiotemporal location, interactions, and state changes of combat units, thereby improving situational interpretability and assisting commanders in quickly understanding the situation and making tactical decisions.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0106] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for acquiring dynamic situation update based on a multi-modal knowledge graph, characterized in that, include: Based on standardized interfaces, image, signal, text and voice data are collected synchronously to identify multi-source and multimodal data. Timestamps and location tags are embedded during the collection process to trace each data segment to a specific spatiotemporal dimension. Based on the characteristics of different modal data, multi-source multimodal data is preprocessed, and standardized data with timestamps and location labels are generated through target feature extraction, signal noise reduction, entity recognition and speech-to-text operations. Define an ontology that includes combat entities, relationships, and attributes; achieve cross-modal feature alignment through an attention mechanism; establish semantic mapping based on a multimodal dictionary; generate multimodal knowledge triples with credibility labels; and construct a multimodal knowledge graph. By combining data increment thresholds and time periods, automatic incremental updates and timed full verification of multimodal knowledge graphs are achieved. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation. 2.The method of claim 1, wherein, Based on a standardized interface, image, signal, text, and voice data are acquired synchronously to determine multi-source, multimodal data, and the following operations are performed: Based on image data collected by satellites, drones and ground cameras, it covers visual information on terrain, equipment appearance and personnel deployment; Signal data collected based on radar and electronic reconnaissance equipment, including signal information such as target speed, azimuth, and electromagnetic characteristics; Text data collected based on command systems and intelligence terminals, including structured or unstructured text of combat instructions, intelligence reports, and equipment parameters; Voice data collected from soldiers' walkie-talkies and command radios, including voice information for real-time tactical communication and emergency response; Multi-source, multi-modal data is generated based on the collected image data, signal data, text data, and voice data. 3.The method of claim 2, wherein, Based on the characteristics of different modal data, multi-source multimodal data are preprocessed by performing the following operations: Image data is preprocessed based on convolutional neural networks to extract target features and generate feature vectors. Image data is also preprocessed based on image enhancement algorithms to reduce the impact of smoke and dust on image quality. Preprocessing of signal data based on wavelet transform filters radar clutter, extracts target motion features including velocity and acceleration, and performs spectral analysis on electromagnetic signals to generate signal fingerprints; Based on the BERT model, text data is preprocessed, and deep semantic representation of text is achieved through bidirectional contextual understanding. Word segmentation and entity recognition are performed, and unstructured text is transformed into structured triples. The speech data is preprocessed using automatic speech recognition technology, converting the speech into text, and then the text preprocessing process is performed to preserve the emotional features of the speech, which helps to determine the degree of urgency.

4. The dynamic situation update acquisition method based on multimodal knowledge graph according to claim 3, characterized in that, Define an ontology containing combat entities, relationships, and attributes; achieve cross-modal feature alignment through an attention mechanism; establish semantic mapping based on a multimodal dictionary; generate multimodal knowledge triples with credibility labels; construct a multimodal knowledge graph; and perform the following operations: Define the ontology structure and design a unique ontology, which includes three core elements: entities, relationships, and attributes. Entities include combat units, equipment, terrain, and personnel; relationships include affiliation, deployment, and attack; and attributes include entity attributes, time attributes, and credibility attributes. A two-stage fusion scheme based on feature alignment and semantic mapping is used for multimodal data fusion. In this scheme, image feature vectors, signal fingerprints and text entity features are mapped to the same semantic space through an attention mechanism. Furthermore, based on a multimodal dictionary generated by a pre-trained large model, cross-modal entity associations are established to generate multimodal knowledge triples. A graph database is used to store the multimodal knowledge graph, which supports fast querying and modification of entities, relations and attributes. A timestamp and credibility tag are attached to each triple for subsequent updates and situation assessment.

5. The dynamic situation update acquisition method based on multimodal knowledge graph according to claim 4, characterized in that, Attach a confidence label to each triple, including: Define a baseline score for the credibility of data sources and assign weights to data sources according to the level of military intelligence. Among them, the primary source of satellite or early warning aircraft data is 90-100, the secondary source of radar or UAV data is 70-89, the tertiary source of human reconnaissance or open source intelligence is 40-69, and the quaternary source of unverified communication interception is 0-39. If a triplet is supported by n data sources, the weighted average score is taken. If there is a conflict source, a conflict penalty score is deducted.

6. The dynamic situation update acquisition method based on multimodal knowledge graph according to claim 5, characterized in that, By combining data increment thresholds and time periods, automatic incremental updates and timed full verification of multimodal knowledge graphs are achieved. Based on data credibility weighting and multi-source modality verification, high-credibility update results are output, and low-credibility data is marked as pending confirmation and feedback is provided, achieving minute-level updates of the situation. The following operations are performed: Configure an incremental triggering mechanism, including two triggering conditions: incremental data triggering and time-based triggering; Specifically, when data increment is triggered, an update is automatically triggered during system idle time when the amount of newly collected multimodal data reaches a threshold; when time-based trigger is triggered, a full-scale situational verification is triggered at a fixed period to ensure that the situational verification remains accurate even when there is no incremental data. Set up conflict detection and solutions, and adopt a confidence weighting and multi-source verification strategy to address conflicts between new data and existing map information; Among them, credibility weighting calculates the credibility of conflicting data and prioritizes the retention of high-credibility data; multi-source verification automatically associates other modal data for verification if the credibility of a single data is low, and marks it as pending confirmation and feeds it back to the command end when it cannot be verified. Set up an update execution plan and perform three types of operations on the map based on the conflict resolution results: adding, modifying, and deleting. Among them, "Add" means adding new entities or relationships; "Modify" means updating entity attributes or relationship status; and "Delete" means removing invalid information.

7. The dynamic situation update acquisition method based on multimodal knowledge graph according to claim 6, characterized in that, When data increment is triggered, if the amount of newly collected multimodal data reaches a threshold, an update will be automatically triggered during system idle time, and the following operations will be performed: All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping. When data increment is triggered, the amount of unprocessed data of each information source or key entity type in the monitoring message queue is monitored. When the accumulated data amount reaches a preset threshold and the system load is lower than the idle time threshold, the incremental update process is immediately triggered. The system idle time is determined by obtaining the system CPU, memory and I / O load in real time through the resource monitor to ensure that the update task does not affect online high-priority transactions. Cross-modal association is performed on the newly added data to identify new entities or new attributes or relationships of existing entities. Real-time conflict detection is performed and compared with the existing data in the graph to find contradictions in position and status. The fused result is updated to the graph in the form of incremental transactions. After the graph is updated, the graph update event is published through the event bus.

8. The dynamic situation update acquisition method based on multimodal knowledge graph according to claim 7, characterized in that, When the timer is triggered, a full situational awareness check is performed at a fixed period to ensure the situational awareness remains accurate even without incremental data, and the following operations are performed: All incoming image, signal, text, and voice data are written to a high-performance message queue after initial timestamps and source marking. The message queue is used for decoupling, buffering, and traffic shaping. When triggered by a timer, a timer independent of the data stream triggers once per period. After triggering, the system load is checked. If the system is busy, a short delay is made or the granularity of this verification is reduced, but it must be executed. The entire knowledge graph and snapshots of all external data sources are checked. Hard constraints are checked based on rules, and logical contradictions are detected based on reasoning and the integrity of the graph. For data that has not been updated for a long time, the credibility is decayed according to a preset model, and the credibility of key entities is recalculated by combining multi-source information. For entities that should have dynamics but no new data, the state is inferred by combining their last known speed, path, and task, and they are marked with low credibility, which may trigger an active probe request. After updating the graph, the graph update event is published through the event bus.

Images