Target monitoring method and device, equipment, storage medium and program product

Abstract

The invention discloses a target monitoring method, device and equipment, a storage medium and a program product, and the method comprises the steps: collecting multi-modal data in real time through multi-sensor equipment deployed in a target region in advance; performing feature extraction and feature fusion processing on the acquired multi-modal data to obtain data features; constructing a knowledge graph related to the target monitoring field; extracting semantic features of the knowledge graph; fusing the data features and the semantic features to generate a feature vector combining quantitative data and semantic knowledge, and recording the feature vector as a target feature vector; and performing target monitoring operation according to the target feature vector and the knowledge graph. By adopting the method and the device, deep fusion can be carried out on the data features acquired by multiple sensors and the semantic features acquired based on the knowledge graph, a complete target monitoring process is cooperatively completed, and the precision and the intelligence of target monitoring are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a target monitoring method, apparatus, device, storage medium, and program product. Background Technology

[0002] Target monitoring technology has application needs in multiple fields, such as low-altitude flight, ground security, industrial operation and maintenance, and urban governance.

[0003] Currently, target monitoring systems commonly employ multi-sensor fusion and static knowledge graph technologies for target detection. Data is collected from various devices such as radar, optical cameras, and infrared sensors, and multimodal information is integrated using traditional weighted averaging or decision-level fusion (such as voting methods) to achieve target identification. Behavior prediction is then performed using predefined rules from a pre-built static knowledge graph. However, because the multi-sensor data fusion and knowledge graph query processes are independent, behavior prediction biases can easily occur, hindering the generation of accurate and effective processing decisions. Summary of the Invention

[0004] The purpose of this invention is to provide a target monitoring method, device, equipment, storage medium, and program product that can deeply integrate data features collected by multiple sensors and semantic features collected based on knowledge graphs to collaboratively complete the entire target monitoring process, thereby improving the accuracy and intelligence of target monitoring.

[0005] To achieve the above objectives, embodiments of the present invention provide a target monitoring method, comprising: Multimodal data is collected in real time using multi-sensor devices pre-deployed in the target area; Feature extraction and feature fusion are performed on the collected multimodal data to obtain data features; Construct a knowledge graph related to the field of target monitoring; Extract the semantic features of the knowledge graph; The data features and semantic features are fused to generate a feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector; Target monitoring is performed based on the target feature vector and the knowledge graph.

[0006] As an improvement to the above scheme, the step of performing feature extraction and feature fusion processing on the collected multimodal data to obtain data features includes: Feature extraction is performed on each modality of the multimodal data; The confidence level of each modal data is assessed based on the modal quality factor; Calculate the weighting coefficient for each modality based on the confidence level; The features of each modality data are weighted and fused according to the weight coefficients to obtain the fused data features.

[0007] As an improvement to the above solution, the multi-sensor device includes at least one of an optical camera, radar, acoustic sensor, and infrared sensor; the multimodal data includes at least one of image data, radar data, acoustic data, and infrared data.

[0008] As an improvement to the above scheme, the construction of a knowledge graph related to the target monitoring domain includes: A knowledge graph related to the target monitoring domain is constructed based on a pre-defined graph database; wherein, the knowledge graph includes several nodes related to the target monitoring domain and edges used to represent the relationships between nodes; Based on the multimodal data, the corresponding nodes and edges in the knowledge graph are updated in real time to obtain a dynamic knowledge graph.

[0009] As an improvement to the above scheme, the step of updating the corresponding nodes and edges in the knowledge graph in real time based on the multimodal data to obtain a dynamic knowledge graph includes: When the preset update triggering condition is met based on the multimodal data, the corresponding nodes and edges in the knowledge graph are updated according to the message passing mechanism of the graph neural network. Generate a natural language description based on the preset prompt word template and the multimodal data; The natural language description is semantically inferred using a large language model to generate new triples and corresponding confidence scores; wherein the triples are in entity-relation-entity format. New triples with a confidence level greater than a preset confidence threshold are added to the knowledge graph to update the knowledge graph.

[0010] As an improvement to the above scheme, the step of fusing the data features and the semantic features to generate a feature vector combining quantized data and semantic knowledge, denoted as the target feature vector, includes: The Concatenate algorithm is used to concatenate the data features and the semantic features along the same dimension to generate a high-dimensional feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector.

[0011] As an improvement to the above solution, the target monitoring operation includes target identification, behavior analysis, risk assessment, and strategy generation; The target detection operation based on the target feature vector and the knowledge graph includes: The target feature vector and the knowledge graph are input into a preset target recognition model to obtain the target category probability output by the target recognition model, and the recognized target is determined based on the target category probability. The target feature vector and the knowledge graph are input into a preset behavior analysis model to obtain a behavior label sequence output by the behavior analysis model. The behavioral label sequence and the knowledge graph are input into a preset risk assessment model to obtain the risk value output by the risk assessment model. The risk value and the knowledge graph are input into a preset decision generation model to obtain the action vector output by the decision generation model, which serves as the disposal strategy.

[0012] As an improvement to the above scheme, the target recognition model adopts a network structure of fully connected layer FC and Softmax classifier; the behavior analysis model adopts a network structure of long short-term memory neural network LSTM and attention mechanism; the risk assessment model adopts a network structure of fully connected layer FC and sigmoid function; and the decision generation model adopts a policy network structure.

[0013] This invention also provides a target monitoring device, comprising: The multimodal data acquisition module is used to acquire multimodal data in real time using multi-sensor devices pre-deployed in the target area; The data feature extraction module is used to extract and fuse features from the collected multimodal data to obtain data features. The knowledge graph construction module is used to build dynamic knowledge graphs related to the target monitoring field. A semantic feature extraction module is used to extract the semantic features of the dynamic knowledge graph; The feature vector fusion module is used to fuse the data features and the semantic features to generate a feature vector that combines quantized data and semantic knowledge, denoted as the target feature vector; The target monitoring module is used to perform target monitoring operations based on the target feature vector and the knowledge graph.

[0014] This invention also provides a target monitoring device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the target monitoring method as described in any of the above embodiments.

[0015] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the target monitoring method as described in any of the above embodiments.

[0016] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the target monitoring method as described in any of the above embodiments.

[0017] Compared with existing technologies, the target monitoring method, apparatus, equipment, storage medium, and program products disclosed in this invention acquire multimodal data in real time, extract and fuse features from the acquired multimodal data to obtain data features, construct a knowledge graph related to the target monitoring field, extract semantic features from the knowledge graph, and fuse the data features and semantic features to generate a target feature vector combining quantitative data and semantic knowledge. Then, based on the target feature vector and the knowledge graph, the entire target monitoring process is collaboratively implemented. Compared with the traditional approach of processing and applying data features and knowledge graphs independently, this invention enhances feature expression through the fusion of data features and semantic features, avoiding the problem of feature semantic fragmentation. Based on the feature vector and knowledge graph, a complete target monitoring process including target identification and behavior analysis is completed collaboratively, improving the accuracy and intelligence of target monitoring and enhancing the efficiency of safety supervision in related fields. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a target monitoring method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal data fusion process in an embodiment of the present invention; Figure 3 This is a schematic diagram of the knowledge graph update process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a target monitoring device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0021] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0022] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0023] See Figure 1 This is a flowchart illustrating a target monitoring method provided in an embodiment of the present invention. The embodiment of the present invention provides a target monitoring method, including steps S11 to S16: S11. Real-time acquisition of multimodal data through multi-sensor devices pre-deployed in the target area; S12. Perform feature extraction and feature fusion processing on the collected multimodal data to obtain data features; S13. Construct a knowledge graph related to the target monitoring field; S14. Extract the semantic features of the knowledge graph; S15. The data features and the semantic features are fused to generate a feature vector that combines quantized data and semantic knowledge, denoted as the target feature vector; S16. Perform target monitoring operation based on the target feature vector and the knowledge graph.

[0024] Preferably, the target monitoring operation includes target identification, behavior analysis, risk assessment, and strategy generation.

[0025] It should be noted that the target monitoring method of this invention can be applied to various fields according to actual needs, such as low-altitude flight, ground security, industrial operation and maintenance, and urban governance. The subsequent explanation of the process and effects of the target monitoring method using any one of these fields does not affect its application in other fields.

[0026] In this embodiment of the invention, in order to achieve target monitoring of a target area, a multi-sensor device is pre-deployed in the target area to collect multimodal data in real time. The multi-sensor device includes at least one of an optical camera, radar, acoustic sensor, and infrared sensor, and the multimodal data includes at least one of image data, radar data, acoustic data, and infrared data.

[0027] Taking low-altitude flight as an example, the multimodal data collected from low-altitude sensor devices, including radar, optical cameras, infrared sensors, and acoustic sensors, are shown in Table 1: Table 1

[0028] It is understood that the data in the table above is only an example. In actual application, relevant multimodal data can be collected according to the actual monitoring scenario, which will not affect the beneficial effects achieved by the present invention.

[0029] After step S11, the method further includes: performing data preprocessing on the multimodal data; wherein the data preprocessing includes data denoising, time synchronization, and data standardization.

[0030] Since these data have different characteristics, in order to improve the efficiency of data utilization, the collected data is first preprocessed.

[0031] Data denoising: Wavelet transform is used to denoise the image data, and Kalman filtering is used to smooth the radar data.

[0032] Timing synchronization: A clock synchronization algorithm is used to ensure that the timestamps of various sensor data are aligned, thus guaranteeing the timing consistency of multimodal data.

[0033] Data standardization: Standardize the outputs of different sensors to ensure that the input data is processed on the same scale.

[0034] After data preprocessing, feature extraction is performed on the collected multimodal data. Different representation methods are used for different modalities to extract features for each modality. Then, feature fusion processing is performed on the multimodal data to obtain the fused features, which are denoted as data features.

[0035] Furthermore, in this embodiment of the invention, a dynamic knowledge graph related to the target monitoring domain is pre-constructed. This dynamic knowledge graph is used to store and associate knowledge related to the target monitoring domain. Taking the low-altitude flight domain as an example, the knowledge graph is used to store and associate knowledge about low-altitude flight targets, such as the trajectories and behavioral patterns of birds and drones. The construction and updating of the dynamic knowledge graph is the foundation for achieving continuous monitoring and intelligent decision-making. Semantic features related to the target monitoring domain are extracted from the dynamic knowledge graph.

[0036] It should be noted that, in traditional solutions, the multi-sensor data fusion and knowledge graph query processes are independent of each other. Multi-sensor fusion outputs the basic features of the target (position, velocity, category), and the static knowledge graph is queried to obtain the preset rules for the target for behavior analysis and risk assessment. Data features and knowledge graph features are processed independently, and they are simply associated through feature matching. Semantic association is missing, which can easily lead to target recognition errors and behavior prediction deviations, making it difficult to generate accurate and effective processing decisions.

[0037] Based on this, embodiments of the present invention fuse the data features and the semantic features to generate a feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector.

[0038] Data characteristics ( Numerical features extracted from collected data. For example, numerical, categorical, or time-series data in structured data may be high-dimensional vector representations obtained through statistical methods or deep learning model preprocessing.

[0039] Semantic features ( Semantic vectors, derived from knowledge graphs, are extracted from dynamic knowledge graphs using graph embedding techniques. The core idea is to transform unstructured semantic information such as entities and relationships in the graph into computable, low-dimensional, dense vectors. A knowledge graph is a knowledge base that uses a graph structure to represent entities and their relationships. In feature fusion, knowledge graph features are typically vector representations extracted from nodes (entities) and edges (relationships) in the graph structure through embedding (e.g., using graph neural networks or graph embedding methods). These features supplement the semantic information lacking in the collected data.

[0040] Furthermore, based on the target feature vector and the knowledge graph, the multimodal data and knowledge graph are coordinated to realize a complete series of target monitoring operations in the target region, including target identification, behavior analysis, risk assessment, and strategy generation.

[0041] By employing the technical means of this invention, multimodal data is collected in real time. Feature extraction and feature fusion processing are performed on the collected multimodal data to obtain data features. Simultaneously, a knowledge graph related to the target monitoring field is constructed, and semantic features of the knowledge graph are extracted. The data features and semantic features are fused to generate a target feature vector combining quantitative data and semantic knowledge. Then, based on the target feature vector and the knowledge graph, the entire target monitoring process is collaboratively implemented. Compared to the traditional approach where data features and knowledge graphs are processed and applied independently, this invention enhances feature expression through the fusion of data features and semantic features, avoiding the problem of feature semantic fragmentation. Based on the feature vector and knowledge graph, a complete target monitoring process, including target identification and behavior analysis, is completed collaboratively, improving the accuracy and intelligence of target monitoring and enhancing the efficiency of safety supervision in related fields.

[0042] As a preferred embodiment, the present invention further implements the above embodiments. Step S12, namely, performing feature extraction and feature fusion processing on the collected multimodal data to obtain data features, includes steps S121 to S124: S121. Extract features from each modality of the multimodal data; S122. Evaluate the confidence level of each modal data based on the modal quality factor; S123. Calculate the weighting coefficient for each modality data based on the confidence level; S124. The features of each modality data are weighted and fused according to the weight coefficients to obtain the fused data features.

[0043] Preferably, the feature extraction for each modality in the multimodal data includes: For different modalities of the multimodal data, different feature extraction methods are used to extract features. The feature extraction methods are as follows: for image data, a convolutional neural network is used to extract features; for radar data, the radar data is converted into a two-dimensional image through polar coordinate transformation for feature extraction; for acoustic data, spectral features are extracted through Fourier transform; for infrared data, a thermal image enhancement algorithm is used to enhance the contrast and clarity of the infrared image, and a convolutional neural network is used to extract features.

[0044] It should be noted that in existing technologies, after multiple sensors acquire multimodal data, they usually rely on fixed weights to fuse multimodal information. This method cannot dynamically adapt to environmental changes. For example, in foggy weather, the data from optical cameras may be distorted, but their weights remain consistent with those of radar. This results in decision-making accuracy being greatly affected by environmental disturbances, which in turn cause the target features to be distorted, leading to an increase in the false alarm rate.

[0045] To solve this problem, see Figure 2This is a schematic diagram of the multimodal data fusion process in an embodiment of the present invention. The embodiment of the present invention evaluates the confidence level of each modal data based on the modal quality factor, and calculates the weight coefficient of each modal data. Then, the features of each modal data are weighted and fused according to the weight coefficient to solve the feature distortion problem caused by environmental disturbance.

[0046] Specifically, for the multimodal data at each time step, the weight coefficient of each modality is first calculated by evaluating the confidence level of each modality. The specific calculation method is as follows.

[0047] The confidence score is calculated for each type of multimodal data, as shown below:

[0048] in, This is the modal quality factor vector. = [SNR (Signal-to-Noise Ratio), Contrast Ratio, Coverage], directly reflects data quality (e.g., low SNR in foggy optical data). (small value) The mode-specific weight matrix consists of learnable parameters obtained during model training. It is the weight matrix corresponding to mode m, and its function is to adjust the quality factor vector of mode m. (Values ​​are SNR, contrast, and coverage) A weighted transformation is performed to quantify the influence of different quality factors on the confidence level of this mode. The modality-specific bias term is a learnable parameter obtained during model training. It is a scalar bias corresponding to the modality m, and its function is to... The linear transformation results are offset-calibrated to compensate for the baseline differences in quality factors between different modes. Each mode m corresponds to a unique... and σ is the Sigmoid activation function, which maps the confidence level to the interval [0,1], where 0 represents completely untrustworthy and 1 represents completely trustworthy.

[0049] Furthermore, based on the confidence assessment results of the modal signals, the weighting coefficients of mode m are calculated. Then normalize it, and the calculation method is as follows:

[0050] in, This is the weight sharpening coefficient, which is set to 1.0 by default in this embodiment of the invention. It can be dynamically adjusted to amplify the weight proportion of high-confidence modes and enhance the weight discrimination.

[0051] Based on the calculated weighting coefficients We perform weighted fusion of data from different modalities. Let the feature representation of each modal data be... The features after fusion are represented as:

[0052] Where M is the number of effective modes.

[0053] The feature fusion process is as follows: For each modality of data, features are extracted using the corresponding feature extraction model (e.g., CNN for image data, RNN for time series data, etc.). The confidence evaluation results for each modality are calculated, and then the weight coefficients are calculated. Based on the dynamically calculated weight coefficients, the features of all modalities are weighted and fused to obtain the final feature representation.

[0054] For example, in foggy conditions, the SNR of optical data is low, so the confidence level C... optical Decrease, weight α optical Automatic reduction; while radar data is less affected by fog, confidence level C radar High, weight α radar This improves upon traditional fixed weights, avoids feature distortion, and solves the problem of environmental adaptability.

[0055] By employing the technical means of this invention, the confidence level of each modal data is evaluated based on the modal quality factor, and the weight coefficients are dynamically adjusted to realize a dynamic weight allocation mechanism and dynamic weighted fusion of multimodal data. This ensures that high-reliability modal features can still be preferentially selected under environmental disturbances, avoids feature distortion caused by traditional fixed weights, and ensures that the fusion result is always tilted towards the most reliable mode in the current environment. This solves the problem that traditional fixed weights cannot adapt to environmental changes and avoids feature distortion caused by environmental disturbances.

[0056] As a preferred embodiment, the present invention further implements the above embodiments. Step S13, namely the construction of a knowledge graph related to the target monitoring field, includes steps S131 to S132: S131. Construct a knowledge graph related to the target monitoring domain based on a preset graph database; wherein, the knowledge graph includes several nodes related to the target monitoring domain and edges used to represent the association between nodes; S132. Based on the multimodal data, update the corresponding nodes and edges in the knowledge graph in real time to obtain a dynamic knowledge graph.

[0057] Specifically, the nodes are used to characterize target objects, environmental factors, geographical factors, behavioral patterns, or treatment methods related to the target detection field.

[0058] Taking low-altitude flight as an example, a dynamic knowledge graph of low-altitude targets is constructed using a graph database. Nodes in the graph represent different targets, flight paths, behavior patterns, etc., while edges represent the relationships between targets. For example, the initial nodes and relationships in the graph database are as follows: Entity nodes: {Drones (multi-rotor / fixed-wing), birds, kites, airport no-fly zones, gale-force winds of level 6}; Relationship edge: {Drone - Flight altitude - 120m, Drone - No ADS-B - Yes, Level 6 gale - Impact - Flight trajectory}.

[0059] It should be noted that most existing knowledge graphs are static, constructed based on predefined rules and manually labeled triples ("entity-relationship-entity" format). They rely on manually entered rules and are typically updated weekly or monthly, resulting in long update cycles. This makes them unable to capture dynamic relationships or respond to environmental changes in real time, leading to delays in responding to new threats and an inability to infer implicit causal chains. While graph databases are used for storage, the graph structure is fixed, supporting only queries and not dynamic expansion.

[0060] Therefore, this invention proposes a dynamic knowledge graph construction and update method. Through real-time tracking and data collection of the target, the nodes and edges in the graph are updated in real time. Employing an incremental learning algorithm, this invention uses an innovative algorithm of GNN-LLM collaboration to dynamically update the knowledge graph.

[0061] For a preferred embodiment, see Figure 3 This is a schematic diagram of the knowledge graph update process in an embodiment of the present invention. Step S132, namely, updating the corresponding nodes and edges in the knowledge graph in real time based on the multimodal data to obtain a dynamic knowledge graph, includes: When the preset update triggering condition is met based on the multimodal data, the corresponding nodes and edges in the knowledge graph are updated according to the message passing mechanism of the graph neural network. Generate a natural language description based on the preset prompt word template and the multimodal data; The natural language description is semantically inferred using a large language model to generate new triples and corresponding confidence scores; wherein the triples are in entity-relation-entity format. New triples with a confidence level greater than a preset confidence threshold are added to the knowledge graph to update the knowledge graph.

[0062] Preferably, the update triggering conditions include, but are not limited to, abnormal target behavior or environmental changes.

[0063] In this embodiment of the invention, when an incremental event is detected that meets the update triggering condition, such as when a change in the target's behavior pattern is detected, the core features of the target, environment, and behavior are extracted from the incremental event and updated through the corresponding nodes and relationships in the graph neural network (GNN) message passing graph, as shown below:

[0064] in, These are the initial features of the target node. This is the set of neighboring nodes of the target node. AGGREGATE is the operation for aggregating the features of neighboring nodes, UPDATE is the operation for updating the features of existing nodes and edges, and there are also operations for adding nodes and making changes.

[0065] For example, the initial characteristics of the target node are "flight altitude 125m, no ADS-B signal", and the adjacent nodes are "level 6 gale, no-fly zone". After the update, the node characteristics are updated with "trajectory hovering label".

[0066] Furthermore, a natural language description is generated based on the preset prompt word template and the multimodal data. A large language model is used to perform semantic inference on the natural language description to generate a new triplet and a corresponding confidence level. It is determined whether the confidence level is greater than a preset confidence level threshold (e.g., 70%). If so, the new triplet is added to the knowledge graph and the knowledge graph is updated; otherwise, the new triplet is discarded.

[0067] For example, the prompt word template for the Large Language Model (LLM) is: "Inferring implicit relations based on [Target ID123 hovers at position (x,y). Wind speed is level 5], output format: [Entity 1][Relation][Entity 2][Confidence]". LLM completes the causal chains missing in traditional GNNs, such as "increased wind speed → increased drone swaying amplitude → abnormal trajectory", improving the accuracy of risk prediction. It utilizes the inference engine in the graph to reason about the target, predict its future behavior, and dynamically adjust security strategies based on the prediction results.

[0068] Preferably, step S14, namely, extracting the semantic features of the knowledge graph, includes: From the dynamically updated knowledge graph, filter out entities and relationships that are strongly related to the current target monitoring scenario; The entities and relation elements are mapped into low-dimensional vectors using graph neural networks (GNNs) or dedicated graph embedding algorithms. For the current target monitoring scenario, all associated entity vectors and relation vectors are aggregated to form the final semantic features.

[0069] By employing the technical means of this invention, combining the real-time message passing capability of Graph Neural Networks (GNNs) with the implicit relational reasoning capability of Large Language Models (LLMs), an event-triggered dynamic update architecture is constructed. When abnormal target behavior is detected, the causal chain missing in traditional rule engines is automatically completed by parsing natural language descriptions through LLM, achieving minute-level knowledge graph updates.

[0070] By rapidly aggregating multi-source features of targets, environment, and geography through GNN message passing, the node status is refreshed in real time, ensuring update response speed and real-time performance. The inference capability of LLM completes the implicit causal chains that traditional GNN cannot capture, solving the problem of rigid relationships in static graphs. Only nodes and relationships related to abnormal events are updated, without reconstructing the entire graph, balancing efficiency and resource consumption, and adapting to the real-time monitoring needs of low-altitude security.

[0071] Furthermore, due to the dynamic real-time updates of the knowledge graph, the updated knowledge graph, through message passing mechanisms and LLM to complete the implicit causal chains, generates new semantic features FVkg in real time using graph embedding technology, which are then synchronously input into the feature fusion stage to ensure the semantic features are updated. With data characteristics The dynamic matching is integrated into the final target feature vector, providing more accurate semantic support for the entire target monitoring process.

[0072] As a preferred embodiment, the present invention further implements the above embodiments. Step S15, namely, fusing the data features and the semantic features to generate a feature vector combining quantized data and semantic knowledge, denoted as the target feature vector, includes: The Concatenate algorithm is used to concatenate the data features and the semantic features along the same dimension to generate a high-dimensional feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector.

[0073] In this embodiment of the invention, different types of information features are combined, and multiple information sources are utilized to improve the accuracy of prediction. Data features ( ) and semantic features of knowledge graphs ( The features are then merged using the Concatenate operation to form the final feature vector. : The concatenate algorithm is a simple yet effective feature fusion method. It concatenates two types of feature vectors along a certain dimension to obtain a higher-dimensional feature vector. This preserves the objective quantitative accuracy of the data while supplementing the semantic reasoning ability of knowledge, thus solving the problem of data and knowledge separation in traditional solutions.

[0074] Based on this and A new feature vector was formed by splicing. :

[0075] The concatenated feature vector contains numerical features from the dataset and semantic information from the knowledge graph. This fusion method does not change the original feature representation, but enhances the model's representational power by merging multi-source information.

[0076] Compared to the simple association between data features and graph query results in the prior art, the present invention employs the technical means of this embodiment to densely fuse numerical feature vectors and semantic feature vectors through feature splicing. The two achieve semantic alignment in the vector space, which enhances the data features with the semantic information of the knowledge graph. This is beneficial to improving the accuracy of the complete target monitoring process, including subsequent target identification, behavior analysis, risk assessment, and decision generation.

[0077] As a preferred embodiment, the present invention further implements the above embodiments, and step S16, namely the target monitoring operation based on the target feature vector and the knowledge graph, includes: The target feature vector and the knowledge graph are input into a preset target recognition model to obtain the target category probability output by the target recognition model, and the recognized target is determined based on the target category probability. The target feature vector and the knowledge graph are input into a preset behavior analysis model to obtain a behavior label sequence output by the behavior analysis model. The behavioral label sequence and the knowledge graph are input into a preset risk assessment model to obtain the risk value output by the risk assessment model. The risk value and the knowledge graph are input into a preset decision generation model to obtain the action vector output by the decision generation model, which serves as the disposal strategy.

[0078] Preferably, the target recognition model adopts a network structure of fully connected layers (FC) and a Softmax classifier; the behavior analysis model adopts a network structure of long short-term memory neural network (LSTM) and attention mechanism; the risk assessment model adopts a network structure of fully connected layers (FC) and sigmoid function; and the decision generation model adopts a policy network structure.

[0079] In this embodiment of the invention, based on the fused target feature vector, combined with the dynamic knowledge graph and real-time collected multimodal data, the entire process of target monitoring is executed through four-dimensional decision-making to complete target identification, behavior analysis, risk assessment and decision generation.

[0080] A four-dimensional decision-making method, comprising target recognition (FC+Softmax), behavior analysis (LSTM+Attention), risk assessment (FC+Sigmoid), and response strategy (Policy Network), is used for target identification, tracking, and behavior prediction. The target monitoring process is shown in Table 2. Table 2

[0081] Taking low-altitude flight scenarios as an example, based on fused feature vectors This involves constructing an end-to-end intelligent decision-making framework. In low-altitude security applications, such as airport airspace clearance, major event security, and border monitoring, the system needs to complete target identity determination, behavioral intent analysis, risk level assessment, and response strategy generation in real time. The four-dimensional decision-making model achieves closed-loop security response through the collaborative efforts of the following sub-modules.

[0082] Target recognition (identity classification): Differentiating targets such as drones, birds, and kites in complex low-altitude environments to avoid misclassification. A combination of a fully connected layer (FC) and a Softmax classifier is used as the input. Output the probability distribution of the target category.

[0083] For example, optical features (shape and texture) and radar micro-motion features (rotor frequency) are used to classify it as a "multi-rotor drone"; thermal infrared temperature distribution and acoustic signature features are used to classify it as a "bird flock". The dynamic knowledge graph provides prior constraints (such as "no kite activity within 10km of the airport") to correct the classification results.

[0084] Behavioral analysis (intent understanding): Identifying abnormal target behavior, such as hovering, maneuvering, and diving, and predicting potential threats, such as reconnaissance, throwing, and impact. Based on LSTM and attention mechanisms, it analyzes the temporal features of the target's motion trajectory.

[0085] For example, the input might be historical trajectory coordinates, real-time wind speed from weather sensors, and knowledge graph relationships, such as "wind speed > level 5 increases the probability of trajectory deviation." The output is a sequence of behavioral labels, such as "hovering → accelerating dive → circling," marking abnormal behavior nodes, such as "dive height below building clusters." By using LLM to complete implicit causal chains, such as dive height and no-fly zone coordinates, suspected unauthorized intrusions can be inferred.

[0086] Risk assessment (threat quantification): This involves comprehensively considering the environment, target attributes, and behavioral patterns to quantify the level of security threats, such as drones approaching high-voltage power lines or densely populated areas. Risk values ​​are output through a fully connected layer (FC) and a sigmoid function. .

[0087] Static risk: Target type weight (drones = 0.9, birds = 0.3) × proximity to no-fly zone; Dynamic risk: Degree of behavioral abnormality (e.g., "circling radius") 50m”→ R+0.2) Environmental factors (such as visibility) 1km→R×1.5).

[0088] Knowledge graphs provide connections between historical events, such as "similar trajectories have caused power outages," improving the accuracy of risk assessment.

[0089] Response strategy: Generate tiered response instructions to balance security efficiency with response costs, such as tracking warnings or physical interception.

[0090] The policy network outputs multi-dimensional action vectors, as shown in Table 3: Table 3

[0091] In the multi-dimensional action vector output by the Policy Network, where R represents a specific threshold for the risk assessment result. Specifically, R is a risk value derived from the assessment results of multiple factors, representing the comprehensive risk level of factors such as target behavior, location, and environment. In this scheme, the calculation of R considers the target type (e.g., birds, drones, etc.), target behavior (e.g., hovering, diving, etc.), target location (whether it is near a no-fly zone or sensitive area), environmental factors (e.g., weather, obstacles, etc.), and other security threats.

[0092] Furthermore, the target is a bird: When the risk value is below 0.3, the target is considered low-risk, and the target is a low-threat target such as a bird. In this case, the handling strategy includes "continuous tracking" and "audio-visual warnings," meaning that high-intensity intervention is not required, but rather potential conflict is avoided through monitoring and warnings.

[0093] When the risk value is between 0.3 and 0.7, the target's behavior or location represents a medium risk, involving unusual behavior or the target being close to the no-fly zone. In this case, employing "electromagnetic interference" and "coordinated drone countermeasures" is to effectively interfere with and deter potential threats.

[0094] Or intrusion into the core area: When the risk value is greater than or equal to 0.7, the target is considered high-risk due to abnormal target behavior (e.g., rapidly approaching a no-fly zone or core area) or the target itself being highly threatening (e.g., a drone). At this time, the system activates "laser interception" and "reporting to the control center" to take action as soon as possible to avoid more serious security risks.

[0095] It should be noted that the dynamic knowledge graph in this embodiment of the invention is not merely a query library, but is deeply integrated into the entire target monitoring process, providing semantic enhancement and reasoning capabilities. By supplementing implicit causality, it explains the logic behind target behavior, provides prior constraints, and corrects monitoring results; by associating with historical knowledge and calling historical event connections, it improves the accuracy of risk assessment.

[0096] This invention proposes a deep collaborative mechanism between cross-modal features and knowledge graph semantics. It fuses data features extracted from sensors with semantic features from the knowledge graph to generate enhanced feature representations. A four-dimensional joint decision head is designed to simultaneously output target recognition results, behavior analysis labels, risk assessment values, and response strategies, achieving end-to-end security response. Unlike traditional solutions that treat data and knowledge in a fragmented manner, this invention enhances feature expression through knowledge semantics and integrates multi-task decision-making capabilities, improving the accuracy of the entire target monitoring process.

[0097] See Figure 4 This is a schematic diagram of the structure of a target monitoring device provided in an embodiment of the present invention. The embodiment of the present invention provides a target monitoring device 10, comprising: The multimodal data acquisition module 11 is used to acquire multimodal data in real time through multi-sensor devices pre-deployed in the target area; The data feature extraction module 12 is used to perform feature extraction and feature fusion processing on the collected multimodal data to obtain data features; Knowledge graph construction module 13 is used to construct dynamic knowledge graphs related to the target monitoring field; Semantic feature extraction module 14 is used to extract the semantic features of the dynamic knowledge graph; The feature vector fusion module 15 is used to fuse the data features and the semantic features to generate a feature vector that combines quantized data and semantic knowledge, denoted as the target feature vector; The target monitoring module 16 is used to perform target monitoring operations based on the target feature vector and the knowledge graph.

[0098] It should be noted that the target monitoring device provided in this embodiment of the invention is used to execute all the process steps of the target monitoring method in the above embodiment. The working principle and beneficial effects of the two are one-to-one, so they will not be described again.

[0099] This invention also provides a target monitoring device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the target monitoring method as described in any of the above embodiments.

[0100] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the target monitoring method as described in any of the above embodiments.

[0101] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the target monitoring method as described in any of the above embodiments.

[0102] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0103] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A target monitoring method, characterized in that, include: Multimodal data is collected in real time using multi-sensor devices pre-deployed in the target area; Feature extraction and feature fusion are performed on the collected multimodal data to obtain data features; Construct a knowledge graph related to the field of target monitoring; Extract the semantic features of the knowledge graph; The data features and semantic features are fused to generate a feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector; Target monitoring is performed based on the target feature vector and the knowledge graph.

2. The target monitoring method as described in claim 1, characterized in that, The process of extracting and fusing features from the collected multimodal data to obtain data features includes: Feature extraction is performed on each modality of the multimodal data; The confidence level of each modal data is assessed based on the modal quality factor; Calculate the weighting coefficient for each modality based on the confidence level; The features of each modality data are weighted and fused according to the weight coefficients to obtain the fused data features.

3. The target monitoring method as described in claim 2, characterized in that, The multi-sensor device includes at least one of an optical camera, radar, acoustic sensor, and infrared sensor; the multimodal data includes at least one of image data, radar data, acoustic data, and infrared data.

4. The target monitoring method as described in claim 1, characterized in that, The construction of a knowledge graph related to the target monitoring domain includes: A knowledge graph related to the target monitoring domain is constructed based on a pre-defined graph database; wherein, the knowledge graph includes several nodes related to the target monitoring domain and edges used to represent the relationships between nodes; Based on the multimodal data, the corresponding nodes and edges in the knowledge graph are updated in real time to obtain a dynamic knowledge graph.

5. The target monitoring method as described in claim 4, characterized in that, The step of updating the corresponding nodes and edges in the knowledge graph in real time based on the multimodal data to obtain a dynamic knowledge graph includes: When the preset update triggering condition is met based on the multimodal data, the corresponding nodes and edges in the knowledge graph are updated according to the message passing mechanism of the graph neural network. Generate a natural language description based on the preset prompt word template and the multimodal data; The natural language description is semantically inferred using a large language model to generate new triples and corresponding confidence scores; wherein the triples are in entity-relation-entity format. New triples with a confidence level greater than a preset confidence threshold are added to the knowledge graph to update the knowledge graph.

6. The target monitoring method as described in claim 1, characterized in that, The process of fusing the data features and the semantic features to generate a feature vector combining quantized data and semantic knowledge, denoted as the target feature vector, includes: The Concatenate algorithm is used to concatenate the data features and the semantic features along the same dimension to generate a high-dimensional feature vector that combines quantified data and semantic knowledge, denoted as the target feature vector.

7. The target monitoring method as described in claim 1, characterized in that, The target monitoring operation includes target identification, behavior analysis, risk assessment, and strategy generation; The target detection operation based on the target feature vector and the knowledge graph includes: The target feature vector and the knowledge graph are input into a preset target recognition model to obtain the target category probability output by the target recognition model, and the recognized target is determined based on the target category probability. The target feature vector and the knowledge graph are input into a preset behavior analysis model to obtain a behavior label sequence output by the behavior analysis model. The behavioral label sequence and the knowledge graph are input into a preset risk assessment model to obtain the risk value output by the risk assessment model. The risk value and the knowledge graph are input into a preset decision generation model to obtain the action vector output by the decision generation model, which serves as the disposal strategy.

8. The target monitoring method as described in claim 7, characterized in that, The target recognition model adopts a network structure of fully connected layers (FC) and a Softmax classifier; the behavior analysis model adopts a network structure of long short-term memory neural network (LSTM) and an attention mechanism; the risk assessment model adopts a network structure of fully connected layers (FC) and a sigmoid function; and the decision generation model adopts a policy network structure.

9. A target monitoring device, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal data in real time using multi-sensor devices pre-deployed in the target area; The data feature extraction module is used to extract and fuse features from the collected multimodal data to obtain data features. The knowledge graph construction module is used to build dynamic knowledge graphs related to the target monitoring field. A semantic feature extraction module is used to extract the semantic features of the dynamic knowledge graph; The feature vector fusion module is used to fuse the data features and the semantic features to generate a feature vector that combines quantized data and semantic knowledge, denoted as the target feature vector; The target monitoring module is used to perform target monitoring operations based on the target feature vector and the knowledge graph.

10. A target monitoring device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the target monitoring method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the target monitoring method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the target monitoring method as described in any one of claims 1 to 8.

Images