A Multimodal Target Fusion and Evaluation Method

Abstract

This invention discloses a multimodal target fusion and evaluation method, comprising: aligning and preprocessing multi-source observation data to generate multi-source time-aligned observation data and multi-source sensor state data; performing single-source target detection and initial threat estimation accordingly to construct single-source target state data and initial target threat assessment data; applying threat consistency constraints to perform multi-sensor target trajectory association and fusion based on the single-source target state data, initial target threat assessment data, and multi-source sensor state data to generate multi-sensor fused target model data; and applying threat reduction benefit assessment to perform multi-sensor collaborative scheduling decision-making to determine multi-sensor scheduling scheme data. This invention achieves deep coupling of association and scheduling with the threat assessment task, improving the accuracy of association and the effectiveness of scheduling decisions.

Description

Technical Field

[0001] This invention belongs to the field of modern surveillance and command control, and in particular to a method for multimodal target fusion and evaluation. Background Technology

[0002] In the field of modern surveillance and command and control, facing increasingly complex electromagnetic and physical environments, the efficient fusion of observational data from multiple heterogeneous sensors, such as radar, optoelectronics, and UAV payloads, has become crucial for enhancing situational awareness capabilities. Accurate and real-time multi-target fusion and threat assessment are core prerequisites for achieving autonomous decision-making, ensuring the security of critical assets, and optimizing resource allocation, and thus have significant research and application value.

[0003] Current multi-sensor data fusion technologies primarily focus on improving the estimation accuracy of target states (such as position and velocity). To this end, researchers have employed state estimation algorithms including Kalman filtering and particle filtering. In track association, widely used methods include multiple hypothesis tracking (MHT) and joint probabilistic data association (JPDA), which mainly rely on matching the target's kinematic characteristics and appearance features. Meanwhile, in the field of sensor management, existing scheduling strategies also primarily aim to maximize target tracking accuracy or coverage.

[0004] However, existing technologies still have significant shortcomings in achieving deep coupling between high-level semantics (such as threats) and low-level data (such as state). Specifically, the following technical problems exist: Sensor scheduling decisions are disconnected from threat assessment tasks. Existing scheduling models often aim to optimize the accuracy of target state estimation (such as position covariance), neglecting the ultimate goal of scheduling—reducing the uncertainty of threat assessment. This state optimization is not threat cognition optimization, resulting in sensor resources not being prioritized for resolving the most critical threat uncertainties. Furthermore, the track association process lacks threat continuity constraints. Traditional association costs only consider motion and appearance, leading to a separation between association (data layer) and threat assessment (semantic layer). When target trajectories intersect or become ambiguous, mis-associations can easily occur, causing unreasonable and drastic jumps in target threat scores over time, reducing the stability and reliability of the overall situational awareness. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal target fusion and evaluation method to solve the aforementioned problems existing in the prior art.

[0006] Technical solution, a multimodal target fusion and evaluation method, including:

[0007] Acquire multi-source observation data and perform alignment preprocessing to generate multi-source time-aligned observation data and multi-source sensor status data; based on this, perform single-source target detection and initial threat estimation to construct single-source target status data and initial target threat assessment data.

[0008] Based on single-source target state data, initial target threat assessment data, and multi-source sensor state data, threat consistency constraints are applied to perform multi-sensor target trajectory association and fusion to generate multi-sensor fused target model data.

[0009] Based on multi-sensor fusion target model data and multi-source sensor status data, the application of threat reduction benefit assessment executes multi-sensor collaborative scheduling decisions to determine multi-sensor scheduling scheme data.

[0010] Beneficial effects: This invention achieves deep coupling of association and scheduling for threat assessment tasks, improving the accuracy of association and the effectiveness of scheduling decisions. Attached Figure Description

[0011] Figure 1 A flowchart illustrating the steps of a multimodal target fusion and evaluation method provided in this application embodiment.

[0012] Figure 2 A flowchart illustrating the steps for determining multi-sensor scheduling scheme data provided in this application embodiment.

[0013] Figure 3 A flowchart illustrating the steps for constructing threat reduction benefit assessment data provided in this application embodiment.

[0014] Figure 4 A flowchart illustrating the steps for deriving the predicted state covariance provided in this application embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to specific embodiments. It should be noted that the embodiments of this invention can be implemented based on various computing environments and system architectures. For example, the method described in this invention can be executed by a processor in a computer system executing computer-executable instructions stored in memory. The computer system may include, but is not limited to, a server, workstation, or embedded system, which includes a processor, memory, communication interface, and I / O devices. The communication interface is used to communicate with external multi-source sensing devices, which exemplarily include radar, fixed optoelectronic stations, and UAV payloads.

[0016] like Figure 1 As shown, a multimodal target fusion and evaluation method is proposed, including the following steps:

[0017] Acquire multi-source observation data and perform alignment preprocessing to generate multi-source time-aligned observation data and multi-source sensor status data.

[0018] In other words, we acquire multi-source observation data, including raw radar echo data, raw fixed optoelectronic images, and raw images from UAV payloads. We then perform alignment and quality assessment on the multi-source observation data to generate multi-source time-aligned observation data and multi-source sensor status data.

[0019] In this embodiment, the multi-source observation data originates from various preset sensing devices, such as raw radar echo data, raw fixed photoelectric image data, and raw image data from UAV payloads. The system reads the raw observation information from these devices and simultaneously acquires the sensor position, attitude, and timestamp data corresponding to the observation information. Alignment preprocessing can resolve inconsistencies in time references and spatial coordinate systems between different sensor data. Specifically, this preprocessing utilizes preset device calibration parameter data to perform time synchronization, coordinate unification, and quality assessment on various types of observation data. Time synchronization maps all data to a unified time axis, while coordinate unification (e.g., transformation to a unified geodetic coordinate system) ensures the comparability of spatial locations. The output multi-source time-aligned observation data and multi-source sensor status data form a structured, unified input basis.

[0020] In some alternative implementations, coordinate unification is not limited to transformation to a geodetic coordinate system; it can also be transformation to a specific coordinate system or a relative coordinate system based on a particular sensor. Furthermore, quality assessment processing may include preliminary screening for data integrity, signal-to-noise ratio, or image sharpness to remove low-quality data.

[0021] Based on multi-source time-aligned observation data and multi-source sensor state data, single-source target detection and initial threat estimation are performed to construct single-source target state data and initial target threat assessment data.

[0022] In this embodiment, the system performs independent analysis and processing on multi-source time-aligned observation data according to the sensor source (e.g., radar, fixed electro-optical, UAV). Specifically, the system combines preset radar target detection model parameter data and electro-optical target recognition model parameter data to perform target detection, target classification, and preliminary state estimation on radar observations, fixed electro-optical observations, and UAV observations, respectively. The output of this step is constructed into two types of key data: single-source target state data and initial target threat assessment data. The single-source target state data describes the physical attributes of the target, exemplarily including motion elements such as target position, velocity, and heading, as well as identification elements such as target category and recognition confidence. The initial target threat assessment data calculates a preliminary threat score and corresponding uncertainty estimate for each detected single-source target based on a preset initial threat assessment model. These two types of data will serve as the core inputs for subsequent multi-sensor trajectory association and threat consistency constraint calculations.

[0023] Optionally, the radar target detection model and the electro-optical target recognition model can be deep learning-based models (such as YOLO, Faster R-CNN) or traditional signal / image processing algorithms. The initial threat assessment model at this stage can be a simplified rule-based model, for example, giving a rough score based only on the target's category and speed (such as high speed).

[0024] Based on single-source target state data, initial target threat assessment data, and multi-source sensor state data, threat consistency constraints are applied to perform multi-sensor target trajectory association and fusion, generating multi-sensor fused target model data.

[0025] Specifically, the system reads single-source target state data and initial target threat assessment data, and performs candidate matching on multi-source target observations from radar, fixed electro-optical systems, and UAVs within a unified temporal and spatial coordinate framework. Furthermore, a threat consistency constraint is introduced during track association. In other words, when performing global association optimization, the system not only comprehensively considers the consistency of target position and motion, and the similarity of appearance features across sensors, but also the smoothness of target threat changes over time, ensuring that the association results remain continuous and consistent at both the physical and semantic (threat) levels. Through association optimization, the system obtains unified target track data that is continuous across time and sensors. State fusion and threat fusion are performed on each unified target track data, outputting multi-sensor fused target model data. For example, the multi-sensor fused target model data is a comprehensive model containing target state time series, threat score time series, and threat uncertainty time series. Optionally, comprehensive target threat assessment data can also be generated, reflecting the fusion results at the current moment. In some embodiments, global association optimization can be implemented using the Hungarian algorithm, the Jonker-Volgenant (JVC) algorithm, or multiple hypothesis tracking (MHT), but their cost functions all include a threat consistency cost term.

[0026] Based on multi-sensor fusion target model data and multi-source sensor status data, the application of threat reduction benefit assessment executes multi-sensor collaborative scheduling decisions to determine multi-sensor scheduling scheme data.

[0027] Unlike traditional scheduling methods that only aim to improve the accuracy of target state estimation, this embodiment aims to maximize the accuracy of threat assessment. Specifically, the system reads multi-sensor fusion target model data and comprehensive target threat assessment data, combines multi-source sensor state data and a preset sensor capability model, and evaluates the possible observation tasks that can be performed between each sensor and each target. Preferably, the threat reduction benefit is quantified: the system evaluates the expected reduction in target threat uncertainty under given observation conditions, and uses this as the benefit of the observation task. Based on the constructed threat reduction benefit assessment data, a collaborative scheduling optimization model is further constructed with the goal of maximizing the reduction in overall threat uncertainty. This model is constrained by sensor motion constraints, task capacity constraints, and time window constraints during solution. The model solution yields the optimal observation targets, observation timings, and observation paths for each sensor, forming executable multi-sensor scheduling scheme data and constructing closed-loop scheduling capability. Optionally, the collaborative scheduling optimization model can not only maximize the overall benefit but also minimize the uncertainty of the highest threat target. Furthermore, scheduling decisions can be executed at fixed time periods or triggered when the global threat situation changes significantly.

[0028] like Figure 2 As shown, in one possible embodiment, determining multi-sensor scheduling scheme data includes:

[0029] Construct target threat uncertainty data, which characterizes the uncertainty index of the current target threat score.

[0030] In this embodiment, simply knowing the target's threat score (e.g., 0.8) is insufficient; the system also needs to know the reliability or uncertainty of that score. Targets with high threat but also high uncertainty should be prioritized for sensor scheduling. Optionally, using the fusion state estimate and its covariance information from the multi-sensor fusion target model data, and combining it with the current threat score from the comprehensive target threat assessment data, an uncertainty index for each target threat score is calculated using a mathematical model. This index can be quantified as a scalar value, such as the variance of the current threat score, Var. Threat_k The final target threat uncertainty data (e.g., a list containing [target number, threat score, threat score variance]) will be used to evaluate the observation gains.

[0031] By combining target threat uncertainty data, multi-sensor fusion target model data, and multi-source sensor state data, a sensor observation effect model is established to predict the target threat uncertainty under assumed observation conditions, thus forming post-observation target threat uncertainty prediction data.

[0032] For example, this embodiment can be used to evaluate how much the threat uncertainty of target k will decrease after the system assigns a sensor (e.g., UAV A) to observe a specific target (e.g., target k). To this end, the system needs to evaluate each sensor-target combination. Specifically, based on multi-source sensor state data (e.g., the current position and velocity of UAV A) and multi-sensor fusion target model data (e.g., the current position and velocity of target k), the system analyzes the reachability of the sensor to effectively observe the target within the current decision cycle. If reachable, the expected update effect of the observation action on the target state covariance is derived based on preset sensor capability model parameter data (e.g., the measurement accuracy of UAV A at a specific distance and angle). Using this expected, better state covariance, it is substituted back into the threat uncertainty calculation model to obtain the predicted value, i.e., the post-observation threat score variance Var. Threat_k_after (s). These predicted values ​​are aggregated to form post-observation target threat uncertainty prediction data.

[0033] Optionally, the sensor observation performance model can consider not only measurement accuracy but also the information dimension of the observation. For example, radar observation may primarily reduce the uncertainty of the target's position / velocity, while electro-optical observation may primarily reduce the uncertainty of the target's category; their contributions to reducing the final threat uncertainty are different. The contributions of these different dimensions can be uniformly measured through the gradient of the threat assessment model.

[0034] By comparing the target threat uncertainty data with the post-observation target threat uncertainty prediction data, the reduction in threat uncertainty is calculated, and threat reduction benefit assessment data is constructed.

[0035] Specifically, for each sensor-target combination, the system extracts the variance of the current threat score Var before observation. Threat_k (From target threat uncertainty data) and post-observation threat score variance Var Threat_k_after (s)(from post-observation target threat uncertainty prediction data). The difference between the two is Var Threat_k -Var Threat_k_after (s) represents the reduction in threat uncertainty brought about by the observation task (s,k) in the dimension of threat uncertainty, ΔU(s,k). The reduction ΔU(s,k) is defined as the threat reduction gain. The gain values ​​of all combinations are calculated and encapsulated to form threat reduction gain assessment data, which (e.g., the gain matrix) will be directly used as the objective function input for subsequent scheduling optimization.

[0036] Based on threat reduction benefit assessment data, a collaborative scheduling optimization model is constructed with the goal of maximizing the reduction in overall threat uncertainty, and the solution is used to generate multi-sensor scheduling scheme data.

[0037] Alternatively, based on threat reduction benefit assessment data, and combined with multi-source sensor status data and multi-sensor fusion target model data, a collaborative scheduling optimization model is constructed with the goal of maximizing the reduction of overall threat uncertainty, and multi-sensor scheduling scheme data is generated by solving the model.

[0038] In this embodiment, the system uses the threat reduction benefit assessment data (i.e., the benefit matrix ΔU) as the weight of the objective function of the optimization model. The optimization objective is to select a set of sensor-target assignment relationships (represented by the decision variable x(s,k)) to maximize the overall threat uncertainty reduction (i.e., maxΣ[ΔU(s,k)*x(s,k)]). While maximizing the benefit, the optimization model also needs to satisfy a series of realistic constraints. These constraints are derived from multi-source sensor state data (such as the task capacity of sensor s) and scheduling constraint parameter data (such as the maximum acceptable number of observations for target k in one cycle). By solving this optimization model (e.g., using an integer programming solver or approximate solutions such as greedy algorithms or Hungarian algorithms), the system obtains the optimal task assignment result. Based on this assignment result, and combined with the sensor maneuverability in the multi-source sensor state data and the target position prediction in the multi-sensor fusion target model data, specific executable observation paths and timings are planned to form the final multi-sensor scheduling scheme data, which is then sent to the sensor control system for execution.

[0039] In a preferred implementation of this embodiment, the quantification of threat uncertainty is achieved through local linear approximation and covariance propagation. It is assumed that the threat assessment model (used to calculate the threat score Threat) is a function of the target state vector X (containing position, velocity, class probability, etc.), i.e., Threat = g(X). In the current fused state X... k Nearby, this function can be approximated as a linear function using a first-order Taylor expansion. The gradient vector of the threat assessment model g(X) with respect to the state vector X is calculated and denoted as ∂g / ∂x. k The gradient vector ▽g k (Its dimension is the same as that of the state vector X) describes the sensitivity of the threat score to changes in each state component (such as speed and distance), and can be derived from the preset threat assessment model parameter data.

[0040] like Figure 3 As shown, in one exemplary embodiment, constructing threat reduction benefit assessment data includes:

[0041] Extract the fusion state covariance from the multi-sensor fusion target model data, and combine it with the pre-stored threat assessment model gradient vector to calculate the current threat score variance contained in the target threat uncertainty data.

[0042] Specifically, the fusion state covariance matrix Σ of target k is extracted from the multi-sensor fusion target model data. k Matrix Σ k Describes the current state of the target X k The uncertainty of the estimate. Using the covariance propagation law, the variance of the current threat score, Var... Threat_k(i.e., the core indicator of target threat uncertainty data) is calculated as: Var Threat_k ≈▽g k T *Σ k *▽g k .in T This represents the transpose of a vector / matrix. It can be seen that this formula incorporates the uncertainty (Σ) of the state estimation. k ) and the sensitivity of the threat model to the state (▽g) k This, combined with other methods, quantifies the uncertainty of threat scoring.

[0043] Derive the predicted state covariance under assumed observation conditions.

[0044] In this embodiment, based on the current threat score variance, prediction is performed to derive the predicted state covariance Σ. state_after (s, k). For example... Figure 4 As shown, in a preferred implementation, deriving the predicted state covariance includes: analyzing the observational reachability of the sensor and the target based on multi-source sensor state data and preset sensor capability model parameter data, predicting the measurement accuracy under assumed observation conditions, and constructing an observation measurement error covariance matrix; combining the fused state covariance and observation measurement error covariance matrix, applying the state covariance update formula, and deriving the predicted state covariance. Specifically, based on multi-source sensor state data and sensor capability model parameter data, the observational reachability and observation geometry of sensor s to target k are analyzed. According to the sensor capability model, the observation measurement error covariance matrix R of sensor s under this observation geometry is determined. meas (s, k). Observation measurement error covariance matrix R meas This describes the measurement noise level of the sensor itself. The system applies a state covariance update formula (e.g., the covariance update part of the Kalman update formula), combined with the fused state covariance Σ. k With the observation measurement error covariance matrix R meas (s, k), the predicted state covariance Σ is derived. state_after (s, k). An exemplary update formula is: Σ state_after =(IK gain *H obs )*Σ k Where I is the identity matrix, H obs Let K be the observation matrix. gain For Kalman gain, K gain =Σ k *H obs T *(H obs *Σ k *H obs T+R meas ) -1 .

[0045] By replacing the fused state covariance with the predicted state covariance and reusing the gradient vector of the threat assessment model, the variance of the post-observation threat score contained in the post-observation target threat uncertainty prediction data is calculated.

[0046] Specifically, after obtaining the predicted state covariance Σ state_after After (s, k), the system reuses the same threat assessment model gradient vector ▽g k Calculate the variance of the threat score after observation: Var Threat_k_after (s)≈▽g k T *Σ state_after (s,k)*▽g k .

[0047] The difference between the current threat score variance and the post-observation threat score variance is calculated as the reduction in threat uncertainty, in order to generate threat reduction benefit assessment data.

[0048] In this embodiment, the difference between the current threat score variance and the observed threat score variance is calculated. Further, to ensure the reasonableness of the gain (avoiding negative gains due to model nonlinearity or computational noise), the reduction in threat uncertainty is obtained by performing a non-negative truncation on the difference between the current threat score variance and the observed threat score variance to ensure that the reduction in threat uncertainty is non-negative. Specifically, the reduction in threat uncertainty ΔU(s,k) is calculated as: ΔU(s,k) = max(0, Var Threat_k -Var Threat_k_after (s)).

[0049] In a specific numerical case, suppose the state vector X of target k is only two-dimensional [position; velocity], and its fused state covariance Σ k =[[4, 0], [0, 1]]. Assume the gradient vector of the threat assessment model is ▽g. k =[0.1, 0.5] T (This indicates that the threat is more sensitive to speed than to location). Calculate the variance of the current threat score: Var Threat_k =[0.1, 0.5]*[[4, 0], [0, 1]]*[0.1, 0.5] T =0.04 + 0.25 = 0.29. Assuming sensor s can only accurately measure velocity, its observation measurement error covariance matrix R... meas The velocity component is very small, resulting in an updated predicted state covariance Σ. state_after(s, k) = [[4, 0], [0, 0.2]] (velocity variance decreases from 1 to 0.2). Calculate the variance of the threat score after observation: Var Threat_k_after (s)=[0.1, 0.5]*[[4, 0], [0, 0.2]]*[0.1, 0.5] T =0.04 + 0.05 = 0.09. Calculate the reduction in threat uncertainty: ΔU(s, k) = max(0, Var Threat_k -Var Threat_k_after (s))=max(0,0.29-0.09)=0.20. The 0.20 is the threat reduction benefit of sensor s to target k, which will be filled into the threat reduction benefit assessment data matrix for subsequent optimization decision-making.

[0050] According to one aspect of this application, the method for generating multi-sensor scheduling scheme data includes:

[0051] A task assignment optimization model is constructed with the reduction in threat uncertainty in the threat reduction benefit assessment data as the objective function weight. The objective of the task assignment optimization model is to maximize the overall reduction in threat uncertainty.

[0052] In this embodiment, the system uses threat reduction benefit assessment data (whose core is the benefit matrix ΔU(s,k)) as the core input. A task assignment optimization model is constructed. This model introduces a decision variable x(s,k), which is a binary variable. For example, x(s,k)=1 indicates that sensor s is assigned to observe target k within the current decision period, and x(s,k)=0 indicates that no assignment is made. The objective function of this optimization model is set to maximize the sum of threat reduction benefits for all selected tasks, which can be mathematically expressed as: maxΣ s Σ k (ΔU(s,k)*x(s,k)). Where Σ s and Σ k These represent summation over all sensors and over all targets, respectively.

[0053] The multi-sensor task assignment results are obtained by applying scheduling constraint parameter data to the task assignment optimization model; the scheduling constraint parameter data includes at least sensor task capacity constraints and target observability number constraints.

[0054] Alternatively, one can apply scheduling constraint parameters, including at least sensor task capacity constraints and target observability constraints, to the task assignment optimization model; solve the task assignment optimization model after applying the scheduling constraint parameters to obtain the multi-sensor task assignment results.

[0055] Specifically, to ensure the optimization results conform to physical and tactical realities, the task assignment optimization model needs to be subject to a series of constraints. Optionally, these constraints are derived from preset scheduling constraint parameter data or multi-source sensor state data. The first type of constraint is the sensor task capacity constraint; for example, for any sensor s, the total number of tasks it executes in the same period cannot exceed its maximum task capacity Cap. sensor (s), that is: Σ k (x(s,k))≤Cap sensor (s). For example, the Capacity of a drone sensor (sensor s=1) may be limited by its onboard processing power or communication bandwidth. sensor (1) = 2 (meaning a maximum of two targets can be tracked simultaneously). The second type of constraint is the target observability count constraint. For example, to avoid redundant observations and resource waste of multiple sensors on the same target, the total number of times target k is observed in the same period should not exceed its maximum observable count Cap. target (k), that is: Σ s (x(s,k))≤Cap target (k). In most cases, Cap target (k)=1 is a preferred setting, indicating that a target is observed by at most one sensor. Furthermore, the task assignment optimization model can also include binary constraints where the decision variable x(s,k) is 0 or 1. In some alternative implementations, more complex constraints can be imposed. For example, sensor motion constraints (ensuring that the sensor can reach the observation position within the decision period), energy consumption constraints (ensuring that the total energy consumption of the sensor does not exceed a threshold), or mutual exclusion constraints (e.g., a radar cannot perform a tracking task while performing a search task).

[0056] After construction, the system solves the task assignment optimization model. This model is essentially a generalized allocation problem or an integer programming problem. There are various methods to solve this model. For example, when the constraints and objective function are relatively simple, standard integer programming solvers (such as CPLEX, Gurobi) or branch and bound methods can be used for an exact solution. In some specific simplified cases (e.g., all capacity constraints are 1), the problem can degenerate into a maximum weight matching problem and be solved efficiently using the Hungarian algorithm or the KM algorithm. In cases with very large problem sizes and high real-time requirements, greedy algorithms (e.g., repeatedly selecting the (s, k) combination with the highest current reward and updating the constraints) or metaheuristic algorithms (such as genetic algorithms, simulated annealing) can be used to obtain an approximate optimal solution. The final output of the solution is a deterministic decision variable matrix x(s, k), i.e., the multi-sensor task assignment result, which clarifies which sensor should observe which target.

[0057] Based on the multi-sensor task assignment results, multi-source sensor status data, and multi-sensor fusion target model data, sensor observation paths and timings are planned, and multi-sensor scheduling scheme data is generated.

[0058] In this embodiment, abstract task assignment is transformed into specific execution actions. The system, based on the multi-sensor task assignment result (e.g., x(1,3)=1, indicating that sensor 1 is assigned to observe target 3), combines multi-source sensor state data (such as the current position, velocity, and maneuverability of sensor 1) and multi-sensor fusion target model data (such as the current position and predicted trajectory of target 3) to perform path and timing planning. For fixed sensors (such as fixed photovoltaic power stations), this planning may only involve generating control commands pointing to specific azimuth and pitch angles and setting the observation start time. For mobile sensors (such as UAVs), the planning is more complex, involving calculating the optimal or feasible path from the sensor's current position to the target's observable area (while simultaneously meeting observation distance and angle requirements), while also avoiding obstacle areas and meeting fuel or range constraints. If a mobile sensor is assigned to observe multiple targets, the planning also needs to optimize the target access order (e.g., solving a local traveling salesman problem). After planning is completed, all sensor control commands, flight paths (if applicable), observation times, observation targets, and expected observation modes are encapsulated to form the final, executable multi-sensor scheduling scheme data.

[0059] In an exemplary embodiment, assume there are 2 sensors (s=1, 2) and 3 targets (k=1, 2, 3). The threat reduction benefit assessment data ΔU matrix is: [[0.5, 0.1, 0.8], [0.3, 0.6, 0.2]]. Assume the scheduling constraint parameters are set as follows: the task capacity Cap of sensor 1. sensor (1) = 1; Capacity of sensor 2 sensor (2) = 1. The number of observables (Cap) for each target (T). target(k) = 1 (for k = 1, 2, 3). The objective function of the optimization model is: max(0.5x(1, 1) + 0.1x(1, 2) + 0.8x(1, 3) + 0.3x(2, 1) + 0.6x(2, 2) + 0.2x(2, 3)). The constraints include: x(1, 1) + x(1, 2) + x(1, 3) ≤ 1 (sensor 1 capacity); x(2, 1) + x(2, 2) + x(2, 3) ≤ 1 (sensor 2 capacity); x(1, 1) + x(2, 1) ≤ 1 (target 1 capacity); x(1, 2) + x(2, 2) ≤ 1 (target 2 capacity); x(1, 3) + x(2, 3) ≤ 1 (target 3 capacity); x(s, k) ∈ {0, 1}. By solving this optimization model (the optimal solution can be found through observation), the optimal multi-sensor task assignment results are x(1,3)=1 and x(2,2)=1, with the rest x(s,k)=0. Based on this assignment result (sensor 1 observes target 3, sensor 2 observes target 2), the system plans specific sensor paths (if sensor 1 is a UAV) or directions (if sensor 2 is photoelectric) and timings, generating multi-sensor scheduling scheme data.

[0060] In one embodiment of this application, generating multi-sensor fusion target model data includes:

[0061] Based on single-source target state data and multi-source sensor state data, candidate data for multi-source target matching is constructed, and state consistency cost data and appearance consistency cost data are calculated.

[0062] Specifically, the system acquires the predicted states (such as predicted position and predicted velocity) of all existing tracks from historical multi-sensor fusion target model data. Simultaneously, it acquires single-source target state data (i.e., new observations) for the current time slice. A threshold filtering process is performed; for example, based on the spatial proximity principle, each predicted track is compared with all new observations, eliminating combinations where the predicted position and observation position are too far apart (e.g., the distance exceeds a preset spatial threshold), thereby constructing multi-source target matching candidate data. For each pair (track i, observation j) in this candidate data, the system calculates the traditional association cost, including: state consistency cost data D. state (i, j) is used to reflect the degree of deviation from the continuity of the target's spatial motion. For example, it can be obtained by calculating the Mahalanobis distance between the predicted state (including position and velocity) of trajectory i and the state of observed j, or simplified to the normalized Euclidean distance, such as D. state =||p pred_i -p obs_j || / D norm , where p pred_i and p obs_j These are the predicted location and the observed location, D. normThis is a normalization factor. It also includes: appearance consistency cost data D. app (i, j) is used to reflect the consistency of the appearance features of the same target under different sensor (such as photoelectric, UAV) observations. For example, it can be based on historical appearance features associated with track i (such as aggregated feature vector f). track_i ) and the appearance features of observation j (f obs_j Cosine similarity between (e.g., D) app =1-cos(f track_i f obs_j It can be calculated using either L2 distance or L2 distance.

[0063] Threat consistency cost data is calculated based on multi-source target matching candidate data, single-source target state data, initial target threat assessment data, and historical multi-sensor fusion target model data.

[0064] Specifically, traditional association methods rely solely on state consistency and appearance consistency cost data. Research has found that for a correct target association, its semantic attributes (such as threat level) should also evolve smoothly over time. A seemingly reasonable match in both space and appearance is questionable and needs to be penalized if it causes a track's threat score to jump instantly from low to high threat. Therefore, the system calculates threat consistency cost data D for each candidate match pair (track i, observation j). threat (i, j). Preferably, the system extracts the historical threat score (Threat) of track i at the previous moment from historical multi-sensor fusion target model data. prev (i) Using information about observation j (such as category, velocity, etc.) from single-source target state data and initial target threat assessment data, assume the match is true, and reassess the updated threat score Threat of track i at the current moment. new (i, j). Threat Consistency Cost Data D threat (i, j) is ultimately quantified as a function of the magnitude of change between the historical threat score and the updated threat score, used to penalize candidate matches that cause drastic, non-smooth jumps in the threat score.

[0065] Based on the cost weights in the preset threat consistency constraint parameter data, the state consistency cost data, appearance consistency cost data, and threat consistency cost data are weighted and fused to construct a comprehensive correlation cost.

[0066] In this embodiment, after calculating three independent costs (state, appearance, and threat) separately, the system needs to merge them into a single comprehensive associated cost C. total (i, j) are used for subsequent global optimization. Specifically, three cost weights are read from the preset threat consistency constraint parameter data: state consistency cost weight w. stateAppearance consistency cost weight w app And the weight of the cost of consistency with the threat w threat All three weights are non-negative, and their sum can (but is not required to) be normalized to 1. They reflect the degree of importance attached to different consistency dimensions. For example, in environments where photoelectric appearance features are easily affected by clouds and shadows, w can be appropriately reduced. app At the same time, improve w state and w threat Comprehensive related costs C total (i, j) is calculated as a weighted sum of three costs: C total (i, j) = w state *D state (i, j) + w app *D app (i, j) + w threat *D threat (i, j).

[0067] Global trajectory association optimization is performed based on comprehensive association cost to generate unified target trajectory data.

[0068] Specifically, the system uses a comprehensive correlation cost C total A matrix (with rows and columns representing tracks and observations, respectively) serves as input to the global optimization algorithm. The goal of this optimization is to find track-observation matching schemes that minimize the sum of the overall association costs of all matching pairs. For example, this problem can be modeled as a minimum-weight perfect matching problem in a bipartite graph. Global optimization algorithms include the Hungarian algorithm, the Jonker-Volgenant (JVC) algorithm, or the auction algorithm. The optimization process also handles unmatched observations (typically used to initialize new tracks) and unmatched tracks (increasing their unmatch count; if the number of consecutive unmatches exceeds a preset termination threshold, the track is marked as terminated). The output of the optimization is a well-defined association result, encapsulated as unified target track data, containing a unique identifier for each target track, a time index, sensor source, and a list of associated observations optimized by threat consistency constraints.

[0069] Perform state fusion and threat fusion on unified target trajectory data to generate multi-sensor fused target model data.

[0070] In this embodiment, after determining the correlation (i.e., unified target track data), the system performs information fusion on the multi-source observation sequences associated within each track. This fusion includes two levels: first, state fusion, where the system extracts the corresponding single-source state estimates and sensor observation error characteristics from the single-source target state data, and uses multi-sensor state fusion methods (such as Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), or Particle Filter) to perform fusion calculations on physical states such as position, velocity, and heading, obtaining fused target state time series data and its covariance; second, threat fusion, where the system calculates the fused threat score and fused threat uncertainty considering multiple initial threat scores and threats uncertainties associated with the track, and can combine the fused physical states (e.g., more accurate velocity estimates), forming fused threat time series data. The fused target state time series data and the fused threat time series data are merged and encapsulated to obtain multi-sensor fused target model data. Simultaneously, the fusion results of all on-orbit tracks at the current moment are extracted to generate comprehensive target threat assessment data.

[0071] In one possible implementation, initial target threat assessment data is constructed, including:

[0072] Based on multi-source time-aligned observation data and multi-source sensor state data, target detection, classification, and preliminary state estimation are performed on single-source observations. According to the preset initial threat assessment model, a preliminary threat score and corresponding uncertainty estimate are calculated for each single-source target. The preliminary threat score and the corresponding uncertainty estimate are encapsulated to generate initial target threat assessment data. The corresponding uncertainty estimate is used as the observation threat uncertainty in the step of constructing the uncertainty correction factor.

[0073] Specifically, to achieve uncertainty correction, initial threat quantification is performed in the single-source processing stage. While calculating the initial threat score for each single-source target, the corresponding uncertainty estimate also needs to be calculated. For example, rule-based threat assessment models (e.g., IF Category = Specific Object AND Speed ​​> 60km / h THEN Threat = 0.8) may have inherent uncertainties (rule fuzziness) or input uncertainties (e.g., category identification confidence is only 0.7). These input uncertainties are quantified into the variance of the initial threat score, Var, through uncertainty propagation (e.g., Monte Carlo simulation, Bayesian methods, or first-order linear approximation). threatobs This variance, along with the initial threat score, is encapsulated in the initial target threat assessment data. This variance value (i.e., the observed threat uncertainty) serves as input for subsequently constructing the uncertainty correction factor.

[0074] In another embodiment of this application, calculating threat consistency cost data includes:

[0075] Based on multi-source target matching candidate data and historical multi-sensor fusion target model data, historical threat scores corresponding to the flight track are extracted to form the threat score data of the previous moment.

[0076] In this embodiment, when calculating the cost for a candidate matching pair (track i, observation j), the threat score data from the previous time step is extracted. Specifically, based on the track number i in the multi-source target matching candidate data, the system retrieves the fusion threat score of track i from the historical multi-sensor fusion target model data (i.e., generated in the previous period), denoted as Threat. prev (i). This score is stored in the threat score data of the previous time step as a benchmark for threat smoothness comparison.

[0077] Based on multi-source target matching candidate data, initial target threat assessment data, single-source target status data, and pre-stored unified threat assessment model parameter data, the updated threat score when a candidate match is established is estimated, thus forming updated threat score data.

[0078] In this embodiment, it is assumed that the match (i, j) is valid, and the new threat score of track i resulting from this match is predicted. In a preferred implementation, forming updated threat score data includes: extracting threat-related state information corresponding to the observation from the single-source target state data based on the observation number in the multi-source target matching candidate data. The threat-related state information includes target category, speed, distance, or heading data. Substituting the threat-related state information into the unified threat assessment model defined by the unified threat assessment model parameter data, the threat of the track at the current moment under the scenario where the candidate match is valid is reassessed to obtain updated threat score data. Specifically, the system extracts the threat-related state information corresponding to the observation j from the single-source target state data based on the observation number j in the multi-source target matching candidate data. Threat-related state information is a key physical quantity required for threat assessment, and exemplarily includes target category data (e.g., a specific object with a confidence level of 0.9), target speed data (e.g., 70 km / h), target distance data (e.g., 5 km, indicating that the target has entered the threat zone), and target heading data. The system substitutes this threat-related state information into a pre-defined, more complex unified threat assessment model (defined by the unified threat assessment model parameter data) that is more complex than the initial model, and reassesses the threat of track i at the current moment to obtain an updated threat score, Threat. new (i, j), and generate updated threat score data.

[0079] The threat change magnitude between the updated threat score data and the previous threat score data is calculated and normalized according to the threat score range in the unified threat assessment model parameter data to obtain threat consistency cost data.

[0080] In one possible embodiment, the magnitude of threat change is calculated and normalized to obtain threat consistency cost data, specifically including:

[0081] Calculate the original threat change between the updated threat score data and the threat score data at the previous moment; construct an uncertainty correction factor based on the observed threat uncertainty in the initial target threat assessment data, the historical threat uncertainty in the historical multi-sensor fusion target model data, and the uncertainty weight parameters in the threat consistency constraint parameter data; apply the uncertainty correction factor to the absolute value of the original threat change to obtain the corrected threat change magnitude; normalize the corrected threat change magnitude according to the threat score range to generate threat consistency cost data.

[0082] Specifically, after obtaining the historical threat score from the previous moment... prev (i) and updated threat score Threat new After (i, j), calculate the original threat change ΔThreat. raw (i, j) = Threat new (i,j)-Threat prev (i) If observation j itself has high uncertainty (e.g., Var) threatobs (j) is very high), then the dramatic threat jump it causes (i.e., ΔThreat) raw A significant change in the initial value (e.g., a large change in the initial value) is likely unreliable and should have its weight reduced in related decision-making. This is because the original threat change is not used directly, but rather adjusted for its reliability. Therefore, an uncertainty correction factor, Weight, is constructed. uncertainty (i, j). Preferably, this correction factor is based on: the observation threat uncertainty Var threatobs (j) (from initial target threat assessment data); historical threat uncertainty Var threatprev (i) (threat variance of trajectory i at the previous moment, extracted from historical multi-sensor fusion target model data); preset uncertainty weight parameters α and β (from threat consistency constraint parameter data). An exemplary formula for calculating the uncertainty correction factor is: Weight uncertainty (i, j) = 1 / (1 + α*Var) threatobs (j)+β*Var threatprev (i) It can be seen that when the threat uncertainty of the observation or track is high, the denominator becomes large, and the correction factor approaches 0. The system applies this correction factor to calculate the magnitude of the threat change after correction: ΔThreat adj (i, j) = |ΔThreat raw (i, j)|*Weight uncertainty(i, j). The adjusted threat change magnitude ΔThreat adj (i,j) is normalized to transform it into a form with other costs (D) state D app The comparable cost at a comparable scale. The range of threat scores on which normalization is based (e.g., the maximum threat score). max =1, Minimum Threat Score min =0, then the range is 1.0) can be obtained from the unified threat assessment model parameter data. The final threat consistency cost data D threat (i, j) is calculated as: D threat (i, j) = ΔThreat adj (i, j) / (Threat) max -Threat min ). The D threat The (i, j) values ​​will be used to construct the comprehensive association cost.

[0083] In a specific numerical case, assuming the initial threat score of observation j is 0.9, its corresponding uncertainty estimate (i.e., the observation threat uncertainty) is calculated to be Var. threatobs (j) = 0.5. Assume the historical threat score of track i is Threat. prev (i) = 0.6, and its historical threat uncertainty is Var threatprev (i) = 0.1. Assume the updated threat score obtained from the reassessment (e.g., by substituting into a more refined model) is Threat. new (i, j) = 0.9. Assume uncertainty weighting parameters α = 1, β = 1. Calculate the original threat change: ΔThreat raw =0.9 - 0.6 = 0.3. Calculate the uncertainty correction factor: Weight uncertainty =1 / (1+1*0.5+1*0.1)=1 / 1.6=0.625. Calculate the magnitude of the change in threat after correction: ΔThreat adj =|0.3|*0.625=0.1875. Assuming the threat score range (Threat... max -Threat min =1.0. Calculate the final threat consistency cost: D threat (i, j) = 0.1875 / 1.0 = 0.1875. In contrast, if the uncertainty of observation j' is extremely high, such as Var... threatobs (j')=4.0, even though it leads to the same threat jump (ΔThreat) raw =0.3), its correction factor will become Weight uncertainty=1 / (1+14.0+10.1)=1 / 5.1≈0.196. The corrected threat change is only |0.3|*0.196≈0.0588. By utilizing uncertainty information, the interference of unreliable observations on threat correlation is effectively suppressed.

[0084] In a preferred embodiment, the process of generating uniform target track data includes: constructing an extended integrated correlation cost matrix. The rows of this matrix correspond to all known historical tracks, and the columns correspond to all new observations in the current time slice. The elements C in the matrix... total (i, j) represents the comprehensive association cost of matching track i with observation j. To handle track creation, termination, and missed detections, this matrix is ​​further extended: an option for an unmatched or terminated track is added to each row (track), and a termination track cost parameter (Cost) is set for it. terminate_track Add an option to create a new track for each column (observation), and set a cost parameter (Cost) for creating a new track. new_track These cost parameters can be obtained from the threat consistency constraint parameter data. Minimizing the total cost is the optimization objective, and a global optimal matching solution is performed on the extended integrated association cost matrix. Various algorithms can be used for this solution, such as the Hungarian algorithm, the Jonker-Volgenant (JVC) algorithm, or the auction algorithm. Based on the global optimal matching solution, the track state is updated and unified target track data is generated. Specifically: if track i is matched with observation j, the observation information is appended to the observation list of track i, and a state update is prepared; if track i is matched but not matched, its predicted state is continued, and its unmatched count is updated; if track i is matched but terminated, its state is marked as terminated; if observation j is matched but a new track is created, the system assigns it a new track number and creates a new track.

[0085] In some preferred embodiments, to further improve the smoothness and robustness of the correlation results over threat time series, a posterior correction mechanism is also provided. This mechanism is triggered after the generation of unified target track data and before the execution of state fusion and threat fusion. Specifically:

[0086] Based on unified target trajectory data and historical multi-sensor fusion target model data, threat time series fluctuation measurement data of the trajectory is calculated.

[0087] In this embodiment, the system performs a smoothness check on the unified target track data. It iterates through each on-orbit track and, based on its associated observation list, extracts the threat score sequence for that track within the most recent sliding window (e.g., the last 5 time slices) from historical multi-sensor fusion target model data. Based on this sequence, the system calculates threat time series volatility metrics, such as the variance, maximum adjacent difference, or number of jump points.

[0088] When the time series fluctuation measurement data of the threat exceeds the preset fluctuation index, the suspicious observation associations that cause the threat to change are identified by combining the threat consistency cost data and the unified target track data.

[0089] Specifically, when the threat time series fluctuation metric exceeds a preset fluctuation index (e.g., variance greater than 0.2, or adjacent difference greater than 0.5), the system determines that the threat sequence of that trajectory has an abnormal jump. At this point, it will look up the associated records of that trajectory within the fluctuation window (from unified target trajectory data) and combine this with the threat consistency cost data (D... threat ), identify the suspicious observational correlation that caused the dramatic shift in threat (e.g., D) threat (A very high correlation value).

[0090] Local posterior correction is performed on suspicious observation associations to generate corrected unified target track data; the steps of performing state fusion and threat fusion are as follows: state fusion and threat fusion are performed on the corrected unified target track data.

[0091] In this embodiment, local posterior correction is performed on the association of suspicious observations. This correction is a local, limited-range reassessment. For example, the system may attempt to disconnect the association between the suspicious observation and the track and try to replace it with a candidate match with suboptimal cost (if one exists), or directly mark the observation as unmatched (i.e., treat it as a new track), while marking the original track as unmatched in that time slice. The system reassesses the threat time series volatility index of this locally adjusted track. If the local posterior correction can significantly reduce the threat time series volatility metric data (e.g., reduce variance by more than 50%), and does not significantly increase (or only slightly increase) the overall association cost of that time slice, the system accepts the local adjustment and generates a corrected unified target track data. When adopting this preferred approach, the corrected unified target track data is passed for subsequent state fusion and threat fusion, resulting in higher temporal continuity and reliability of the final output multi-sensor fusion target model data in the threat dimension.

[0092] According to one aspect of this application, calculating threat consistency cost data can also involve: reading multi-source target matching candidate data and historical multi-sensor fusion target model data pre-stored in the system, and parsing out the track number and time index involved in each candidate match. For each track number, using the fused threat time series recorded in the historical multi-sensor fusion target model data, and based on the current time slice index corresponding to the candidate match, the historical threat score (Threat) for the previous moment is retrieved backwards. prev(i) If some tracks did not have a valid threat score in the previous time step, they are completed according to the preset initialization rules in the threat consistency constraint parameter data (e.g., using the most recent valid threat score or using the average threat score of the current time slice). Threat score data for the previous time step is generated, corresponding one-to-one with each candidate matching pair. This data includes the track number and historical threat score. prev (i) Correspondence. Read multi-source target matching candidate data, initial target threat assessment data, and single-source target state data, as well as preset unified threat assessment model parameter data. For each candidate match, based on the observation number in the candidate match, extract threat-related state information such as target category data, target velocity data, target distance data, and target heading data corresponding to the observation from the single-source target state data; simultaneously, extract the initial threat score and initial threat uncertainty data corresponding to the observation from the initial target threat assessment data as a measure of the reliability of the current observation. Substitute the above state information into the unified threat assessment model, and through the calculation processes of type score, velocity score, distance score, and heading score in the model, reassess the threat of the trajectory at the current moment under the scenario where the candidate match is established, and obtain the updated threat score Threat. new (i, j). In this process, the upper limit of the threat score (Threat) in the unified threat assessment model parameter data is... max Threat score lower limit min This data will be read synchronously and used for normalization of subsequent threat change magnitudes. Intermediate updated threat scores will be generated, corresponding one-to-one with each candidate match. This data records the threat score (Threat) at the time the candidate match was successful. new (i, j). Read the threat score data from the previous time step, update the intermediate threat score data, and the threat uncertainty data included in the initial target threat assessment data. Simultaneously, read the uncertainty weight parameters from the threat consistency constraint parameter data. For each candidate match, obtain the historical threat score from the threat score data from the previous time step based on the track number, and obtain the threat uncertainty corresponding to the current observation from the initial target threat assessment data based on the observation number, for example, using the threat score variance Var. Threat_obs (j) or an equivalent uncertainty index; when necessary, the historical threat uncertainty Var of the trajectory at the previous moment can also be obtained from historical multi-sensor fusion target model data. Threat_prev (i), as a supplement. Calculate the original threat change under candidate matching: ΔThreat raw (i, j) = Threat new (i,j)-Threat prev(i) To avoid unreliable observations from excessively influencing the threat change measure, an uncertainty correction factor, Weight, is constructed based on the observed threat uncertainty and the historical threat uncertainty. uncertainty (i, j), for example: Weight uncertainty (i, j) = 1 / (1 + α*Var) threatobs (j)+β*Var threatprev (i)). After obtaining the uncertainty correction factor, calculate the corrected threat change magnitude: ΔThreat adj (i, j) = |ΔThreat raw (i, j)|*Weight uncertainty (i, j). Generate revised threat change magnitude data for each candidate matching pair, which integrates the magnitude of threat change and observation reliability. Read the revised threat change magnitude data and the upper limit of the threat score (Threat) from the unified threat assessment model parameter data. max Threat score lower limit min And based on the normalization rules in the threat consistency constraint parameter data, the corrected threat change magnitude is converted into a dimensionless threat consistency cost. Specifically, for each candidate match, normalization is performed using the following formula: D threat (i, j) = ΔThreat adj (i, j) / (Threat) max -Threat min After normalization, if necessary, the threat consistency cost value that is too small can be truncated to a lower bound based on the threshold in the threat consistency constraint parameter data to suppress numerical noise. However, the truncated cost value is still included as part of the threat consistency cost data. The final output is the threat consistency cost data.

[0093] According to another aspect of this application, generating threat reduction benefit assessment data can also involve: reading target threat uncertainty data and post-observation target threat uncertainty prediction data. For each sensor-target combination, the current threat score variance Var of the target is extracted from the target threat uncertainty data based on the target number. Threat_before (k); Based on the combination of sensor number and target number, extract the predicted threat score variance Var of the target under the observation conditions performed by the sensor from the post-observation target threat uncertainty prediction data. Threat_after(s, k). Through the above indexing and matching operations, the threat uncertainty before and after observation for each sensor and target combination is paired to obtain intermediate threat uncertainty comparison data, which includes target number, sensor number, threat uncertainty before observation, and threat uncertainty after observation. Read the intermediate threat uncertainty comparison data. For each sensor and target combination, based on the recorded threat uncertainty data, calculate the corresponding threat uncertainty reduction: ΔVar Threat (s, k) = Var Threat_before (k)-Var Threat_after (s, k); where the threat score variance Var Threat_before (k) represents the threat uncertainty of target k at the current moment, and the threat score variance Var Threat_after (s, k) represents the predicted threat uncertainty after sensor s performs an observation; both are derived from intermediate threat uncertainty comparison data. To avoid unwanted scenarios where observations lead to increased threat uncertainty and interfere with subsequent optimization, for combinations where the threat uncertainty reduction is less than zero or the absolute value is below a preset threshold, the corresponding threat uncertainty reduction is set to zero based on preset threat reduction benefit truncation parameters. This indicates that the observation action has no positive benefit or negligible benefit in the threat reduction dimension. This truncation process generates threat uncertainty reduction data for each combination, retaining only non-negative, meaningful threat reductions. The threat uncertainty reduction data and preset threat reduction benefit normalization parameters are read. The threat uncertainty reduction for all sensor-target combinations is statistically analyzed to obtain the maximum threat uncertainty reduction (Max) among all combinations within the current decision-making period. Δ and the minimum non-zero threat uncertainty reduction Min Δ This is used for subsequent normalization calculations. For each sensor-target combination, the threat uncertainty reduction ΔVar is calculated. Threat (s, k) is mapped to the normalized threat reduction gain value ΔU in the following form. norm (s, k): ΔU norm (s, k) = {0, if ΔVar} Threat (s, k) = 0; (ΔVar) Threat (s,k)-Min Δ ) / (Max Δ -Min Δ If ΔVar Threat (s, k)>0}; where the normalized threat reduction gain ΔU norm (s, k) lies within the interval [0, 1] and is used to represent the relative effectiveness of the observation action in reducing threat uncertainty compared to other candidate observations. If Max Δ With MinΔ If the values ​​are too close, a simplified normalization method can be used based on the strategy in the threat reduction benefit normalization parameter data, such as directly normalizing to the maximum value, to avoid numerical instability. Read the normalized threat reduction benefit data, as well as the sensor identification information from the target threat uncertainty data and multi-source sensor status data. For each sensor-target combination, associate the normalized threat reduction benefit with the corresponding sensor number and target number to form a threat reduction benefit evaluation matrix data, or an equivalent list structure, with sensors as row indices and targets as column indices. During this process, some unexecutable combinations (e.g., sensors cannot reach the observable area of ​​a target in the current cycle) can be marked based on the scheduling constraint parameter data, and their corresponding threat reduction benefits can be directly set to zero or marked as unavailable, thus ensuring that the threat reduction benefit evaluation data remains consistent with the constraints of the subsequently constructed task assignment optimization model.

[0094] In summary, this application proposes a multimodal target fusion and evaluation method, which includes: aligning and processing multi-source observation data into single-source data. In the track association stage, a threat consistency cost is constructed and weighted with state and appearance costs for global association optimization to ensure the smoothness of threat scoring. In the collaborative scheduling stage, a threat reduction benefit evaluation model is constructed. This model quantifies the expected reduction in threat uncertainty caused by observation actions by fusing state covariance and threat evaluation gradient. An optimization model with the goal of maximizing this benefit is constructed to generate multi-sensor scheduling scheme data. This achieves deep coupling between association and scheduling for the threat assessment task, improving association accuracy and the effectiveness of scheduling decisions.

[0095] This invention mathematically quantifies the threat uncertainty of high-level semantics by combining the target fusion state covariance with the gradient of the threat assessment model, and can proactively predict the reduction effect of different observation actions on this uncertainty. By using this benefit as an optimization objective for collaborative scheduling, sensor resources are prioritized to address the most critical threat perception ambiguity, achieving deep coupling between scheduling decisions and assessment tasks. A threat consistency cost is introduced into the global association optimization. This cost, together with traditional state and appearance costs, constitutes a comprehensive association cost function, forcing association decisions to simultaneously satisfy the smoothness of kinematics and threat assessment over time. This effectively suppresses drastic jumps in threat scores caused by ambiguity or misassociations, ensuring the stability and reliability of the situational assessment results. Furthermore, the introduced uncertainty weighting mechanism avoids excessive interference from unreliable observations on threat continuity, improving the system's robustness.

[0096] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A multi-modal target fusion and evaluation method, characterized in that, The method comprises the following steps: Obtaining multi-source observation data and performing alignment preprocessing to generate multi-source time alignment observation data and multi-source sensor state data; Accordingly, single-source target detection and initial threat estimation are performed to construct single-source target state data and initial target threat evaluation data; Based on the single-source target state data, the initial target threat evaluation data and the multi-source sensor state data, multi-sensor target track association and fusion are performed based on threat consistency constraints to generate multi-sensor fusion target model data; Based on the multi-sensor fusion target model data and the multi-source sensor state data, multi-sensor cooperative scheduling decisions are made based on threat reduction benefit evaluation to determine multi-sensor scheduling scheme data; Determining the multi-sensor scheduling scheme data comprises: Constructing target threat uncertainty data, which describes the uncertainty index of the current target threat score; combining the multi-sensor fusion target model data and the multi-source sensor state data, establishing a sensor observation effect model, predicting the target threat uncertainty under the assumption of observation conditions, and forming observation-after-target threat uncertainty prediction data; Comparing the target threat uncertainty data and the observation-after-target threat uncertainty prediction data, calculating the threat uncertainty reduction, and constructing the threat reduction benefit evaluation data; According to the threat reduction benefit evaluation data, a cooperative scheduling optimization model is constructed to maximize the overall threat uncertainty reduction, and the multi-sensor scheduling scheme data is solved and generated; Generating multi-sensor fusion target model data comprises: Based on the single-source target state data and the multi-source sensor state data, multi-source target matching candidate data is constructed, and state consistency and appearance consistency cost data is calculated; Based on the multi-source target matching candidate data, the single-source target state data, the initial target threat evaluation data and the historical multi-sensor fusion target model data, threat consistency cost data is calculated; According to the cost weight in the preset threat consistency constraint parameter data, the state consistency, appearance consistency and threat consistency cost data are weighted and fused to construct a comprehensive association cost; Based on the comprehensive association cost, global track association optimization is performed to generate unified target track data; State and threat fusion is performed on the unified target track data to generate multi-sensor fusion target model data.

2. The method of claim 1, wherein, Constructing threat reduction benefit evaluation data comprises: Extracting the fusion state covariance in the multi-sensor fusion target model data, and combining the pre-stored threat evaluation model gradient vector to calculate the current threat score variance contained in the target threat uncertainty data; Deriving the predicted state covariance under the assumption of observation conditions; Using the predicted state covariance to replace the fusion state covariance, and reusing the threat evaluation model gradient vector to calculate the observation-after-threat score variance contained in the observation-after-target threat uncertainty prediction data; The difference between the current threat score variance and the observation-after-threat score variance is calculated as the threat uncertainty reduction, and the threat reduction benefit evaluation data is generated.

3. The method of claim 2, wherein, Deriving the predicted state covariance comprises: Based on the multi-source sensor state data and the preset sensor capability model parameter data, the observation reachability of the sensor and the target is analyzed, and the measurement accuracy under the assumed observation condition is predicted, and an observation measurement error covariance matrix is constructed; Combined with the fusion state covariance and the observation measurement error covariance matrix, a state covariance update formula is applied to derive the predicted state covariance.

4. The method of claim 1, wherein, The multi-sensor scheduling scheme data is solved, including: A task assignment optimization model is constructed with the threat uncertainty reduction amount in the threat reduction benefit evaluation data as the objective function weight, and the objective of the task assignment optimization model is to maximize the overall threat uncertainty reduction amount; After applying scheduling constraint parameter data in the task assignment optimization model, the multi-sensor task assignment result is solved; the scheduling constraint parameter data at least includes sensor task capacity constraint and target observable number constraint; Based on the multi-sensor task assignment result, the multi-source sensor state data and the multi-sensor fusion target model data, the sensor observation path and timing are planned, and the multi-sensor scheduling scheme data is generated.

5. The method of claim 2, wherein, The threat uncertainty reduction amount is further included: The difference between the current threat score variance and the threat score variance after observation is subjected to non-negative truncation processing to ensure that the threat uncertainty reduction amount is non-negative.

6. The method of claim 1, wherein, The threat consistency cost data is calculated, including: Based on the multi-source target matching candidate data and the historical multi-sensor fusion target model data, the historical threat score corresponding to the track is extracted to form the threat score data at the last time; Based on the multi-source target matching candidate data, the initial target threat evaluation data, the single-source target state data and the pre-stored unified threat evaluation model parameter data, the updated threat score when the candidate matching is established is estimated to form the updated threat score data; The threat change amplitude between the updated threat score data and the threat score data at the last time is calculated, and is normalized according to the threat score range in the unified threat evaluation model parameter data to obtain the threat consistency cost data.

7. The method of claim 6, wherein, The threat consistency cost data is calculated by calculating the threat change amplitude and normalizing, including: The original threat change amount between the updated threat score data and the threat score data at the last time is calculated; Based on the observation threat uncertainty contained in the initial target threat evaluation data, the historical threat uncertainty contained in the historical multi-sensor fusion target model data, and the uncertainty weight parameter in the threat consistency constraint parameter data, an uncertainty correction factor is constructed; The absolute value of the original threat change amount is weighted and corrected by using the uncertainty correction factor to obtain the corrected threat change amplitude; The corrected threat change amplitude is normalized according to the threat score range to generate the threat consistency cost data.

8. The method of claim 6, wherein, The updated threat score data is formed, including: According to the observation number in the multi-source target matching candidate data, the threat related state information corresponding to the observation is extracted from the single-source target state data, including target category, speed, distance or heading data; The threat-related state information is substituted into a unified threat assessment model defined by unified threat assessment model parameter data, and the threat of the current time of the candidate matching established scene track is re-evaluated to obtain updated threat score data.

Images