Multi-modal target fusion and evaluation method

Abstract

The invention discloses a multi-modal target fusion and evaluation method, which comprises the steps of performing alignment preprocessing on multi-source observation data to generate multi-source time alignment observation data and multi-source sensor state data; executing single-source target detection and initial threat estimation, and constructing single-source target state data and initial target threat evaluation data; based on the single-source target state data, the initial target threat assessment data and the multi-source sensor state data, executing multi-sensor target track association and fusion by applying threat consistency constraint, and generating multi-sensor fusion target model data; and based on the multi-sensor fusion target model data and the multi-source sensor state data, executing a multi-sensor collaborative scheduling decision by applying threat reduction income evaluation, and determining multi-sensor scheduling scheme data. According to the method, deep coupling of association and scheduling on the threat assessment task is realized, and the association accuracy and the scheduling decision effectiveness are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of modern surveillance and command control, and particularly relates to a multi-modal target fusion and evaluation method. BACKGROUND

[0002] In the field of modern surveillance and command control, in the face of increasingly complex electromagnetic and physical environment, how to efficiently fuse the observation data of multi-source heterogeneous sensors such as radar, photoelectricity, and unmanned aerial vehicle load has become the key to improving the situation awareness capability. Accurate and real-time multi-target fusion and threat evaluation is the core prerequisite for realizing autonomous decision-making, guaranteeing the safety of key assets, and optimizing resource allocation, and has important research significance and application value.

[0003] Current multi-sensor data fusion technology mainly focuses on improving the estimation accuracy of target state (such as position and speed). For this purpose, researchers have adopted state estimation algorithms including Kalman Filtering, Particle Filtering, etc. In the aspect of track association, widely used methods include Multiple Hypothesis Tracking (MHT) and Joint Probabilistic Data Association (JPDA), etc., which mainly rely on the matching of kinematic characteristics and appearance features of targets. At the same time, in the field of sensor management, existing scheduling strategies are mostly targeted at maximizing target tracking accuracy or coverage range.

[0004] However, the existing technology still has significant deficiencies in realizing the deep coupling of high-level semantics (such as threat) and low-level data (such as state). Specifically, there are the following technical problems: the scheduling decision of the sensor is decoupled from the threat evaluation task. Existing scheduling models mostly aim to optimize the state estimation accuracy (such as position covariance) of the target, while ignoring the ultimate purpose of scheduling, which is to reduce the uncertainty of threat evaluation. Such state optimization is not threat cognition optimization, resulting in that the sensor resources are not prioritized for resolving the most critical threat uncertainty. In addition, the track association process lacks threat continuity constraints. Traditional association cost only considers motion and appearance, resulting in the separation of association (data layer) and threat evaluation (semantic layer). When target trajectories cross or ambiguity occurs, track mis-association is prone to occur, which makes the target threat score jump unreasonably in the time series, reducing the stability and credibility of the global situation. SUMMARY

[0005] The application aims to provide a multi-modal target fusion and evaluation method to solve the above problems existing in the prior art.

[0006] The technical scheme provides a multi-modal target fusion and evaluation method, which comprises the following steps:

[0007] Obtain multi-source observation data and perform alignment preprocessing to generate multi-source time alignment observation data and multi-source sensor state data; on this basis, perform single-source target detection and initial threat estimation to construct single-source target state data and initial target threat evaluation data;

[0008] Based on the single-source target state data, the initial target threat evaluation data and the multi-source sensor state data, a threat consistency constraint is applied to perform multi-sensor target track association and fusion to generate multi-sensor fusion target model data;

[0009] Based on the multi-sensor fusion target model data and the multi-source sensor state data, a threat reduction benefit evaluation is applied to perform multi-sensor cooperative scheduling decision to determine multi-sensor scheduling scheme data.

[0010] Beneficial effects, the present application realizes the deep coupling of association and scheduling to threat evaluation task, and improves the association accuracy and the effectiveness of scheduling decision. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A step flowchart of a multi-modal target fusion and evaluation method provided for an embodiment of the present application.

[0012] Figure 2 A step flowchart of determining multi-sensor scheduling scheme data provided for an embodiment of the present application.

[0013] Figure 3 A step flowchart of constructing threat reduction benefit evaluation data provided for an embodiment of the present application.

[0014] Figure 4 A step flowchart of deriving predicted state covariance provided for an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below with specific embodiments. It should be noted that the embodiments of the present application can be realized based on various computing environments and system architectures. For example, the methods described in the present application can be completed by a processor in a computer system executing computer executable instructions stored in a memory. The computer system can include but is not limited to a server, a workstation or an embedded system, which includes a processor, a memory, a communication interface and an I / O device. The communication interface is used to communicate with external multi-source perception devices, which exemplarily include radars, fixed photoelectric stations, unmanned aerial loads, etc.

[0016] As shown in Figure 1 A multi-modal target fusion and evaluation method is proposed, which includes the following steps:

[0017] The multi-source observation data is acquired and preprocessed for alignment to generate multi-source time-aligned observation data and multi-source sensor state data.

[0018] In other words, multi-source observation data including radar raw echo data, fixed photoelectric raw image data, and unmanned aerial vehicle payload raw image data is acquired, and the multi-source observation data is preprocessed for alignment and quality assessment to generate multi-source time-aligned observation data and multi-source sensor state data.

[0019] In the present embodiment, the multi-source observation data is sourced from a plurality of preset perception devices, which may, for example, include radar raw echo data, fixed photoelectric raw image data, and unmanned aerial vehicle payload raw image data. The system reads raw observation information from these devices and synchronously acquires sensor position and attitude data and time stamp data corresponding to the observation information. The preprocessing for alignment can solve the inconsistency of different sensor data in the time reference and the spatial coordinate system. Specifically, the preprocessing utilizes preset device calibration parameter data to perform time synchronization, coordinate unification, and quality assessment processing on various observation data. Time synchronization enables all data to be mapped to a unified time axis, and coordinate unification (e.g., conversion to a unified geodetic coordinate system) ensures the comparability of spatial positions. The output multi-source time-aligned observation data and multi-source sensor state data are structured and unified input bases.

[0020] In some optional embodiments, coordinate unification is not limited to conversion to a geodetic coordinate system, but can also be conversion to a specific coordinate system or a relative coordinate system with a certain sensor as a reference. In addition, the quality assessment processing can also include preliminary screening of data integrity, signal-to-noise ratio, or image clarity to eliminate low-quality data.

[0021] Based on the multi-source time-aligned observation data and multi-source sensor state data, single-source target detection and initial threat estimation are performed to construct single-source target state data and initial target threat assessment data.

[0022] In this embodiment, the system conducts independent analysis and processing for multi-source time-aligned observation data by sensor source (e.g. radar, fixed photoelectric, unmanned aerial vehicle). Specifically, the system combines preset radar target detection model parameter data and photoelectric target recognition model parameter data to respectively conduct target detection, target classification and preliminary state estimation on radar observation, fixed photoelectric observation and unmanned aerial vehicle observation. The output of this step is constructed into two types of key data: single-source target state data and initial target threat assessment data. Among them, the single-source target state data is used to describe the physical properties of the target, exemplarily including target position, speed, heading and other motion elements, and target category, recognition confidence and other recognition elements. The initial target threat assessment data calculates the preliminary threat score and the corresponding uncertainty estimate for each detected single-source target according to the preset initial threat assessment model. These two types of data will be the core input for subsequent multi-sensor track association and threat consistency constraint calculation.

[0023] Optionally, the radar target detection model and the photoelectric target recognition model can be deep learning-based models (such as YOLO, FasterR-CNN) or traditional signal / image processing algorithms. The initial threat assessment model at this stage can be a simplified rule model, for example, only based on the category and speed (such as high speed) of the target to give a rough score.

[0024] Based on the single-source target state data, the initial target threat assessment data and the multi-source sensor state data, multi-sensor target track association and fusion are performed by applying threat consistency constraints to generate multi-sensor fusion target model data.

[0025] Specifically, the system reads single-source target state data and initial target threat assessment data, and performs candidate matching on multi-source target observations from radars, fixed photoelectric devices, and unmanned aerial vehicles in a unified time and space coordinate framework. In addition, threat consistency constraints are introduced when performing track association. In other words, when performing global association optimization processing, the system not only comprehensively considers target position and motion consistency, cross-sensor appearance feature similarity, but also considers the smoothness of target threat changes over time, so that the association result remains continuous and consistent on both physical and semantic (threat) levels. Through association optimization, the system obtains unified target track data that is continuous across time and sensors. State fusion and threat fusion are performed on each piece of unified target track data, and multi-sensor fusion target model data is output. Exemplarily, the multi-sensor fusion target model data is a comprehensive model including a target state time series, a threat score time series, and a threat uncertainty time series. Optionally, comprehensive target threat assessment data reflecting the fusion result at the current time can also be generated. In some embodiments, the global association optimization processing can be implemented using the Hungarian algorithm, the Jonker-Volgenant (JVC) algorithm, or multi-hypothesis tracking (MHT), but the threat consistency cost term is included in the cost function of each algorithm.

[0026] Based on the multi-sensor fusion target model data and the multi-source sensor state data, threat reduction benefit assessment is applied to perform multi-sensor cooperative scheduling decision, and multi-sensor scheduling scheme data is determined.

[0027] Unlike traditional scheduling aimed only at improving target state estimation accuracy, the present embodiment aims to maximize the accuracy of threat assessment. Specifically, the system reads multi-sensor fusion target model data and comprehensive target threat assessment data, and combines multi-source sensor state data and a pre-set sensor capability model to evaluate the observation tasks that can be performed between each sensor and each target. Preferably, the threat reduction benefit is quantified: the system evaluates the expected degree of reduction in target threat uncertainty under given observation conditions, which is used as the benefit of the observation task. Based on the constructed threat reduction benefit assessment data, a cooperative scheduling optimization model is further constructed, with the optimization objective being to maximize the overall threat uncertainty reduction. The model is limited by sensor motion constraints, task capacity constraints, and time window constraints when being solved. The model solution obtains the optimal observation object, observation time, and observation path of each sensor, forming executable multi-sensor scheduling scheme data and constructing a closed-loop scheduling capability. Optionally, the cooperative scheduling optimization model can not only maximize the overall benefit, but also minimize the uncertainty of the highest threat target. In addition, the scheduling decision can be executed at fixed time periods or triggered for execution when the overall threat situation changes significantly.

[0028] AsFigure 2 In one possible embodiment, the multi-sensor scheduling scheme data is determined, including:

[0029] The target threat uncertainty data is constructed, which characterizes the uncertainty indicator of the current target threat score.

[0030] In this embodiment, it is not enough to know only the threat score of a target (e.g. 0.8), the system also needs to know the credibility or uncertainty of the score. A target with high threat but high uncertainty should be the priority of sensor scheduling. Optionally, the fusion state estimate and its covariance information in the multi-sensor fusion target model data are utilized, and combined with the current threat score in the comprehensive target threat assessment data, the uncertainty indicator of each target threat score is calculated through a mathematical model. This indicator can be quantified as a scalar value, such as the current threat score variance Var Threat_k The finally generated target threat uncertainty data (e.g. a list containing [target number, threat score, threat score variance]) will be used to evaluate the observation gain.

[0031] The sensor observation effect model is established in combination with the target threat uncertainty data, the multi-sensor fusion target model data and the multi-source sensor state data, the target threat uncertainty after observation is predicted under the assumption of observation conditions, and the post-observation target threat uncertainty prediction data is formed.

[0032] Exemplarily, this embodiment can be used to evaluate: if the system assigns a certain sensor (e.g. UAV A) to observe a certain target (e.g. target k), how much will the threat uncertainty of this target k decrease after the observation is completed. For this purpose, the system needs to evaluate each sensor-target combination. Specifically, based on the multi-source sensor state data (such as the current position and speed of UAV A) and the multi-sensor fusion target model data (such as the current position and speed of target k), the reachability of the sensor to implement effective observation on the target within the current decision period is analyzed. If it is reachable, the expected update effect of the observation action on the target state covariance is derived according to the preset sensor capability model parameter data (e.g. the measurement accuracy of UAV A at a certain distance and angle). Using this expected, more optimal state covariance, the threat uncertainty calculation model is substituted again to obtain the predicted value, i.e. the post-observation threat score variance Var Threat_k_after (s). These predicted values are summarized to form the post-observation target threat uncertainty prediction data.

[0033] Optionally, the sensor observation effect model not only considers the measurement accuracy, but also further considers the information dimension of the observation. For example, radar observation may mainly reduce the position / speed uncertainty of the target, while photoelectric observation may mainly reduce the category uncertainty of the target, and the contributions of the two to the reduction of the final threat uncertainty are different. The contributions of these different dimensions can be measured uniformly through the gradient of the threat assessment model.

[0034] By comparing the target threat uncertainty data with the post-observation target threat uncertainty prediction data, the threat uncertainty reduction amount is calculated, and threat reduction benefit evaluation data is constructed.

[0035] Specifically, for each sensor-target combination, the system extracts the current threat score variance Var Threat_k (from the target threat uncertainty data) and the post-observation threat score variance Var Threat_k_after (s) (from the post-observation target threat uncertainty prediction data). The difference between the two, Var Threat_k -Var Threat_k_after (s), is the threat uncertainty reduction amount ΔU(s, k) brought by the observation task (s, k) in the threat uncertainty dimension. The reduction amount ΔU(s, k) is defined as the threat reduction benefit. The benefit values of all combinations are calculated and encapsulated to form the threat reduction benefit evaluation data, which (for example, the benefit matrix) will be directly input as the objective function of the subsequent scheduling optimization.

[0036] According to the threat reduction benefit evaluation data, a collaborative scheduling optimization model is constructed to maximize the overall threat uncertainty reduction amount, and a multi-sensor scheduling scheme data is solved.

[0037] In other words, according to the threat reduction benefit evaluation data, and in combination with the multi-source sensor state data and the multi-sensor fusion target model data, a collaborative scheduling optimization model is constructed to maximize the overall threat uncertainty reduction amount, and a multi-sensor scheduling scheme data is solved.

[0038] In this embodiment, the system uses threat reduction benefit evaluation data (i.e. benefit matrix ΔU) as the objective function weight of the optimization model. The goal of the optimization is to select a set of sensor-target assignment relationships (denoted by decision variable x(s, k)) to maximize the total amount of threat uncertainty reduction (i.e. max∑[ΔU(s, k)*x(s, k)]). While maximizing the benefit, the optimization model also needs to satisfy a series of realistic constraints, which are derived from multi-source sensor state data (e.g. task capacity of sensor s) and scheduling constraint parameter data (e.g. maximum number of observations acceptable for target k in a period). By solving the optimization model (e.g. using integer programming solver or greedy algorithm, Hungarian algorithm, etc. for approximate solution), the system obtains the optimal task assignment result. Based on the assignment result, and in combination with the sensor maneuvering capability in the multi-source sensor state data and the target position prediction in the multi-sensor fusion target model data, the system plans the specific executable observation path and timing, and forms the final multi-sensor scheduling scheme data for issuing to the sensor control system for execution.

[0039] In a preferred implementation of this embodiment, the quantification of threat uncertainty is achieved through local linear approximation and covariance propagation. Assuming that the threat assessment model (used to calculate threat score Threat) is a function of target state vector X (containing position, velocity, class probability, etc.), i.e. Threat = g(X). In the vicinity of the current fusion state X k , the function can be approximated as a linear function through Taylor expansion of the first order. Calculate the gradient vector of the threat assessment model g(X) with respect to the state vector X, denoted as ▽g k . The gradient vector ▽g k (which has the same dimension as the state vector X) describes the sensitivity of the threat score to the change of each state component (such as velocity, distance), which can be derived from the pre-stored threat assessment model parameter data.

[0040] As shown in Figure 3 , in an exemplary embodiment, the threat reduction benefit evaluation data is constructed, including:

[0041] Extract the fusion state covariance in the multi-sensor fusion target model data, and combine the pre-stored threat assessment model gradient vector to calculate the current threat score variance contained in the target threat uncertainty data.

[0042] Specifically, the fusion state covariance matrix Σ k of target k is extracted from the multi-sensor fusion target model data. Matrix Σ k describes the current uncertainty of the estimate of target state X k . Using the law of covariance propagation, the current threat score variance Var Threat_k(i.e., the core indicator of target threat uncertainty data) is calculated as: Var Threat_k ≈▽g k T *Σ k *▽g k .in T This represents the transpose of a vector / matrix. It can be seen that this formula incorporates the uncertainty (Σ) of the state estimation. k ) and the sensitivity of the threat model to the state (▽g) k This, combined with other methods, quantifies the uncertainty of threat scoring.

[0043] Derive the predicted state covariance under assumed observation conditions.

[0044] In this embodiment, based on the current threat score variance, prediction is performed to derive the predicted state covariance Σ. state_after (s, k). For example... Figure 4 As shown, in a preferred implementation, deriving the predicted state covariance includes: analyzing the observational reachability of the sensor and the target based on multi-source sensor state data and preset sensor capability model parameter data, predicting the measurement accuracy under assumed observation conditions, and constructing an observation measurement error covariance matrix; combining the fused state covariance and observation measurement error covariance matrix, applying the state covariance update formula, and deriving the predicted state covariance. Specifically, based on multi-source sensor state data and sensor capability model parameter data, the observational reachability and observation geometry of sensor s to target k are analyzed. According to the sensor capability model, the observation measurement error covariance matrix R of sensor s under this observation geometry is determined. meas (s, k). Observation measurement error covariance matrix R meas This describes the measurement noise level of the sensor itself. The system applies a state covariance update formula (e.g., the covariance update part of the Kalman update formula), combined with the fused state covariance Σ. k With the observation measurement error covariance matrix R meas (s, k), the predicted state covariance Σ is derived. state_after (s, k). An exemplary update formula is: Σ state_after =(IK gain *H obs )*Σ k Where I is the identity matrix, H obs Let K be the observation matrix. gain For Kalman gain, K gain =Σ k *H obs T *(H obs *Σ k *H obs T+R meas ) -1 .

[0045] The post-observation target threat uncertainty prediction data contains the post-observation threat score variance, which is calculated by replacing the fused state covariance with the predicted state covariance and reusing the threat assessment model gradient vector.

[0046] Specifically, after obtaining the predicted state covariance Σ state_after (s, k), the system reuses the same threat assessment model gradient vector ∇g k , and calculates the post-observation threat score variance: Var Threat_k_after (s) ≈ ∇g k T *Σ state_after (s, k) * ∇g k .

[0047] The difference between the current threat score variance and the post-observation threat score variance is calculated as the threat uncertainty reduction amount, to generate threat reduction reward evaluation data.

[0048] In this embodiment, the difference between the current threat score variance and the post-observation threat score variance is calculated. Further, to ensure the rationality of the reward (avoid negative reward due to model nonlinearity or calculation noise), the threat uncertainty reduction amount is obtained, which also includes: performing non-negative clipping processing on the difference between the current threat score variance and the post-observation threat score variance, to ensure that the threat uncertainty reduction amount is a non-negative value. Specifically, the threat uncertainty reduction amount ΔU(s, k) is calculated as: ΔU(s, k) = max(0, Var Threat_k -Var Threat_k_after (s)).

[0049] In a specific numerical case, assume that the state vector X of target k is only two-dimensional [position; velocity], and its fused state covariance Σ k = [[4, 0], [0, 1]]. Assume that the threat assessment model gradient vector ∇g k = [0.1, 0.5] T (indicating that the threat is more sensitive to the velocity than to the position). The current threat score variance Var Threat_k = [0.1, 0.5] * [[4, 0], [0, 1]] * [0.1, 0.5] T = 0.04 + 0.25 = 0.29. Assume that sensor s can only accurately measure the velocity, and its observation measurement error covariance matrix R meas is very small on the velocity component, resulting in the updated predicted state covariance Σ state_after(s, k) = [[4, 0], [0, 0.2]] (the variance of speed decreases from 1 to 0.2). Calculate the variance of the threat score after observation: Var Threat_k_after (s) = [0.1, 0.5] * [[4, 0], [0, 0.2]] * [0.1, 0.5] T = 0.04 + 0.05 = 0.09. Calculate the threat uncertainty reduction amount: AU(s, k) = max(0, Var Threat_k (s) - Var Threat_k_after (s)) = max(0, 0.29 - 0.09) = 0.20. This 0.20 is the threat reduction benefit of sensor s to target k, which will be filled into the threat reduction benefit evaluation data matrix for subsequent optimization decision-making.

[0050] According to an aspect of the present application, the generated multi-sensor scheduling scheme data is solved, comprising:

[0051] A task assignment optimization model is constructed with the threat uncertainty reduction amount in the threat reduction benefit evaluation data as the objective function weight, and the objective of the task assignment optimization model is to maximize the overall threat uncertainty reduction amount.

[0052] In this embodiment, the system takes the threat reduction benefit evaluation data (the core of which is the benefit matrix AU(s, k)) as the core input. A task assignment optimization model is constructed. The model introduces decision variables x(s, k), which are binary variables, for example, x(s, k) = 1 indicates that the sensor s is assigned to observe the target k in the current decision period, and x(s, k) = 0 indicates that it is not assigned. The objective function of the optimization model is set to maximize the sum of the threat reduction benefits of all selected tasks, which can be mathematically expressed as: max∑ s ∑ k (AU(s, k) * x(s, k)). Where∑ s and∑ k represent the summation of all sensors and all targets, respectively.

[0053] After applying scheduling constraint parameter data in the task assignment optimization model, the multi-sensor task assignment result is obtained; the scheduling constraint parameter data at least includes sensor task capacity constraints and target observable times constraints.

[0054] It can also be said that the scheduling constraint parameter data at least includes sensor task capacity constraints and target observable times constraints is applied in the task assignment optimization model; the multi-sensor task assignment result is obtained by solving the task assignment optimization model after applying the scheduling constraint parameter data.

[0055] Specifically, to make the optimization result consistent with physical and tactical realities, the task assignment optimization model needs to be subject to a series of constraints. Optionally, these constraints are derived from pre-set scheduling constraint parameter data or multi-source sensor state data. Among them, the first type of constraint is the sensor task capacity constraint, for example, for any sensor s, the total number of tasks it performs in the same period cannot exceed its maximum task capacity Cap sensor (s), that is:∑ k (x(s, k))≤Cap sensor (s). Exemplarily, a UAV sensor (sensor s = 1) may have its Cap sensor (1) = 2 (i.e. at most track two targets at the same time). The second type of constraint is the target observable times constraint, for example, in order to avoid redundant observation of the same target by multiple sensors and waste of resources, the total number of times a target k is observed in the same period should not exceed its maximum observable times Cap target (k), that is:∑ s (x(s, k))≤Cap target (k). In most cases, Cap target (k) = 1 is the preferred setting, indicating that a target is at most observed by one sensor. In addition, the task assignment optimization model can also include binary constraints that make the decision variable x(s, k) be 0 or 1. In some optional embodiments, more complex constraints can also be imposed. For example, sensor motion constraints (so that the sensor has the ability to reach the observation position within the decision period), energy consumption constraints (so that the total energy consumption of the sensor does not exceed a threshold), or mutual exclusion constraints (for example, a certain radar cannot perform a search task at the same time as a tracking task).

[0056] After construction, the system solves the task assignment optimization model. The model is essentially a generalized assignment problem or an integer programming problem. There are many ways to solve the model. For example, in the case of relatively simple constraints and objective functions, standard integer programming solvers (such as CPLEX, Gurobi) or branch and bound methods can be used for exact solution. In some specific simplified cases (for example, all capacity constraints are 1), the problem can be degenerated into a maximum weight matching problem, and the Hungarian algorithm or KM algorithm can be used for efficient solution. In the case of very large problem size and high real-time requirement, greedy algorithm (for example, repeatedly selecting the (s, k) combination with the highest current return and updating the constraints) or meta-heuristic algorithm (such as genetic algorithm, simulated annealing) can be used to obtain an approximate optimal solution. The final output of the solution is the determined decision variable matrix x(s, k), i.e. the multi-sensor task assignment result, which clearly shows which sensor should observe which target.

[0057] Based on the multi-sensor task assignment result, multi-source sensor state data and multi-sensor fusion target model data, the sensor observation path and timing are planned, and the multi-sensor scheduling scheme data is generated.

[0058] In this embodiment, the abstract task assignment is converted into specific execution actions. The system plans the path and timing for the multi-sensor task assignment result (e.g., x(1,3)=1, indicating that sensor 1 is assigned to observe target 3), combined with multi-source sensor state data (such as the current position, speed, and maneuvering performance of sensor 1) and multi-sensor fusion target model data (such as the current position and predicted trajectory of target 3). For fixed sensors (such as fixed photoelectric stations), this planning may only involve generating control instructions pointing to a specific azimuth and elevation angle, and setting the observation start time. For mobile sensors (such as unmanned aerial vehicles), the planning is more complex, involving calculating the optimal or feasible path from the current position of the sensor to the target observable area (while meeting the observation distance and angle requirements), while avoiding obstacle areas and meeting fuel or range constraints. If a mobile sensor is assigned to observe multiple targets, the planning also needs to optimize the target access order (e.g., solve the local traveling salesman problem). After planning, the control instructions, flight path (if applicable), observation time, observation target, and expected observation mode of all sensors are encapsulated to form the final multi-sensor scheduling scheme data that can be executed.

[0059] In an exemplary embodiment, assume there are 2 sensors (s=1, 2) and 3 targets (k=1, 2, 3). The threat reduction benefit evaluation data ΔU matrix is: [[0.5, 0.1, 0.8], [0.3, 0.6, 0.2]]. Assume the scheduling constraint parameter data is set as: the task capacity of sensor 1 Cap sensor (1)=1; the task capacity of sensor 2 Cap sensor (2)=1. The T observable times of each target Cap target(k) = 1 (for k = 1, 2, 3). The objective function of the optimization model is: max(0.5x(1,1) + 0.1x(1,2) + 0.8x(1,3) + 0.3x(2,1) + 0.6x(2,2) + 0.2x(2,3)). The constraints include: x(1,1) + x(1,2) + x(1,3) ≤ 1 (sensor 1 capacity); x(2,1) + x(2,2) + x(2,3) ≤ 1 (sensor 2 capacity); x(1,1) + x(2,1) ≤ 1 (target 1 capacity); x(1,2) + x(2,2) ≤ 1 (target 2 capacity); x(1,3) + x(2,3) ≤ 1 (target 3 capacity); x(s,k) ∈ {0,1}. By solving this optimization model (the optimal solution can be found by observation), the optimal multi-sensor task assignment result is x(1,3) = 1 and x(2,2) = 1, and the rest x(s,k) = 0. Based on this assignment result (sensor 1 observes target 3, sensor 2 observes target 2), the system plans the specific sensor path (if sensor 1 is a drone) or pointing (if sensor 2 is photoelectric) and timing, generates multi-sensor scheduling scheme data.

[0060] In an embodiment of the present application, the multi-sensor fusion target model data is generated, including:

[0061] Based on the single-source target state data and the multi-source sensor state data, multi-source target matching candidate data is constructed, and state consistency cost data and appearance consistency cost data are calculated.

[0062] Specifically, the system obtains the predicted state (such as predicted position, predicted speed) of all stored tracks from the historical multi-sensor fusion target model data. At the same time, the single-source target state data (i.e. new observation) of the current time slice is obtained. A threshold screening process is performed, for example, based on the spatial proximity principle, each predicted track is compared with all new observations, and combinations of predicted positions and observed positions that are too far apart (for example, a distance greater than a preset spatial threshold) are eliminated, thereby constructing multi-source target matching candidate data. For each pair (track i, observation j) combination in the candidate data, the system calculates the traditional association cost. Including: state consistency cost data D state (i,j), which reflects the degree of deviation of the target spatial motion continuity, for example, it can be obtained by calculating the Mahalanobis distance between the predicted state (including position, speed) of track i and the state of observation j, or simplified as the normalized Euclidean distance, such as D state =||p pred_i -p obs_j || / D norm , where p pred_i and p obs_j are the predicted position and the observed position, respectively, and D normis a normalization factor. Also included are: appearance consistency cost data D app (i, j) to reflect the consistency degree of the appearance features of the same target under different sensor (e.g. photoelectric, UAV) observations, which can be calculated based on the cosine similarity (e.g. D app = 1 - cos(f track_i , f obs_j )) or L2 distance between the historical appearance features (e.g. aggregated feature vector f track_i ) associated with track i and the appearance features (f obs_j ) of observation j.

[0063] Based on the multi-source target matching candidate data, the single-source target state data, the initial target threat assessment data, and the historical multi-sensor fusion target model data, the threat consistency cost data is calculated.

[0064] Specifically, the traditional association method only relies on the state consistency and appearance consistency cost data. Research has found that the semantic attributes (e.g. threat level) of correct target association should also evolve smoothly over time. If a match that seems reasonable in space and appearance leads to an instant jump in the threat score of a track from low threat to high threat, then the match is suspicious and needs to be penalized. Therefore, the system calculates the threat consistency cost data D threat (i, j) for each candidate matching pair (track i, observation j). Preferably, the system extracts the historical threat score Threat prev (i) of track i at the previous time from the historical multi-sensor fusion target model data; uses the information (e.g. category, speed, etc.) about observation j in the single-source target state data and the initial target threat assessment data to assume that the match is correct, and re-evaluates the updated threat score Threat new (i, j) of track i at the current time. The threat consistency cost data D threat (i, j) is finally quantified as a function of the change amplitude between the historical threat score and the updated threat score, to penalize those candidate matches that lead to a dramatic, non-smooth jump in the threat score.

[0065] According to the cost weights in the preset threat consistency constraint parameter data, the state consistency cost data, the appearance consistency cost data, and the threat consistency cost data are weighted and fused to construct a comprehensive association cost.

[0066] In this embodiment, after the three independent costs (state, appearance, threat) are calculated respectively, the system needs to fuse them into a single comprehensive association cost C total (i, j) for subsequent global optimization. Specifically, three cost weights are read from the preset threat consistency constraint parameter data: the state consistency cost weight w state, appearance consistency cost weight w app , and threat consistency cost weight w threat . All three weights are non-negative, and their sum can (but not must) be normalized to 1. They reflect the importance of different consistency dimensions. For example, in an environment where photo-optical appearance features are easily disturbed by clouds, shadows, etc., w app can be appropriately reduced, while w state and w threat can be increased. The overall association cost C total (i, j) is calculated as the weighted sum of the three costs: C total (i, j) = w state *D state (i, j) + w app *D app (i, j) + w threat *D threat (i, j).

[0067] Perform global track association optimization based on the overall association cost, to generate unified target track data.

[0068] Specifically, the system takes the overall association cost C total matrix (whose rows and columns are tracks and observations, respectively) as the input of a global optimization algorithm. The goal of the optimization is to find a matching scheme of tracks-observations such that the sum of overall association costs of all matched pairs is minimized. Exemplarily, this problem can be modeled as the minimum weight perfect matching problem of a bipartite graph. The global optimization algorithm includes the Hungarian algorithm, the JVC (Jonker-Volgenant) algorithm, or the auction algorithm. The optimization process also needs to handle unmatched observations (which are usually used to initialize new tracks) and unmatched tracks (which increase their unmatched counts, and if the number of consecutive unmatched times exceeds a pre-set termination threshold, the track is marked as terminated). The output of the optimization is the explicit association result, which is encapsulated as unified target track data, containing the unique number, time index, sensor source, and the list of associated observations after threat consistency constraint optimization for each target track.

[0069] Perform state fusion and threat fusion on the unified target track data, to generate multi-sensor fusion target model data.

[0070] In the embodiment, after the association relationship (i.e. unified target track data) is determined, the system performs information fusion on the associated multi-source observation sequence in each track. The fusion includes two levels: one is state fusion, the system extracts the corresponding single-source state estimation and sensor observation error characteristics from the single-source target state data, and adopts a multi-sensor state fusion method (such as extended Kalman filter EKF, unscented Kalman filter UKF or particle filter) to perform fusion calculation on the physical states such as position, velocity and heading, to obtain fused target state time series data and its covariance; the other is threat fusion, the system calculates the fused threat score and fused threat uncertainty based on the multiple initial threat scores and threat uncertainties associated with the track, and can combine the fused physical state (for example, more accurate velocity estimation), to form fused threat time series data. The fused target state time series data and the fused threat time series data are merged and packaged to obtain multi-sensor fused target model data. At the same time, the fusion results of all on-orbit tracks at the current time are extracted to generate comprehensive target threat assessment data.

[0071] In one possible implementation, the initial target threat assessment data is constructed, including:

[0072] Based on the multi-source time-aligned observation data and the multi-source sensor state data, target detection, classification and preliminary state estimation are performed on the single-source observation; the preliminary threat score and the corresponding uncertainty estimate are calculated for each single-source target according to the preset initial threat assessment model; the preliminary threat score and the corresponding uncertainty estimate are packaged to generate the initial target threat assessment data; wherein the corresponding uncertainty estimate is used as the observation threat uncertainty in the step of constructing the uncertainty correction factor.

[0073] Specifically, in order to realize the uncertainty correction, the threat is initially quantified in the single-source processing stage. While calculating the preliminary threat score for each single-source target, the corresponding uncertainty estimate also needs to be calculated. For example, a rule-based threat assessment model (such as IF category = specific object AND velocity > 60km / h THEN threat = 0.8) may have model uncertainty (fuzziness of the rule) or input uncertainty (such as the category recognition confidence is only 0.7). Through uncertainty propagation (such as Monte Carlo simulation, Bayesian method or first-order linear approximation), these input uncertainties are quantified as the variance of the initial threat score, i.e. Var threatobs . The variance is packaged together with the preliminary threat score in the initial target threat assessment data. The variance value (i.e. observation threat uncertainty) is the input for constructing the uncertainty correction factor subsequently.

[0074] In another embodiment of the present application, the threat consistency cost data is calculated, including:

[0075] Based on the multi-source target matching candidate data and the historical multi-sensor fusion target model data, the threat score corresponding to the track in the history is extracted to form the threat score data in the last time.

[0076] In the embodiment, the threat score data in the last time is extracted when the cost of the candidate matching pair (track i, observation j) is calculated. Specifically, the system retrieves the fusion threat score of the track i in the last time from the historical multi-sensor fusion target model data (i.e. generated in the last cycle) according to the track number i in the multi-source target matching candidate data, and records it as Threat prev (i). The score is stored in the threat score data in the last time as a reference for threat smoothing comparison.

[0077] Based on the multi-source target matching candidate data, the initial target threat assessment data, the single-source target state data and the pre-stored unified threat assessment model parameter data, the updated threat score when the candidate matching is established is estimated to form the updated threat score data.

[0078] In the embodiment, it is assumed that the matching (i, j) is established, and the new threat score of the track i caused by the matching is predicted. In a preferred implementation, the updated threat score data is formed, including: extracting the threat-related state information corresponding to the observation from the single-source target state data according to the observation number in the multi-source target matching candidate data, the threat-related state information including target category, speed, distance or heading data; substituting the threat-related state information into the unified threat assessment model defined by the unified threat assessment model parameter data to re-evaluate the threat of the track at the current time in the candidate matching established scenario to obtain the updated threat score data. Specifically, the system extracts the threat-related state information corresponding to the observation j from the single-source target state data according to the observation number j in the multi-source target matching candidate data. The threat-related state information is a key physical quantity required for threat assessment, which exemplarily includes target category data (such as a specific object with a confidence of 0.9), target speed data (such as 70 km / h), target distance data (such as 5 km, indicating that it has entered the threat zone) and target heading data. The system substitutes these threat-related state information into the pre-set unified threat assessment model (defined by the unified threat assessment model parameter data) which is more complex than the initial model to re-evaluate the threat of the track i at the current time to obtain the updated threat score Threat new (i, j) and form the updated threat score data.

[0079] The threat change amplitude between the updated threat score data and the threat score data in the last time is calculated, and is normalized according to the threat score range in the unified threat assessment model parameter data to obtain the threat consistency cost data.

[0080] In one possible embodiment, the threat change amplitude is calculated and normalized to obtain threat consistency cost data, specifically including:

[0081] The original threat change amount between the updated threat score data and the threat score data at the previous time is calculated; an uncertainty correction factor is constructed based on the observation threat uncertainty contained in the initial target threat assessment data, the historical threat uncertainty contained in the historical multi-sensor fusion target model data, and the uncertainty weight parameter in the threat consistency constraint parameter data; the absolute value of the original threat change amount is weighted and corrected by using the uncertainty correction factor to obtain a corrected threat change amplitude; and the corrected threat change amplitude is normalized according to the threat score range to generate threat consistency cost data.

[0082] Specifically, after obtaining the historical threat score Threat prev (i) at the previous time, the updated threat score Threat new (i,j) is calculated. raw (i,j)=Threat new (i,j)-Threat prev (i). If the observation j itself has a high uncertainty (such as Var threatobs (j) is high), then the dramatic threat jump (i.e., ΔThreat raw is large) caused by it is likely to be unreliable and should be reduced in weight in the association decision. Because the original threat change amount is not directly used, but is corrected for reliability. For this purpose, an uncertainty correction factor Weight uncertainty (i,j) is constructed. Preferably, the correction factor is based on: the observation threat uncertainty Var threatobs (j) (from the initial target threat assessment data); the historical threat uncertainty Var threatprev (i) (the threat variance of the track i at the previous time extracted from the historical multi-sensor fusion target model data); and the preset uncertainty weight parameters α and β (from the threat consistency constraint parameter data). An exemplary uncertainty correction factor calculation formula is: Weight uncertainty (i,j)=1 / (1+α*Var threatobs (j)+β*Var threatprev (i)). As can be seen, when the threat uncertainty of the observation or track is high, the denominator becomes large, and the correction factor tends to 0. The system applies the correction factor to calculate the corrected threat change amplitude: ΔThreat adj (i,j)=|ΔThreat raw (i,j)|*Weight uncertainty(i, j). The revised threat change magnitude ΔThreat adj (i, j) is normalized to convert it to a cost value that is comparable to other costs (D state , D app ) in the same scale. The threat score range (e.g., maximum threat score Threat max = 1, minimum threat score Threat min = 0, so the range is 1.0) upon which the normalization is based can be obtained from the unified threat assessment model parameter data. The final threat consistency cost data D threat (i, j) is calculated as: D threat (i, j) = ΔThreat adj (i, j) / (Threat max - Threat min ). This D threat (i, j) value will be used to construct the integrated association cost.

[0083] In one specific numerical case, assume that the preliminary threat score of observation j is 0.9, and its corresponding uncertainty estimate (i.e., observation threat uncertainty) is Var threatobs (j) = 0.5. Assume that the history threat score of track i is Threat prev (i) = 0.6, and its history threat uncertainty is Var threatprev (i) = 0.1. Assume that the updated threat score from reassessment (e.g., plugging in a more refined model) is Threat new (i, j) = 0.9. Assume that the uncertainty weight parameters are a = 1, β = 1. Calculate the original threat change: ΔThreat raw = 0.9 - 0.6 = 0.3. Calculate the uncertainty weight factor: Weight uncertainty = 1 / (1 + 1 * 0.5 + 1 * 0.1) = 1 / 1.6 = 0.625. Calculate the revised threat change magnitude: ΔThreat adj = |0.3| * 0.625 = 0.1875. Assume that the threat score range (Threat max - Threat min ) = 1.0. Calculate the final threat consistency cost: D threat (i, j) = 0.1875 / 1.0 = 0.1875. As a comparison, if the uncertainty of observation j' is extremely high, e.g., Var threatobs (j') = 4.0, even though it results in the same threat jump (ΔThreat raw = 0.3), its weight factor will become Weight uncertainty=1 / (1+14.0+10.1)=1 / 5.1≈0.196. Its revised threat change magnitude is only |0.3|*0.196≈0.0588. With the uncertainty information, the disturbance of unreliable observations on threat association is effectively suppressed.

[0084] In a preferred embodiment, the process of generating the unified target track data comprises constructing an extended comprehensive association cost matrix. The rows of the matrix correspond to all known historical tracks, and the columns correspond to all new observations of the current time slice. The element C total (i,j) in the matrix is the comprehensive association cost of matching track i with observation j. To handle track birth, termination and missed detection, the matrix is further extended: an option of unmatched or terminated track is added to each row (track) with a terminated track cost parameter Cost terminate_track ; and an option of new track is added to each column (observation) with a new track cost parameter Cost new_track . These cost parameters can be obtained from the threat consistency constraint parameter data. The global optimal matching solution is performed on the extended comprehensive association cost matrix, with the total cost minimization as the optimization objective. Various algorithms can be employed for the solution, such as the Hungarian algorithm, the JVC (Jonker-Volgenant) algorithm or the auction algorithm. According to the solution of the global optimal matching, the track states are updated and the unified target track data is generated. Specifically: if track i is matched to observation j, the observation information is appended to the observation list of track i and is ready for state update; if track i is matched as unmatched, only its predicted state is continued and its unmatched count is updated; if track i is matched as terminated, its state is marked as terminated; and if observation j is matched as new track, the system assigns a new track number to it and creates a new track.

[0085] In some preferred embodiments, to further improve the smoothness and robustness of the association result in the threat time series, a posteriori correction mechanism is also provided. The mechanism is triggered after the unified target track data is generated and before the state fusion and threat fusion are performed. Specifically:

[0086] Based on the unified target track data and the historical multi-sensor fusion target model data, the threat time series fluctuation metric data of the tracks are calculated.

[0087] In this embodiment, the system performs smoothness check on the unified target track data. Each on-track track is traversed, and according to its associated observation list, the threat score sequence of the track in the latest sliding window (e.g. the last 5 time slices) is extracted from the historical multi-sensor fusion target model data. Based on the sequence, the system calculates the threat time series fluctuation metric data, such as the variance, the maximum adjacent difference or the number of jump points of the sequence.

[0088] When the threat time series fluctuation metric data exceeds the preset fluctuation index, the suspicious observation association causing the threat mutation is identified in combination with the threat consistency cost data and the unified target track data.

[0089] Specifically, when the threat time series fluctuation metric data exceeds the preset fluctuation index (for example, the variance is greater than 0.2, or the adjacent difference is greater than 0.5), the system determines that the threat sequence of the track has an abnormal jump. At this time, the association record (from the unified target track data) of the track within the fluctuation window is rechecked, and the suspicious observation association (for example, the association with a very high D threat value) causing the threat to jump sharply is identified in combination with the threat consistency cost data (D threat ).

[0090] The suspicious observation association is subjected to local posterior correction to generate corrected unified target track data; wherein the steps of performing state fusion and threat fusion are performed on the corrected unified target track data.

[0091] In this embodiment, the suspicious observation association is subjected to local posterior correction. The correction is a local, limited-range reevaluation. For example, the system can attempt to disconnect the suspicious observation from the association of the track, and attempt to replace it with a suboptimal candidate match (if there is one), or directly mark the observation as unmatched (i.e., as a new track), while marking the original track as unmatched at the time slice. The system reevaluates the threat time series fluctuation index of the track after the local adjustment. If the local posterior correction can significantly reduce the threat time series fluctuation metric data (for example, the variance is reduced by more than 50%) and does not significantly increase (or only slightly increases) the integrated association cost of the time slice, the system accepts the local adjustment and generates a corrected unified target track data. When this preferred scheme is adopted, the corrected unified target track data is passed for subsequent state fusion and threat fusion, so that the final output of the multi-sensor fusion target model data has higher temporal continuity and credibility in the threat dimension.

[0092] According to one aspect of the present application, the threat consistency cost data is calculated, and the multi-source target matching candidate data and the historical multi-sensor fusion target model data pre-stored in the system are also read to parse the track number and time index involved in each candidate match. For each track number, the fusion threat time series recorded in the historical multi-sensor fusion target model data is retrieved to the previous time according to the current time slice index corresponding to the candidate match, to obtain the historical threat score Threat prev(i); if the part track has no valid threat score at the last time instant, fill it according to the initialization rule pre-set in the threat consistency constraint parameter data (e.g. using the last valid threat score or using the average threat score of the current time slice). Generate the last time instant threat score data corresponding to each candidate match pair, which contains the track number and the historical threat score Threat prev (i) of the corresponding relationship. Read the multi-source target matching candidate data, the initial target threat assessment data and the single-source target state data, and the pre-set unified threat assessment model parameter data. For each candidate match pair, extract the threat-related state information of the target category data, target speed data, target distance data and target heading data corresponding to the observation from the single-source target state data according to the observation number in the candidate match; at the same time, extract the initial threat score and initial threat uncertainty data corresponding to the observation from the initial target threat assessment data as a measure of the reliability of the current observation. Substitute the above state information into the unified threat assessment model, and re-evaluate the threat of the track at the current time instant under the candidate match scenario through the type score, speed score, distance score and heading score calculation process in the model, to obtain the updated threat score Threat new (i,j). In this process, the threat score upper limit Threat max and the threat score lower limit Threat min in the unified threat assessment model parameter data will be read synchronously for subsequent normalization processing of the threat change amplitude. Generate the updated threat score intermediate data corresponding to each candidate match pair, which records the threat score Threat new (i,j) when the candidate match is established. Read the last time instant threat score data, the updated threat score intermediate data, and the threat uncertainty data contained in the initial target threat assessment data, and read the uncertainty weight parameter in the threat consistency constraint parameter data at the same time. For each candidate match pair, obtain the historical threat score from the last time instant threat score data according to the track number, and obtain the threat uncertainty corresponding to the current observation from the initial target threat assessment data according to the observation number, for example, using the threat score variance Var Threat_obs (j) or equivalent uncertainty indicator; if necessary, the historical threat uncertainty Var Threat_prev (i) of the last time instant of this track can also be obtained from the historical multi-sensor fusion target model data as a supplement. Calculate the original threat change amount under the candidate match: ΔThreat raw (i,j)=Threat new (i,j)-Threat prev(i). In order to avoid unreliable observations over-influencing the threat change metric, an uncertainty correction factor Weight uncertainty (i,j) is constructed based on the observation threat uncertainty and the historical threat uncertainty uncertainty (i,j) = 1 / (1 + a*Var threatobs (j) + b*Var threatprev (i)). After obtaining the uncertainty correction factor, the corrected threat change magnitude is calculated as: ΔThreat adj (i,j) = |ΔThreat raw (i,j)|*Weight uncertainty (i,j). The corrected threat change magnitude data corresponding to each candidate match pair is generated, which integrates the threat change magnitude and the observation reliability. The corrected threat change magnitude data is read, together with the threat score upper limit Threat max and the threat score lower limit Threat min in the unified threat assessment model parameter data, and the corrected threat change magnitude is converted into a dimensionless threat consistency cost value according to the normalization rule in the threat consistency constraint parameter data. Specifically, for each candidate match pair, the following formula is used for normalization: D threat (i,j) = ΔThreat adj (i,j) / (Threat max -Threat min ). After normalization, if necessary, the threat consistency cost value that is too small can be lower truncated according to the threshold in the threat consistency constraint parameter data to suppress numerical noise, but the truncated value is still part of the threat consistency cost data. The threat consistency cost data is finally output.

[0093] According to another aspect of the present application, generating threat reduction benefit assessment data can also be: reading target threat uncertainty data, and observation target threat uncertainty prediction data. For each sensor-target combination, the current threat score variance Var Threat_before (k) of the target is extracted from the target threat uncertainty data according to the target number; the predicted threat score variance Var Threat_after (k) of the target under the observation condition of the sensor is extracted from the observation target threat uncertainty prediction data according to the combination of the sensor number and the target number.(s, k). By the above index and match operations, the pre-observation threat uncertainty and post-observation threat uncertainty of each sensor-target combination are paired to obtain threat uncertainty comparison intermediate data, which includes target number, sensor number, pre-observation threat uncertainty and post-observation threat uncertainty. The threat uncertainty comparison intermediate data is read. For each sensor-target combination, the threat uncertainty reduction amount corresponding to the combination is calculated according to the threat uncertainty data recorded therein: AVa Threat (s, k) = Var Threat_before (k) - Var Threat_after (s, k); wherein the threat score variance Var Threat_before (k) represents the threat uncertainty of target k at the current time, and the threat score variance Var Threat_after (s, k) represents the predicted threat uncertainty after the observation performed by sensor s, both of which are derived from the threat uncertainty comparison intermediate data. In order to avoid the interference of non-desirable scenarios in which the observation leads to the increase of threat uncertainty on the subsequent optimization, for the combinations whose threat uncertainty reduction amount is less than zero or whose absolute value is lower than a preset threshold, the threat uncertainty reduction amount corresponding thereto is set to zero according to preset threat reduction reward truncation parameter data, indicating that the observation action has no positive reward or negligible reward in the threat reduction dimension. This truncation processing generates threat uncertainty reduction amount data corresponding to each combination, wherein only the non-negative and practically meaningful threat reduction amount is retained. The threat uncertainty reduction amount data and preset threat reduction reward normalization parameter data are read. The threat uncertainty reduction amounts of all sensor-target combinations are counted to obtain the maximum threat uncertainty reduction amount Max Δ and the minimum non-zero threat uncertainty reduction amount Min Δ in all combinations in the current decision cycle, which are used for subsequent normalization calculation. For each sensor-target combination, its threat uncertainty reduction amount AVa Threat (s, k) is mapped to a normalized threat reduction reward value AU norm (s, k) as follows: AU norm (s, k) = {0, if AVa Threat (s, k) = 0; (AVa Threat (s, k) - Min Δ ) / (Max Δ - Min Δ ), if AVa Threat (s, k) > 0}; wherein the normalized threat reduction reward AU norm (s, k) is located in the interval [0, 1], which is used to represent the relative effectiveness of the observation action in terms of threat uncertainty reduction relative to other candidate observations. If Max Δ and MinΔ If too close, a simplified normalization method, such as direct normalization by maximum value, can be used to avoid numerical instability according to the threat reduction benefit normalization parameter data. The normalized threat reduction benefit data, target threat uncertainty data, and sensor identification information in multi-source sensor state data are read. For each sensor-target combination, the normalized threat reduction benefit is associated with the corresponding sensor number and target number to form a threat reduction benefit evaluation matrix data indexed by sensors in rows and targets in columns, or an equivalent list structure. In this process, part of the unexecutable combinations (for example, the sensor cannot reach the observable area of a certain target in the current period) can be marked according to the scheduling constraint parameter data, and the corresponding threat reduction benefit is directly set to zero or marked as unavailable, so as to ensure that the threat reduction benefit evaluation data is consistent with the subsequent task assignment optimization model constraints.

[0094] In summary, the multi-modal target fusion and evaluation method proposed in the application includes: aligning and single-source processing of multi-source observation data. In the track association stage, a threat consistency cost is constructed, and it is weighted and fused with state and appearance costs for global association optimization to ensure the smoothness of threat scoring; in the cooperative scheduling stage, a threat reduction benefit evaluation model is constructed, which quantifies the expected reduction of threat uncertainty by observation action through the fusion of state covariance and threat evaluation gradient; an optimization model with the goal of maximizing the benefit is constructed to generate multi-sensor scheduling scheme data. The association and scheduling are deeply coupled to the threat evaluation task, and the association accuracy and the effectiveness of scheduling decision are improved.

[0095] The application quantifies the threat uncertainty with high-level semantics by combining the target fusion state covariance and the threat evaluation model gradient, and can prospectively predict the reduction effect of different observation actions on the uncertainty. Through cooperative scheduling with this benefit as the optimization goal, the sensor resources are preferentially used to solve the most critical threat recognition ambiguity, realizing the deep coupling of scheduling decision and evaluation task. In the global association optimization, a threat consistency cost is introduced. The cost and the traditional state and appearance costs jointly constitute a comprehensive association cost function, forcing the association decision to simultaneously satisfy the smoothness of kinematics and threat evaluation in time series. The threat score is effectively inhibited from jumping sharply due to ambiguity or misassociation, ensuring the stability and reliability of the situation result. At the same time, the further introduced uncertainty weighting mechanism also avoids excessive interference of unreliable observations on threat continuity, improving the robustness of the system.

[0096] The preferred embodiments of the present application are described in detail above, but the present application is not limited to the specific details of the above-described embodiments, and various equivalent transformations of the technical solutions of the present application can be made within the technical concept of the present application, and these equivalent transformations all belong to the protection scope of the present application.

Claims

1. A multi-modal target fusion and evaluation method, characterized in that, The method comprises the following steps: Obtaining multi-source observation data and performing alignment preprocessing to generate multi-source time alignment observation data and multi-source sensor state data; Accordingly, single-source target detection and initial threat estimation are performed to construct single-source target state data and initial target threat evaluation data; Based on the single-source target state data, the initial target threat evaluation data and the multi-source sensor state data, multi-sensor target track association and fusion are performed based on threat consistency constraints to generate multi-sensor fusion target model data; Based on the multi-sensor fusion target model data and the multi-source sensor state data, multi-sensor cooperative scheduling decisions are made based on threat reduction benefit evaluation to determine multi-sensor scheduling scheme data.

2. The method of claim 1, wherein, Determining the multi-sensor scheduling scheme data comprises: Constructing target threat uncertainty data, which describes the uncertainty index of the current target threat score; combining the multi-sensor fusion target model data and the multi-source sensor state data, establishing a sensor observation effect model, predicting the target threat uncertainty under the assumption of observation conditions, and forming observation-after-target threat uncertainty prediction data; Comparing the target threat uncertainty data and the observation-after-target threat uncertainty prediction data, calculating the threat uncertainty reduction, and constructing the threat reduction benefit evaluation data; According to the threat reduction benefit evaluation data, a cooperative scheduling optimization model is constructed to maximize the overall threat uncertainty reduction, and the multi-sensor scheduling scheme data is solved and generated.

3. The method of claim 2, wherein, Constructing the threat reduction benefit evaluation data comprises: Extracting the fusion state covariance in the multi-sensor fusion target model data, and combining the pre-stored threat evaluation model gradient vector to calculate the current threat score variance contained in the target threat uncertainty data; Deriving the predicted state covariance under the assumption of observation conditions; Using the predicted state covariance to replace the fusion state covariance, and reusing the threat evaluation model gradient vector to calculate the observation-after-threat score variance contained in the observation-after-target threat uncertainty prediction data; The difference between the current threat score variance and the observation-after-threat score variance is calculated as the threat uncertainty reduction, and the threat reduction benefit evaluation data is generated.

4. The method of claim 3, wherein, Deriving the predicted state covariance comprises: Based on the multi-source sensor state data and the preset sensor capability model parameter data, the observation accessibility of the sensor and the target is analyzed, and the measurement accuracy under the assumption of observation conditions is predicted to construct an observation measurement error covariance matrix; Combining the fusion state covariance and the observation measurement error covariance matrix, the state covariance update formula is applied to derive the predicted state covariance.

5. The method of claim 2, wherein, Solving and generating the multi-sensor scheduling scheme data comprises: Constructing a task assignment optimization model with the threat uncertainty reduction in the threat reduction benefit evaluation data as the objective function weight, the objective of the task assignment optimization model being to maximize the overall threat uncertainty reduction; After applying scheduling constraint parameter data to the task assignment optimization model, the multi-sensor task assignment result is obtained; the scheduling constraint parameter data at least includes sensor task capacity constraints and target observable times constraints; Based on the multi-sensor task assignment result, multi-source sensor state data and multi-sensor fusion target model data, a sensor observation path and timing are planned, and multi-sensor scheduling scheme data is generated.

6. The method of claim 3, wherein, The threat uncertainty reduction amount is obtained, and further comprising: The difference between the current threat score variance and the threat score variance after observation is subjected to non-negative truncation processing to ensure that the threat uncertainty reduction amount is non-negative.

7. The method of claim 1, wherein, The multi-sensor fusion target model data is generated, including: Based on single-source target state data and multi-source sensor state data, multi-source target matching candidate data is constructed, and state consistency and appearance consistency cost data are calculated; Based on multi-source target matching candidate data, single-source target state data, initial target threat assessment data and historical multi-sensor fusion target model data, threat consistency cost data is calculated; According to the cost weight in the preset threat consistency constraint parameter data, the state consistency, appearance consistency and threat consistency cost data are weighted and fused to construct a comprehensive correlation cost; Global track association optimization is performed based on the comprehensive correlation cost to generate unified target track data; State and threat fusion is performed on the unified target track data to generate multi-sensor fusion target model data.

8. The method of claim 7, wherein, The threat consistency cost data is calculated, including: Based on multi-source target matching candidate data and historical multi-sensor fusion target model data, the historical threat score corresponding to the track is extracted to form the last time threat score data; Based on multi-source target matching candidate data, initial target threat assessment data, single-source target state data and pre-stored unified threat assessment model parameter data, the updated threat score of the candidate matching is estimated to form the updated threat score data; The threat change amplitude between the updated threat score data and the last time threat score data is calculated, and normalized according to the threat score range in the unified threat assessment model parameter data to obtain the threat consistency cost data.

9. The method of claim 8, wherein, The threat change amplitude is calculated and normalized to obtain the threat consistency cost data, including: The original threat change amount between the updated threat score data and the last time threat score data is calculated; Based on the observation threat uncertainty contained in the initial target threat assessment data, the historical threat uncertainty contained in the historical multi-sensor fusion target model data, and the uncertainty weight parameter in the threat consistency constraint parameter data, an uncertainty correction factor is constructed; The absolute value of the original threat change amount is weighted and corrected by the uncertainty correction factor to obtain the corrected threat change amplitude; The corrected threat change amplitude is normalized according to the threat score range to generate the threat consistency cost data.

10. The method of claim 8, wherein, The updated threat score data is formed, including: According to the observation number in the multi-source target matching candidate data, the threat related state information corresponding to the observation is extracted from the single-source target state data, including target category, speed, distance or heading data; The threat related state information is substituted into the unified threat assessment model defined by the unified threat assessment model parameter data to re-evaluate the threat of the track at the current time under the candidate matching scenario, and the updated threat score data is obtained.

Images