Joint optimization method for multi-passive sensor and multi-target state based on two-factor graph

Abstract

The application belongs to the technical field of multi-target state estimation. The application provides a multi-passive sensor and multi-target state joint optimization method based on a double-factor graph. The disclosure embodiment first constructs a passive detection multi-target measurement solving layer to obtain a multi-target coarse positioning point as a state measurement; then constructs a joint optimization framework containing a multi-target management layer, a multi-target state optimization layer and a multi-sensor state optimization layer, models and fuses the non-Gaussian distribution characteristics of the measurement and the non-Gaussian distribution characteristics of the measurement re-projection error through a three-dimensional Gaussian mixture model and a two-dimensional Gaussian mixture model respectively, and then establishes a double-factor graph based on the three-dimensional and two-dimensional Gaussian mixture models; finally, the sensor attitude state and the target state are jointly optimized and estimated through a sliding window nonlinear least squares, and the optimization result is fed back to the multi-target management layer to form a closed loop update.

Description

Technical Field

[0001] This disclosure relates to the field of multi-objective state estimation technology, and in particular to a method for joint optimization of multiple passive sensors and multiple objective states based on a two-factor graph. Background Technology

[0002] Distributed multi-passive sensor cooperative sensing technology, with its advantages of wide spatial coverage, strong concealment, and outstanding resistance to single-point failures, has been widely applied in key scenarios such as multi-platform cooperative detection and multi-target monitoring in complex environments, becoming a core technical path for achieving high-precision positioning and long-term tracking of multiple targets. This technology overcomes the limitations of incomplete observation by a single passive sensor through the complementarity of multi-sensor, multi-view observations, providing richer geometric constraint information for multi-target state perception, and has significant application value in engineering fields such as civilian low-altitude cooperative monitoring.

[0003] Currently, in distributed multi-passive sensor collaborative sensing scenarios, traditional multi-target localization and tracking methods generally adopt the presupposition of "fixed sensor pose," treating sensor pose as an invariant for localization calculations, without considering the influence of sensor pose deviations and external interference. Affected by factors such as sensor extrinsic parameter calibration deviations, equipment installation errors, and degradation of observation conditions, sensor pose is prone to drift. Traditional methods cannot achieve coordinated correction between sensor pose and target pose, leading to the continuous accumulation of localization errors and the formation of systematic geometric errors. Ultimately, this makes it difficult to maintain localization accuracy and tracking stability in complex scenarios involving multiple targets and long time sequences.

[0004] Meanwhile, existing multi-passive sensor multi-target processing methods have significant shortcomings in modeling the distribution characteristics of fused measurement points (coarse positioning points for multiple targets) and reprojection errors. They often simplify the modeling process by assuming a single Gaussian distribution, neglecting the non-Gaussian distribution characteristics caused by multi-source error coupling in real-world scenarios. This easily leads to model mismatch and further reduces state estimation accuracy. Furthermore, existing technologies lack an effective closed-loop update mechanism. Sensor attitude correction and target state estimation are independent and cannot achieve coordinated convergence through dynamic feedback, resulting in the continuous propagation of systematic geometric errors. This severely restricts the long-term stability and engineering adaptability of multi-target tracking. Existing related patents mainly focus on solving single technical pain points such as multi-target association matching and non-Gaussian measurement modeling, failing to incorporate sensor attitude and target states into the same optimization framework, and lacking effective closed-loop solutions for the problem of systematic geometric error accumulation and propagation. Therefore, they cannot meet the practical engineering requirements of long-term, high-precision multi-target tracking in distributed multi-passive sensor collaborative sensing scenarios.

[0005] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this disclosure is to provide a method for joint optimization of multiple passive sensors and multiple target states based on a two-factor graph, thereby overcoming, to at least some extent, one or more problems caused by the limitations and defects of related technologies.

[0008] According to a first aspect of the present disclosure, a method for joint optimization of multiple passive sensors and multiple target states based on a two-factor graph is provided, comprising:

[0009] Step S1, Passive detection multi-target measurement solution layer solves for multi-target state measurements:

[0010] Based on all passive sensors, a primary sensor and an auxiliary sensor are set up, and a cooperative coordinate system is constructed with the zero-time position of the Earth-centered Earth-fixed system as the origin.

[0011] In the cooperative coordinate system, batch numbers are assigned to all targets by combining the multi-target identity determination results between any main sensor and auxiliary sensor.

[0012] Based on the pixel coordinates of multiple targets of the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined. The direction finding lines of the same batch number are then fused by the least squares method and weighted average to obtain a coarse localization point set of multiple targets, which is used as the state measurement of multiple targets.

[0013] Step S2: The multi-target management layer performs trajectory matching and management.

[0014] Using a coarse set of positioning points for multiple targets as input, the current state measurement of the target is matched with the existing target trajectory through trajectory data association;

[0015] If a match is found, the corresponding target trajectory is updated;

[0016] If a match fails and the number of consecutive occurrences of an unmatched measurement exceeds the preset confirmation count, a new target trajectory will be generated.

[0017] Step S3: Solve for the multi-objective state estimation results in the multi-objective state optimization layer.

[0018] Calculate the observation error based on the coarse location point set of multiple targets;

[0019] Based on the observation error, a three-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the three-dimensional Gaussian mixture model;

[0020] The parameters of the three-dimensional Gaussian mixture model are used as observation factors to construct observation factor nodes, which are then introduced into the multi-objective state optimization factor graph to estimate the multi-objective state estimation results at the current time.

[0021] Step S4: The multi-sensor state optimization layer solves for the multi-sensor state optimization estimation results.

[0022] Calculate the reprojection error based on the coarse location point set of multiple targets;

[0023] Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the two-dimensional Gaussian mixture model;

[0024] The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors to construct reprojection error nodes, which are then introduced into the sensor state optimization factor graph to estimate the multi-sensor state optimization estimation results at the current time.

[0025] Step S5: Perform joint optimization of state estimation using multi-sensor, multi-target factor graphs.

[0026] The target state and the sensor state are jointly optimized using nonlinear least squares optimization within the same sliding window to obtain the joint optimization result.

[0027] Furthermore, step S1 specifically includes:

[0028] Based on all passive sensors, one passive sensor is designated as the master sensor, and the rest of the passive sensors are designated as auxiliary sensors.

[0029] A collaborative coordinate system for distributed passive sensors is constructed with the origin of the coordinate system at the position point where the main sensor is at a height of 0 at time zero.

[0030] The position information, attitude information, and target observation information of each auxiliary sensor are uniformly converted into the cooperative coordinate system;

[0031] The identity of multiple targets in the field of view of different passive sensors is determined, and the one-to-one correspondence between the two-dimensional observation targets of each passive sensor and multiple real targets in space is determined.

[0032] Based on the multi-target identity determination results, the number of the multi-target number in the imaging plane of the main sensor is used as the basis, and the number is used as the unique batch number of the corresponding real target in space.

[0033] For any auxiliary sensor, each target in its imaging plane is assigned the target number of the main sensor associated with it through identity determination, which serves as the batch number of the real spatial target represented by that target.

[0034] By combining the pixel coordinates of multiple targets observed in the two-dimensional imaging plane of any passive sensor, the direction finding lines pointing from the optical centers of different passive sensors to all targets are calculated;

[0035] By combining direction finding lines pointing to the same batch of targets in pairs, and using the least squares method to calculate the midpoint of the common perpendicular of any combination of direction finding lines, the estimated position of the same target can be obtained.

[0036] Multiple estimated location points of the same target are fused to obtain the coarse location point of the target;

[0037] Traverse all two-dimensional observation targets within the main sensor to obtain a coarse set of positioning points for multiple targets.

[0038] Furthermore, step S2 specifically includes:

[0039] Based on the coarse positioning point set of multiple targets and the existing multi-target trajectory set at historical time, calculate the Mahalanobis distance between each state measurement at the current time and the target trajectory at the previous time.

[0040] A correlation similarity matrix is ​​constructed based on Mahalanobis distance to solve the one-to-one matching relationship between the trajectory at the current time and the trajectory at the previous time.

[0041] For successfully matched state measurements, update their corresponding target trajectories;

[0042] For state measurements that fail to match, i.e., unmatched measurements, perform new target determination and initialization:

[0043] Set the preset number of confirmations Continuous tracking and statistics are performed on unmatched measurements. If a certain unmatched measurement is continuously... If a measurement occurs at all observation times and does not match any existing target trajectory each time, it is determined that the unmatched measurement corresponds to a new real spatial target, triggering the new target generation process, completing the initialization of the new target trajectory, and adding the new target trajectory to the existing trajectory set after initialization.

[0044] Furthermore, step S3 specifically includes:

[0045] The deviation between the current state prediction and the coarse location points of multiple targets is calculated to obtain the observation error;

[0046] Based on observation errors, a three-dimensional Gaussian mixture model is constructed using the target optimization EM algorithm, and the parameters of the three-dimensional Gaussian mixture model are obtained by solving the algorithm.

[0047] The parameters of the three-dimensional Gaussian mixture model are used as observation factors, and observation factor nodes are constructed by combining prior factors, motion model factors and mutually exclusive factors.

[0048] The observed factor nodes are introduced into the multi-objective state optimization factor graph, and nonlinear minimization is performed on the multi-objective state optimization factor graph within the current window to estimate the multi-objective state;

[0049] The objective optimization EM algorithm is iterated repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-objective state estimation result at the current time.

[0050] Furthermore, step S4 specifically includes:

[0051] The coarse localization points of multiple targets are reprojected onto the multi-sensor imaging plane to obtain the reprojected points of the multiple targets;

[0052] The reprojection error is obtained based on the deviation between the current multi-target observation point and the corresponding reprojection point;

[0053] Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed using the sensor-optimized EM algorithm, and the parameters of the two-dimensional Gaussian mixture model are obtained by solving the problem.

[0054] The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors, and combined with sensor factors, reprojection error nodes are constructed.

[0055] The reprojection error nodes are introduced into the sensor state optimization factor graph, and nonlinear minimization is performed on the sensor state optimization factor graph in the current window to estimate the sensor state.

[0056] The sensor state EM algorithm iterates repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-sensor state optimization estimation result at the current moment.

[0057] Furthermore, step S5 specifically includes:

[0058] Based on the current multi-objective state estimation results and the current multi-sensor state optimization estimation results, a nonlinear least squares optimization problem is constructed with the objective of minimizing the total error of the correlation factor nodes between all sensor states and target state variables within the sliding window.

[0059] Solve the nonlinear least squares optimization problem to obtain the joint optimized estimate of the state of all targets and all sensor attitude states within the current window;

[0060] The jointly optimized estimated state obtained at each time step is synchronously fed back to the multi-target management layer to form a closed-loop update, thereby realizing the joint optimization of attitude correction of multiple passive sensors and state estimation of multiple targets.

[0061] According to a second aspect of the present disclosure, a multi-passive sensor and multi-target state joint optimization system based on a two-factor graph is provided, comprising:

[0062] The passive detection multi-target measurement solution layer is used to establish a cooperative coordinate system based on all passive sensors, setting the main sensor and auxiliary sensor, with the zero-time position of the Earth-centered Earth-fixed system as the origin. In the cooperative coordinate system, the multi-target identity determination results between any main sensor and auxiliary sensor are combined to assign batch numbers to all targets. Based on the multi-target pixel coordinates of targets with the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined. The direction finding lines of targets with the same batch number are fused by the least squares method and weighted average to obtain a coarse location point set of the multi-target, which is used as the state measurement of the multi-target.

[0063] The multi-target management layer is used to take a coarse set of positioning points of multiple targets as input, and match the current target state measurement with the existing target trajectory through trajectory data association; if the match is successful, the corresponding target trajectory is updated; if the match fails and the number of consecutive occurrences of unmatched measurements exceeds the preset confirmation number, a new target trajectory is generated.

[0064] The multi-objective state optimization layer is used to calculate the observation error based on the coarse location point set of multiple objects; based on the observation error, a three-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the three-dimensional Gaussian mixture model; the parameters of the three-dimensional Gaussian mixture model are used as observation factors to construct observation factor nodes and are introduced into the multi-objective state optimization factor graph to estimate the multi-objective state estimation results at the current time.

[0065] The multi-sensor state optimization layer is used to calculate the reprojection error based on the coarse localization point set of multiple targets. Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the two-dimensional Gaussian mixture model. The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors to construct reprojection error nodes and are introduced into the sensor state optimization factor graph to estimate the multi-sensor state optimization estimation result at the current time.

[0066] The joint optimization layer is used to perform joint nonlinear least squares optimization on the target state and the sensor state within the same sliding window to obtain the joint optimization result.

[0067] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0068] In the embodiments of this disclosure, the above-described joint optimization method for multiple passive sensors and multiple target states based on dual-factor graphs achieves the following: First, it realizes the joint optimization of the attitude of multiple passive sensors and the state of multiple targets. A two-layer optimization structure is used to jointly optimize the target and sensor states, and combined with sliding window nonlinear least squares optimization, the two converge collaboratively, improving positioning accuracy and tracking stability. Second, it accurately models non-Gaussian distribution characteristics, avoiding model mismatch. A three-dimensional Gaussian mixture model is used to model the coarse positioning point three-dimensional measurement error, and a two-dimensional Gaussian mixture model is used to model the reprojection error. Parameters are solved using the expectation-maximization algorithm to fit the actual error distribution, improving state estimation accuracy. Simultaneously, a closed-loop update mechanism is constructed to effectively suppress the cumulative propagation of systematic geometric errors. The joint optimization results are fed back to the multi-target management layer, achieving dynamic collaborative optimization of sensor attitude and target state, ensuring tracking stability and consistency under multiple targets and long-term time sequences. Finally, it realizes full lifecycle management of multi-target trajectories, improving adaptability in complex scenarios. Through the multi-target management layer scheduling trigger mechanism, combined with two-layer optimization and closed-loop updates, it ensures stable system operation in complex scenarios. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0070] Figure 1 This diagram illustrates the steps of a multi-passive sensor and multi-target state joint optimization method based on a two-factor graph in an exemplary embodiment of this disclosure.

[0071] Figure 2 This diagram illustrates a distributed passive sensor cooperative sensing scenario in a cooperative coordinate system according to an exemplary embodiment of this disclosure.

[0072] Figure 3 This diagram illustrates the method and framework for constructing a passive detection multi-target measurement solution layer in an exemplary embodiment of this disclosure.

[0073] Figure 4 This diagram illustrates the method and framework for constructing a multi-objective management layer in an exemplary embodiment of this disclosure.

[0074] Figure 5 This diagram illustrates the structure of a multi-objective state optimization factor graph in an exemplary embodiment of this disclosure.

[0075] Figure 6 This diagram illustrates the method and framework for constructing a multi-objective state optimization layer in an exemplary embodiment of this disclosure.

[0076] Figure 7 This diagram illustrates the reprojection error in an exemplary embodiment of this disclosure.

[0077] Figure 8 This diagram illustrates the structure of the multi-sensor state optimization factor graph in an exemplary embodiment of this disclosure.

[0078] Figure 9 This diagram illustrates the method and framework for constructing a multi-sensor state optimization layer in an exemplary embodiment of this disclosure.

[0079] Figure 10 This diagram illustrates the closed-loop flowchart of the multi-passive sensor and multi-target state joint optimization method based on a two-factor graph in an exemplary embodiment of this disclosure. Detailed Implementation

[0080] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0081] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0082] This example implementation provides a method for joint optimization of multiple passive sensor and multiple target states based on a two-factor graph. (Reference) Figure 1 As shown, the multi-passive sensor and multi-target state joint optimization method based on two-factor graphs may include:

[0083] Step S1, Passive detection multi-target measurement solution layer solves for multi-target state measurements:

[0084] Based on all passive sensors, a primary sensor and an auxiliary sensor are set up, and a cooperative coordinate system is constructed with the zero-time position of the Earth-centered Earth-fixed system as the origin.

[0085] In the cooperative coordinate system, batch numbers are assigned to all targets by combining the multi-target identity determination results between any main sensor and auxiliary sensor.

[0086] Based on the pixel coordinates of multiple targets of the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined. The direction finding lines of the same batch number are then fused by the least squares method and weighted average to obtain a coarse localization point set of multiple targets, which is used as the state measurement of multiple targets.

[0087] Step S2: The multi-target management layer performs trajectory matching and management.

[0088] Using a coarse set of positioning points for multiple targets as input, the current state measurement of the target is matched with the existing target trajectory through trajectory data association;

[0089] If a match is found, the corresponding target trajectory is updated;

[0090] If a match fails and the number of consecutive occurrences of an unmatched measurement exceeds the preset confirmation count, a new target trajectory will be generated.

[0091] Step S3: Solve for the multi-objective state estimation results in the multi-objective state optimization layer.

[0092] Calculate the observation error based on the coarse location point set of multiple targets;

[0093] Based on the observation error, a three-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the three-dimensional Gaussian mixture model;

[0094] The parameters of the three-dimensional Gaussian mixture model are used as observation factors to construct observation factor nodes, which are then introduced into the multi-objective state optimization factor graph to estimate the multi-objective state estimation results at the current time.

[0095] Step S4: The multi-sensor state optimization layer solves for the multi-sensor state optimization estimation results.

[0096] Calculate the reprojection error based on the coarse location point set of multiple targets;

[0097] Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the two-dimensional Gaussian mixture model;

[0098] The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors to construct reprojection error nodes, which are then introduced into the sensor state optimization factor graph to estimate the multi-sensor state optimization estimation results at the current time.

[0099] Step S5: Perform joint optimization of state estimation using multi-sensor, multi-target factor graphs.

[0100] The target state and the sensor state are jointly optimized using nonlinear least squares optimization within the same sliding window to obtain the joint optimization result.

[0101] The aforementioned method for joint optimization of multiple passive sensor and target states based on a two-factor graph achieves the following: First, it realizes the joint optimization of the attitude of multiple passive sensors and the state of multiple targets. A two-layer optimization structure is used to jointly optimize the target and sensor states, combined with sliding window nonlinear least squares optimization to achieve synergistic convergence, improving positioning accuracy and tracking stability. Second, it accurately models non-Gaussian distribution characteristics, avoiding model mismatch. A three-dimensional Gaussian mixture model is used to model the coarse positioning point 3D measurement error, and a two-dimensional Gaussian mixture model is used to model the reprojection error. Parameters are solved using the expectation-maximization algorithm to closely match the actual error distribution, improving state estimation accuracy. Simultaneously, a closed-loop update mechanism is constructed to effectively suppress the cumulative propagation of systematic geometric errors, feeding the joint optimization results back to the multi-target management layer. This enables dynamic collaborative optimization of sensor attitude and target state, ensuring tracking stability and consistency under multiple targets and long-term time sequences. Finally, it achieves full lifecycle management of multi-target trajectories, improving adaptability in complex scenarios. Through a multi-target management layer scheduling trigger mechanism, combined with two-layer optimization and closed-loop updates, it ensures stable system operation in complex scenarios.

[0102] Below, we will refer to Figures 1 to 10 The steps of the above-described method for joint optimization of multiple passive sensors and multiple target states based on two-factor graphs in this example embodiment will be described in more detail.

[0103] In step S1, a passive detection multi-target measurement solution layer is constructed. One passive sensor in the distributed passive sensor system is designated as the master sensor, and its position at time zero in the Earth-centered Earth-fixed system is taken as the origin of the coordinate system. A distributed passive sensor cooperative coordinate system is constructed. In the cooperative coordinate system, the multi-target identity determination results between any two passive sensors are combined to obtain the multi-target distribution batch number. Based on the multi-target pixel coordinates of the two-dimensional imaging plane of any two sensors, the direction finding line pointing from the sensor optical center to the target is calculated. The direction finding lines of the same batch number are combined, and the set of multi-target cross-location points is obtained by least squares method. The set of multi-target coarse location points (fusion measurement points) is obtained by weighted average fusion and used as the multi-target state measurement.

[0104] In step S2, a multi-target management layer is constructed as the scheduling and triggering module of the joint optimization method. Taking the coarse positioning points of the multi-targets output by the passive detection multi-target measurement solution layer as input, the matching relationship between the new target measurement and the existing trajectory at the current moment is determined through data association. If the matching is successful, it is determined that the target state already exists and the corresponding target trajectory is updated. If the matching fails and the number of consecutive occurrences of the unmatched measurement exceeds the preset confirmation number, the new target generation process is triggered to complete the initialization of the new target trajectory, thereby realizing the full life cycle management of the multi-target trajectory and providing clear optimization objects and trigger signals for the subsequent optimization layer.

[0105] In step S3, a multi-target state optimization layer is constructed, and the deviation between the predicted target state value at the current moment and the coarse positioning points of the multi-target is calculated to obtain the observation error. Considering the fusion characteristics of the coarse positioning points of the multi-target, their distribution exhibits non-Gaussian characteristics. A three-dimensional Gaussian mixture model is used to accurately model this non-Gaussian distribution, and the parameters of the three-dimensional Gaussian mixture model are solved by the expectation-maximization algorithm. The solved parameters of the three-dimensional Gaussian mixture model are introduced into the multi-target state optimization factor graph. The variable nodes of this factor graph represent the position and velocity state of the target at each moment. The factor nodes include prior factors, motion model factors, observation factors, and mutually exclusive factors, thereby realizing accurate modeling and optimization preparation of the multi-target state.

[0106] In step S4, a multi-sensor state optimization layer is constructed. Based on the coarse localization points of the multi-target obtained from the passive detection multi-target measurement solution layer, these points are reprojected onto the multi-sensor imaging plane to obtain multi-target reprojection points. The deviation between the current multi-target observation point and the corresponding reprojection point is calculated to obtain the reprojection error. A two-dimensional Gaussian mixture model is used to model the reprojection error. The parameters of the two-dimensional Gaussian mixture model are solved by the expectation-maximization algorithm and introduced into the multi-sensor state optimization factor graph. The variable nodes of this factor graph represent the attitude of the sensors at each time step, and the factor nodes include reprojection factors and sensor factors, thereby realizing accurate modeling and optimization preparation of the multi-sensor attitude state.

[0107] In step S5, a multi-sensor, multi-target factor graph joint optimization state estimation is carried out. A nonlinear least squares optimization problem is constructed with the objective of minimizing the total error of the correlation factor nodes between the attitude states of all sensors and the target state variables within the sliding window. The joint optimization estimation state of all target states and all sensor attitude states within the current window is obtained by solving the problem. Finally, the joint optimization estimation state of the target state and sensor attitude states obtained at each time step is synchronously fed back to the multi-target management layer to form a closed-loop update, thereby achieving the coordinated convergence of sensor attitude correction and target state estimation and suppressing the cumulative propagation of systematic geometric errors.

[0108] In a specific embodiment, this application proposes a joint optimization method for multiple passive sensors and multiple target states based on a two-factor graph. This embodiment combines a typical practical application scenario where three dynamic platforms are equipped with passive sensors respectively and jointly detect four targets. It describes in detail the complete process steps of this application to complete the joint optimization of multiple passive sensors and multiple target states based on a factor graph. The specific process steps are as follows: Figure 1 As shown. Figure 2The schematic diagram of a distributed passive sensor cooperative sensing scenario in a cooperative coordinate system illustrates three mobile platforms, each equipped with one passive sensor. These three platforms collaboratively detect four moving targets within a certain area. For a period of time, all four moving targets are within the common field of view of the three passive sensors. At a certain moment, the four moving targets are observed in the imaging plane of the passive sensors on each mobile platform, and the images are as follows: Figure 2 The target is a two-dimensional observation point occupying several pixels, shown in the lower right corner. To facilitate subsequent calculations, all sensors and targets are placed in a cooperative coordinate system. The origin of this cooperative coordinate system is set as the location of the main sensor in the Earth-centered Earth-fixed system at time zero. The main sensor is a designated passive sensor in the distributed passive sensor system.

[0109] This application assumes that the spatial coordinates of the sensor's centroid or optical center coincide. Furthermore, the number of sensors described in this embodiment is not limited to the following. Figure 2 The scenario shown illustrates three infrared sensors working together to detect four targets, and this can be extended to multiple sensors working together to detect multiple targets. (See attached...) Figure 1 The specific process steps shown are in conjunction with the appendix. Figure 2 The typical scenario shown in this embodiment involves the following steps:

[0110] Step S1: Construct a passive detection multi-target measurement solution layer, designate one passive sensor in the distributed passive sensor system as the master sensor, and use its position at time zero in the Earth-centered Earth-fixed system as the origin of the coordinate system to construct a distributed passive sensor cooperative coordinate system; in the cooperative coordinate system, combine the multi-target identity determination results between any two passive sensors to obtain the multi-target distribution batch number; based on the multi-target pixel coordinates of the two-dimensional imaging plane of any two sensors, calculate the direction finding line from the sensor optical center to the target, combine the direction finding lines of the same batch of targets, and obtain the multi-target cross-location point set by the least squares method, and obtain the multi-target coarse location point set (fusion measurement point) by weighted average fusion, which is used as the multi-target state measurement.

[0111] In a distributed passive sensor system, any one of the passive sensors is designated as the master sensor, denoted as . The remaining sensors are auxiliary sensors, denoted as... Taking the moment the main sensor is powered on as time zero, and the position of the main sensor in the Earth-solid system at that moment as... A distributed passive sensor cooperative coordinate system is constructed with the origin as the coordinate system. This coordinate system follows a right-handed coordinate system, with the X-axis pointing east, the Y-axis pointing north, and the Z-axis perpendicular to the ground and upward. This ensures that the position and attitude of all sensors are described in a unified coordinate system, reducing errors caused by spatial registration. The main sensor is located in the Earth's core and solid system.x axis, y shaft and z The coordinates of the axis.

[0112] To achieve a complete transformation from the Earth-centered Earth-fixed coordinate system to the Northeast-Sky Coordinate System, the origin must first be translated, followed by attitude alignment. The integrated coordinate transformation relationship is shown in the following equation:

[0113]

[0114] in, Auxiliary sensors In the The longitude and latitude coordinates of the time. For any auxiliary sensor The coordinates of the Earth's core and solid system at all times. Let it be its corresponding coordinates in the cooperative coordinate system.

[0115] In the cooperative coordinate system, the result of the multi-target identity determination between any two passive sensors determines the one-to-one correspondence between each two-dimensional observation target and the same real target in space in the observation field of different passive sensors. Through identity determination, the target represented by any two-dimensional observation in the main sensor has a unique association and matching relationship with one and only one two-dimensional observation target in each auxiliary sensor, and there is no one-to-many or many-to-one matching ambiguity.

[0116] Since the focus of this embodiment is not on the determination of the identity of multiple targets, therefore, we assume that the first target within the main sensor is... Two-dimensional observation With the auxiliary sensor Two-dimensional observation Corresponding to the same real-world target in space. Based on the above identity determination results, a unique batch number is assigned to all real-world targets in space using the "master sensor reference number" principle, ensuring that the same spatial target has a unique identifier in the observations of all sensors. The specific assignment rule is: based on the master sensor... Using the multi-target numbering within the imaging plane as a baseline, this number is taken as the unique batch number of the corresponding real-world target in space, denoted as... ,in The total number of targets in space. For any auxiliary sensor, each target in its imaging plane is assigned the same target number as the main sensor target number associated with it through identity determination, which serves as the batch number of the real spatial target represented by that target.

[0117] For example, a two-dimensional observation numbered "Target 1" in the main sensor corresponds to the actual "Target 1" in space, and its batch number is assigned as follows: Therefore, among all auxiliary sensors, those two-dimensional observations that are identified as the same target as "Target 1" by the main sensor are all assigned batch numbers. This batch number distribution method enables unified identification of the same space target in multi-sensor observations, providing a unified basis for subsequent direction finding line combinations and cross-positioning calculations for targets with the same batch number, and ensuring the accuracy of multi-target state measurement solutions.

[0118] Intrinsic parameter matrix based on main sensor and arbitrary auxiliary sensors and attitude matrix Calculate the first image on the two-dimensional imaging plane of the main sensor. Two-dimensional observation With the auxiliary sensor Two-dimensional observation Pointing to the same target The two unit direction-finding vectors and the direction vectors of the two direction-finding lines are calculated as follows:

[0119]

[0120] In the formula, These respectively indicate that the main sensor and auxiliary sensor point to the batch number. The unit direction finding vector of the target. The first on the two-dimensional imaging plane of the main sensor Two-dimensional observation The pixel horizontal and vertical coordinates, Represents the first auxiliary sensor within any auxiliary sensor Two-dimensional observation The pixel horizontal and vertical coordinates.

[0121] The unit vector of the direction finding line pointing to the same target by the combined main sensor and any auxiliary sensor, solution batch number: The target intersection point Let the position of the main sensor in the cooperative coordinate system be... The location of the auxiliary sensor is The parametric equation of the direction finding line is shown in the following equation.

[0122]

[0123] in, , Here, represents the first and second proportionality coefficients of the direction finding lines. Due to errors in actual observation, the direction finding lines cannot intersect perfectly. The least squares method is used to solve the above equations, minimizing the objective function as shown in the following equation, to obtain the intersection points corresponding to this set of direction finding lines. :

[0124]

[0125] Iterate through all combinations of primary and auxiliary sensors to find the set of cross-location points for multiple batches of targets. ,in To assist in determining the number of sensors, for batch number... The objective, which can be solved For each intersection location point, all intersection location points of the same target are fused to obtain the coarse target location point. The fusion formula is as follows:

[0126]

[0127] By traversing the set of all target intersection points, a coarse localization set for multiple targets can be obtained. ,in The number of targets is [number]. At this point, the passive detection multi-target measurement solution layer is complete. The coarse localization points of the multi-targets obtained through the above steps can be used as state measurements for the multi-targets.

[0128] like Figure 3 The diagram shows the construction method and framework of the passive detection multi-target measurement solution layer. It illustrates the complete implementation process of collaborative coordinate system construction, multi-target identity determination and batch number distribution, direction finding line calculation, cross-location point solution and coarse location point fusion in this step. It clarifies the entire process from multi-sensor two-dimensional pixel observation to multi-target coarse location point (fusion measurement point) output in the above steps, and finally completes the solution of multi-target state measurement.

[0129] Step S2: Construct a multi-target management layer as the scheduling and triggering module of the joint optimization method. Take the multi-target coarse positioning points (fused measurement points) output by the passive detection multi-target measurement solution layer as input. Determine the matching relationship between the new target measurement and the existing trajectory at the current moment through data association. If the match is successful, it is determined that the target state already exists and the corresponding target trajectory is updated. If the match fails and the number of consecutive occurrences of the unmatched measurement exceeds the preset confirmation number, the new target generation process is triggered to complete the initialization of the new target trajectory, thereby realizing the full life cycle management of the multi-target trajectory and providing clear optimization objects and trigger signals for the subsequent optimization layer.

[0130] The current output of the passive detection multi-target measurement solution layer Set of coarse localization points for multiple targets at any time As input, it combines existing multi-target trajectory sets from historical moments. ,in Representing history The number of targets at any given time is determined by data correlation to assess the matching relationship between new target measurements and existing trajectories at the current time. The Mahalanobis distance method is used to define... Time of the first Individual target measurement and Time of the first Target trajectory Association weight between for:

[0131]

[0132] in, It is a state The covariance inverse matrix can be used to construct the correlation similarity matrix. The trajectory association matching result is obtained by using the following formula:

[0133]

[0134] in, To establish the associated decision variables, a value of 1 indicates association, and a value of 0 indicates no association.

[0135] It can determine a one-to-one relationship between the trajectory at the current time and the trajectory at the previous time: for trajectories that match successfully in the data association, their corresponding trajectory status measurements are updated. For coarse positioning points (unmatched measurements) that fail to match in the data association, new target determination and initialization are performed. A preset number of confirmations is set. Continuous tracking and statistics are performed on unmatched measurements. If a certain unmatched measurement is continuously... If a measurement occurs at all observation times and does not match any existing trajectory each time, it is determined that the unmatched measurement corresponds to a new real spatial target, triggering the new target generation process, completing the initialization of the new target trajectory, and adding the new target trajectory to the existing trajectory set after initialization.

[0136] In addition, the multi-target management layer is equipped with a closed-loop feedback mechanism. This mechanism receives the joint optimization estimation results of the target state and sensor attitude state output by the subsequent joint optimization layer in real time, feeds the optimized target state back to the trajectory management module, corrects and updates the existing target trajectory, and ensures that the target trajectory can be dynamically adjusted according to the optimization results. This provides a more accurate initial optimization object for the subsequent optimization layer and suppresses the accumulation of systematic geometric errors.

[0137] Appendix Figure 4 A schematic diagram of the construction method and framework for the multi-objective management layer is provided. This layer takes coarse positioning points of multiple objects as input, determines the matching relationship between the current measurement and the existing trajectory through the trajectory data association module, and executes the trajectory management strategy accordingly. It should be noted that the multi-objective tracking layer itself does not directly participate in the numerical optimization of continuous variables. Instead, it controls the dynamic evolution of the subsequent factor graph structure through the matching decision and lifecycle management mechanism of prediction and tracking, thereby ensuring the scalability and stability of joint optimization in multi-objective scenarios.

[0138] The third step involves constructing a multi-target state optimization layer, calculating the deviation between the predicted target state at the current moment and the coarse positioning points (fusion measurement points) of the multiple targets, and obtaining the observation error. Considering the fusion characteristics of the coarse positioning points (fusion measurement points) of the multiple targets, their distribution exhibits non-Gaussian characteristics. A three-dimensional Gaussian mixture model is used to accurately model this non-Gaussian distribution, and the parameters of the three-dimensional Gaussian mixture model are solved by the target optimization expectation maximization algorithm. The solved parameters of the three-dimensional Gaussian mixture model are then introduced into the multi-target state optimization factor graph. The variable nodes of this factor graph represent the position and velocity state of the target at each moment, and the factor nodes include prior factors, motion model factors, observation factors, and mutually exclusive factors.

[0139] In multi-passive sensor collaborative sensing systems, the multi-target fusion measurements output by the passive detection multi-target measurement solution layer often exhibit significant non-unimodal distribution characteristics due to the fusion of observation data from multiple passive sensors. This is mainly due to the heterogeneity of multi-source observations, namely, differences in baseline length, relative viewing angle, and imaging conditions among different UAV platforms, as well as sensor noise, attitude estimation errors, and possible transient false correlations, which lead to multiple structural peaks in the probability space of the fused measurement set. A single Gaussian model is insufficient to fully characterize such complex uncertainties; therefore, a three-dimensional Gaussian mixture model is introduced to accurately model the multimodal distribution of the fused measurements.

[0140] Within the maximum a posteriori estimation framework It is the set of all observations in the time series. It is the optimal estimate of the target's full-state sequence. Assuming that the observation conditions are independent, the posterior probability can be decomposed into the product of all observation condition probabilities, as follows:

[0141]

[0142] in, For the first observation in the observation set One observation sample, For the first There are several state estimation samples. Taking the negative logarithm of the above equation transforms the problem into an optimization problem of minimizing the log-likelihood, as shown in the following equation:

[0143]

[0144] in This is the maximum likelihood estimate of the state, and this expression provides a unified energy minimization formulation for subsequent optimization solutions based on factor graphs. When the observation noise follows a single multidimensional Gaussian distribution, the above equation can be simplified to a standard least squares problem, and the corresponding observation likelihood has the following form:

[0145]

[0146] in For observation error, The square root of the information matrix. This represents the mean of the observation error. However, in multi-UAV observation scenarios, noise distribution often exhibits significant multimodal characteristics, and a single Gaussian model cannot effectively characterize this complex probability structure formed by the superposition of multiple sources of uncertainty. Therefore, the observation probability is modeled as a weighted sum of multiple Gaussian distributions, i.e., a Gaussian mixture model is adopted, as follows:

[0147]

[0148] in, Let be the number of mixture components in the Gaussian mixture model. They represent the first The mean of each Gaussian component and the square root of the information matrix. Indicates the first The system employs a hybrid modeling approach, which involves the mixing weights of Gaussian components and their scaling effects. By using this hybrid modeling method, the system can approximate the complex, multi-modal observation distribution in the real world with a finite number of Gaussian components, allowing each potential observation mode to be expressed in an explicit probabilistic form.

[0149] After establishing the Gaussian mixture observation model, to achieve joint optimization of the position states of multiple targets over time, a factor graph is used as a unified probabilistic graphical modeling framework to construct a multi-target state optimization factor graph. The factor graph can express state variables and various types of constraints in a graphical structure, enabling the unified solution of multi-source observation fusion, motion continuity modeling, and interaction constraints within the same optimization system. The entire state estimation process continuously adds new state nodes and factors within a sliding time window and performs nonlinear minimization on the graph structure within the current window, thereby obtaining a globally consistent and smooth target trajectory. A schematic diagram of the multi-target state optimization factor graph structure is shown below. Figure 5 As shown in the figure, there are two targets. Each target, combined with the state variable node, is the target state variable that needs to be optimized. They are connected by four types of factor nodes. The prior factor is used to ensure the uniqueness of the solution to the optimization problem; the state transition factor is used to describe the transition process of the target position at adjacent time steps, which can be understood as the target's motion model; the observation factor is used to describe the sensor's observation error of the target; and the mutual exclusion factor is used to effectively distinguish two targets that are close to each other.

[0150] The system first processes each timestamp The multi-objective fusion observation set was extracted from the multi-objective management layer and converted into a standardized data format. Subsequently, factor maps were constructed frame-by-frame along the time axis. Let's assume the initial time... The system extracts a set of 3D target observations from the sensor dataset and creates a corresponding state node for each observed target, as follows:

[0151]

[0152] in, For the first The three-dimensional position of the target For its initial velocity estimation, each state node is connected to its corresponding observation through the observation factor, and its residual It can be defined as follows:

[0153]

[0154] in, Indicates the time of the first... Three-dimensional state measurement of a target Indicates the time of the first... The state estimation results This is a three-dimensional identity matrix. To enhance robustness to clustering errors, false associations, and outliers, the observation factors are modeled using a Gaussian mixture model instead of a single Gaussian model, as follows:

[0155]

[0156] in, Represents the probability of observation factor error. This represents the number of components in the Gaussian mixture model. Indicates the first Gaussian mixture model weights, In mixture models, a single Gaussian model follows a Gaussian distribution. Indicates the first The root mean square matrix of the Gaussian mixture model. The above equation characterizes the residual distributions of normal and abnormal observations in a probabilistic sense. This observation factor provides a robust statistical constraint on the initial state nodes; its mean is derived from the fusion of initial observations, and its covariance is determined by measurement noise.

[0157] At any subsequent time The system generates predicted state variable nodes for the current time step based on the number of state variable nodes maintained in the previous time step. :

[0158]

[0159] in, For constant velocity or constant acceleration state transition matrix, The state transition noise is represented by the following residual:

[0160]

[0161] This factor imposes constraints on state changes over time, penalizing state jumps that do not conform to the target's motion patterns, thus ensuring the target trajectory remains continuous and smooth in the time series. Its corresponding quadratic cost is:

[0162]

[0163] in, for Time of the first One target state, for Time of the first One target state, Let be the state transition noise covariance matrix.

[0164] To associate the current observation with the predicted state node, the system calculates the Euclidean distance between the observation and the predicted location and performs explicit data association. For successfully matched observation-state pairs... The system introduces observation factors to update the state estimate using new observations:

[0165]

[0166] in, for Time of the first State measurement of an objective.

[0167] The above formula represents the observation factors established through a three-dimensional Gaussian mixture model. Through the soft allocation mechanism of the three-dimensional Gaussian mixture model, abnormal residuals automatically receive lower weights during optimization, thereby suppressing the biasing effect of erroneous or noisy observations on trajectory estimation. For unmatched observations, the system creates new state nodes and introduces observation factors to achieve dynamic target tracking; while for unmatched state nodes, virtual observations are constructed and weakly constrained observation factors are added to maintain short-term trajectory continuity and prevent immediate trajectory loss due to brief occlusion.

[0168] To avoid trajectory fusion or identity confusion caused by different targets getting too close in the state space, the system performs a check on all state nodes at each time step. Calculate its spatial distance:

[0169]

[0170] in, For the first The goal is Location at any given moment For the first The goal is The position of time, when Less than the threshold When an exclusion factor is introduced, its cost function is defined as:

[0171]

[0172] in, This represents the exclusion factor weight, which guides the optimization process to automatically separate adjacent trajectories by imposing a geometric penalty on target states that are too close, thereby improving the ability to separate multiple targets and suppressing identity switching in the trajectory intersection area. The distance threshold is set. This type of constraint is integrated into the factor graph in a unified optimization form, which is difficult to explicitly model using traditional filtering methods. After constructing all state nodes and factors within the current time window, the system jointly solves the following optimization problem within the sliding window:

[0173]

[0174] in, These represent the prior factor, state transition factor, observation factor, and exclusion factor, respectively. The optimal mean and covariance of all target states within the window are simultaneously estimated through nonlinear minimization. After constructing these factors, the system performs joint optimization on the states within the time window, solving for the optimal mean and covariance of all nodes within the current window. After optimization, the system updates the internal feature variables for each trajectory, including the state mean, velocity estimate, hit count, and time without update, and determines the trajectory's continued validity based on these statistics. When a trajectory has not been observed for a long time, the corresponding state node is marginalized and removed from the factor graph to maintain the sparsity of the graph structure and computational efficiency.

[0175] The entire process proceeds in a sliding window manner. At each moment, the system adds new state nodes and factors while removing states and factors outside the window, thus ensuring that the computational complexity and graph structure size remain stable. Ultimately, this factor graph framework achieves stable, continuous, and globally optimized fusion localization of multiple objectives, and effectively utilizes the synergistic constraints of observation factors, motion factors, and repulsion factors across different dimensions.

[0176] After describing the observation likelihood using a Gaussian mixture model, the state estimation problem can no longer be solved directly using the traditional squared form. Because the logarithm of the likelihood in a three-dimensional Gaussian mixture model includes a summation term, its negative log-likelihood cannot be reduced to a single quadratic error as in a single Gaussian model, and therefore cannot be directly incorporated into the standard Gauss-Newton framework. To address this difficulty, the Expectation-Maximization (EM) method is employed to transform the optimization problem involving mixture likelihoods into a set of solvable weighted squared subproblems, allowing the three-dimensional Gaussian mixture model structure to be naturally solved in the factor graph.

[0177] Let the first Observation The corresponding residual is Its three-dimensional Gaussian mixture model likelihood is as follows:

[0178]

[0179] in, Let be the number of mixture components in the Gaussian mixture model. They represent the first The mean of each Gaussian component and the square root of the information matrix. Indicates the first The mixture weights of the Gaussian components and their scaling effects are considered. Due to the summation of multiple Gaussian components in this formula, its negative logarithmic form cannot be directly expressed as a standard quadratic form, and therefore cannot be directly incorporated into the least squares-based optimization framework. The objective optimization EM algorithm effectively addresses this mathematical difficulty by iteratively solving the maximum likelihood estimate by alternately executing the E-step and M-step. In the E-step, the algorithm estimates based on the current state... Calculate the number of observations belonging to the first... The posterior probability of each Gaussian component, i.e., the latent variable. :

[0180]

[0181] Latent variables It has a clear physical meaning; it quantifies the observation error under the current state estimate. Which noise mode is most likely to generate it? Through this explicit probability partitioning, the objective optimization EM algorithm transforms the complex multimodal likelihood structure into multiple independently weighted simple Gaussian modes. Then, in the M-step, the mixture weights, mean, and covariance are updated based on the latent variables, making the parameters of the 3D Gaussian mixture model better fit the current observation structure. Then, the ... Effective sample size for each component :

[0182]

[0183] in, This represents the total number of three-dimensional state measurements (i.e., multiple targets), and the mixed weights are updated subsequently. mean Covariance matrix and its corresponding information matrix :

[0184]

[0185] Through repeated iterations of the EM model, the mixture distribution adaptively adjusts its shape to accurately cover multiple modes of the actual error distribution. More importantly, the 3D Gaussian mixture model does not forcibly aggregate errors into a single-peak Gaussian, but allows multiple peaks to coexist, thus maintaining stability even when faced with outliers, perspective degradation, and data association errors. However, to integrate the 3D Gaussian mixture model into factor graph optimization, the mixture probability needs to be converted into a continuously differentiable residual form, allowing it to participate as a factor in least squares optimization. This process is achieved using the Sum-Mixture exact modeling method. The mixture probability is normalized as follows:

[0186]

[0187] The normalized mixture probability, and the normalization constant. This ensures the correctness of the probability values. Subsequently, the corresponding optimized residuals are defined. :

[0188]

[0189] This transforms the original maximum likelihood estimation problem into a least squares form, as shown below:

[0190]

[0191] in The significant advantage of Sum-Mixture, which estimates the state by maximum likelihood, lies in its ability to fully preserve multimodal information without simplifying or approximating the mixture distribution. This allows all noise components to participate in state optimization, without overlooking any potential interpretation modes. Furthermore, the residual function is continuous and differentiable, fully compatible with factor graph solvers. This enables the system to maintain high robustness and estimation accuracy when facing dynamically changing noise structures and cross-UAV fusion errors. Figure 6The diagram illustrates the construction method and framework of a multi-target state optimization layer, which combines the target optimization EM algorithm to solve for observation factors and the factor graph to solve for the multi-target position and state estimation results. Specifically, the entire multi-target state optimization layer first calculates the observation error—the deviation between the predicted target state and the 3D observation—based on the target fusion positioning result output by the target fusion measurement layer at the current moment. Considering the possibility of miscorrelation in real-world scenarios, the target optimization EM algorithm is then used to statistically model the observation error, forming a 3D Gaussian mixture model of the observation error. This 3D Gaussian mixture model of the observation error is subsequently introduced into the multi-target state optimization factor graph, acting as observation factors on the target state nodes. The entire multi-target state optimization layer iterates continuously until the target optimization EM algorithm iterates to a set number of iterations or reaches its optimal state. By solving the multi-target state optimization factor graph, the multi-target state estimation results at the current moment can be obtained. Finally, the multi-target state optimization layer outputs the multi-target position and state estimation results.

[0192] The fourth step involves constructing a multi-sensor state optimization layer. Based on the coarse localization points (fused measurement points) of the multi-targets obtained from the passive detection multi-target measurement solution layer, these points are reprojected onto the multi-sensor imaging plane to obtain multi-target reprojection points. The deviation between the current multi-target observation point and the corresponding reprojection point is calculated to obtain the reprojection error. A two-dimensional Gaussian mixture model is used to model the reprojection error, and the parameters of the two-dimensional Gaussian mixture model are solved using the expectation-maximization algorithm. These parameters are then introduced into the multi-sensor state optimization factor graph. The variable nodes of this factor graph represent the sensor attitude at each time step, and the factor nodes include reprojection factors and sensor factors, thus achieving accurate modeling and optimization preparation of the multi-sensor attitude state.

[0193] In multi-UAV collaborative perception scenarios, the three-dimensional spatial position of a target needs to be estimated through multi-view two-dimensional image observation. However, due to the indirect nature of visual observation, the estimation error of the target's three-dimensional position is inevitably affected by factors such as image noise, target detection error, camera calibration error, and viewpoint degradation. Therefore, how to reasonably measure the inconsistency between the three-dimensional position estimation and the actual visual observation, and optimize the target position accordingly, is a key issue in multi-view target localization.

[0194] Therefore, the multi-target observation and localization optimization problem is uniformly formulated as a reprojection error minimization problem. The core idea of ​​this problem is: assuming the target's 3D position estimate is correct, then this position, after being projected onto the image plane through the camera imaging model at each observation viewpoint, should be consistent with the actually observed pixel position. Based on this assumption, the optimized estimation of the target's 3D position can be achieved by minimizing the deviation between the predicted projection and the actual observation, such as... Figure 7The diagram shows the reprojection error. The target fusion measurement point is obtained by fusing two-dimensional observation data (actual observed pixels) from multiple passive sensors. By considering the internal and external participation positions and attitudes of different passive sensors, the reprojected pixels on the imaging planes of passive sensors mounted on different airborne platforms can be calculated. The reprojection error is the pixel difference between the actual observed pixels and the reprojected pixels.

[0195] Let the first The fused measurement value obtained by multi-sensor fusion for each target is This target can be observed from different angles by passive sensors mounted on any airborne platform. The actual pixel coordinates of the target observed by the passive sensor are as follows:

[0196]

[0197] in They represent the first The first goal in The x-coordinate and y-coordinate of pixels in a passive sensor.

[0198] According to the The fused measurement value of the target obtained through multi-sensor fusion In addition to observing the intrinsic and extrinsic parameters of the sensor, the reprojected pixel position of the target in the sensor's imaging plane can be calculated using the sensor's imaging model. The solution method is as follows:

[0199]

[0200] in, Indicates the first The intrinsic parameter matrix of each sensor, They represent the first The attitude and position of each sensor. This represents the sensor imaging model function. The reprojection error can be defined as... Its definition is as follows:

[0201]

[0202] Reprojection error directly reflects the projection matching degree of the current target's 3D position estimate under that viewpoint, and is a fundamental error term for subsequent optimization. In multi-target, multi-view scenarios, reprojection error often exhibits significant non-Gaussian characteristics. On the one hand, visual noise, detection bias, and viewpoint degradation can lead to a long tail in the error distribution; on the other hand, potential mismatched observations can introduce anomalous residuals. To improve the system's robustness to these conditions, instead of using a single Gaussian model to model the reprojection error, a two-dimensional Gaussian mixture model is introduced, and adaptive weight allocation is achieved through the expectation-maximization algorithm. The probability distribution of the reprojection error is modeled as follows:

[0203]

[0204] in, This represents the probability of the reprojection error factor. This represents the number of components in a two-dimensional Gaussian mixture model. Indicates the first Weights of a two-dimensional Gaussian mixture model This indicates that a single Gaussian model in a two-dimensional Gaussian mixture model follows a Gaussian distribution. Indicates the first Mean of a two-dimensional Gaussian mixture model For the first The root mean square matrix of a two-dimensional Gaussian mixture model. In the maximum a posteriori estimation framework, the negative log-likelihood form corresponding to the above equation can be written as:

[0205]

[0206] in This refers to the reprojection error factor established through a two-dimensional Gaussian mixture model. This problem is a typical nonlinear least squares optimization problem, aiming to find the optimal 3D position estimate of the target that can explain all visual observations under multi-view conditions. From an optimization perspective, this problem is equivalent to a structural optimization problem based on reprojection error, that is, estimating the 3D position of the target by minimizing the multi-view reprojection error under known sensor pose conditions, and it can be used as a reprojection factor in subsequent sensor parameter optimization and factor graph optimization frameworks.

[0207] In practical multi-distributed dynamic platform cooperative perception systems, the visual observation model not only depends on the target's 3D position but is also closely related to the pose state of the airborne platform and its onboard sensors. If the sensor pose is directly treated as a precisely known constant, its measurement error will inevitably be transmitted to the target localization result, thus affecting the overall localization accuracy and geometric constraints. Therefore, it is necessary to explicitly introduce the sensor pose state into the set of optimization variables and, by constructing sensor error constraints, jointly estimate and correct the sensor state, thereby achieving cooperative optimization of the target position and sensor state within the same optimization framework.

[0208] For the An airborne sensor, whose state in the world coordinate system is described by both position and attitude, is defined as follows:

[0209]

[0210] in This represents the spatial location of the sensor within the world system, while This represents the coordinate transformation relationship between the sensor coordinate system and the world coordinate system. The introduction of this state variable has two meanings: firstly, it characterizes the sensor's actual motion state in three-dimensional space; secondly, as a key parameter in the visual projection model, it directly participates in the mapping process from the target's three-dimensional position to the image plane. Since even small deviations in the sensor's state directly affect the reprojection error, this modeling method is necessary to allow these small errors in sensor pose to be gradually corrected under visual geometric constraints, thereby improving the accuracy of the overall perception results.

[0211] In practical systems, sensor pose is typically provided by external positioning systems, inertial measurement units (IMUs), and other modules. While these measurement modules can provide relatively accurate pose information, their measurement results are inevitably affected by noise and system errors. Let the... The measurement results of the attitude and position of the airborne sensor are as follows: Sensor errors can be displayed as optimized residuals, position residuals. Defined as:

[0212]

[0213] This residual characterizes the deviation between the actual sensor position and the measurement result. For the attitude error, since the rotation matrix belongs to a non-Euclidean space, Lie algebras are used to model the attitude residual:

[0214]

[0215] in, This represents the transpose matrix of the sensor attitude measurement results. This expression maps the rotation error to the Lie algebraic tangent space, allowing it to participate in subsequent least-squares optimization in a minimal parameter form. In this way, the attitude error can be processed in a unified vector space with the position error. Combining the position residual and the attitude residual yields the sensor error residual vector:

[0216]

[0217] Under weighted least squares, the constraints corresponding to the sensor state can be expressed as the following cost function term:

[0218]

[0219] in This represents the sensor factor. The joint covariance matrix representing sensor errors is defined as:

[0220]

[0221] in, Represents the sensor attitude covariance matrix. This represents the sensor position covariance matrix, where the joint cost term is expressed in relation to the sensor state variables. By imposing constraints, an error constraint model of the sensor state is constructed based on sensor measurements, thereby introducing the sensor pose state as an optimization variable into the multi-objective state estimation optimization problem.

[0222] A multi-sensor state optimization factor map is constructed by combining sensor factors and reprojection error factors. The multi-sensor state optimization factor map is shown below. Figure 5As shown in the figure, there are two sensors. The state variable node of each sensor is the sensor state variable that needs to be optimized, connected by a sensor factor and a reprojection error factor. The reprojection error factor is introduced into the sensor state optimization factor graph to directly connect the target state node and the sensor state node. Since the reprojection error factor depends on both the target state and the sensor state, during the joint optimization process, the reprojection error can be corrected by adjusting the sensor pose, allowing the sensor state to be jointly corrected under multi-target, multi-view geometric constraints. However, relying solely on the visual reprojection error factor constraint may cause the sensor state to drift in scenarios with poor observation conditions. Therefore, a sensor factor is further introduced into the sensor state optimization factor graph to directly apply external sensor measurements (such as inertia or pose priors) to the sensor state nodes. When the visual constraints are sufficient, the sensor state can be fine-tuned near the sensor measurements; when the visual conditions degrade or observations are insufficient, the sensor factor provides a stabilizing constraint for the system, thereby ensuring the robustness and controllability of the joint optimization process. Finally, the joint optimization objective function corresponding to the multi-sensor state optimization layer within the sliding window can be uniformly expressed as:

[0223]

[0224] In the final constructed factor graph structure, the target state and sensor state are directly coupled through reprojection factors, while different targets are indirectly coupled through shared sensor state nodes. Simultaneously, sensor factors uniformly incorporate physical measurement constraints into the sensor state optimization process, enabling the system to handle multi-target geometric constraints, sensor state corrections, and temporal continuity constraints within a unified framework. At each time step, the optimized target state and sensor state are synchronously fed back to the multi-target tracking layer, forming a closed-loop update. Through this joint optimization mechanism, sensor state corrections simultaneously affect the reprojection errors of multiple targets, thereby improving the stability and accuracy of multi-target localization results at the global level.

[0225] like Figure 9The diagram illustrates the construction method and framework of a multi-sensor state optimization layer, which combines the sensor-optimized EM algorithm to solve for the reprojection error factor and the factor map to solve for the multi-sensor state estimation result. Specifically, the method first reprojects the target fusion localization output from the target fusion measurement layer at the current moment onto the imaging planes of each sensor. The reprojection error between the reprojected pixels and the actual observed pixels is calculated. Based on this error, a two-dimensional Gaussian mixture model is constructed using the sensor-optimized EM algorithm. The reprojection factor represented by the two-dimensional Gaussian mixture model is then introduced into the sensor state optimization factor map, along with sensor factors. The entire multi-sensor state optimization layer iterates continuously until the sensor state EM algorithm reaches a set number of iterations or the optimal value. By solving the sensor state optimization factor map, the multi-sensor state optimization estimation result at the current moment can be obtained. Finally, the multi-sensor state optimization layer outputs the multi-sensor pose state optimization result.

[0226] The fifth step involves conducting joint optimization state estimation using multi-sensor, multi-target factor graphs. This involves constructing a nonlinear least squares optimization problem with the objective of minimizing the total error of the factor nodes associated with the states of all sensors and the target state variables within a sliding window. The solution yields the joint optimization estimate of the states of all targets and the attitude states of all sensors within the current window. Finally, the optimized joint optimization estimate of the target state and sensor attitude states obtained at each time step is synchronously fed back to the multi-target management layer, forming a closed-loop update and achieving coordinated convergence of sensor attitude correction and target state estimation.

[0227] like Figure 10 The diagram shows the closed-loop flowchart of the multi-passive sensor and multi-target state joint optimization method based on two-factor graphs. The framework can be structurally divided into four mutually coupled layers. The passive detection multi-target measurement solution layer combines multi-sensor passive detection with pose input to solve multi-target fusion measurements and serves as input to the multi-target management layer. The multi-target management layer is responsible for discrete-level target management and data association. The multi-target state optimization layer is responsible for time-series modeling of the target's 3D state. The multi-sensor state optimization layer achieves joint correction of sensor states through reprojection consistency and sensor constraints.

[0228] The passive detection multi-target measurement solution layer is the outermost layer of the overall framework, used to solve for the system's original input. This layer takes passive detection and pose from multiple sensors as input, and combines multi-target identity determination and numbering to achieve cross-localization and fusion of multiple targets, thereby solving for the fused measurement results of multiple targets (coarse localization points of multiple targets).

[0229] The multi-objective management layer is the scheduling and triggering module of the joint optimization process. It takes multi-objective fusion measurement results as input, determines the matching relationship between the current observation and existing trajectories through the data association module, and executes trajectory management strategies accordingly. When a new target observation arrives, the multi-objective management layer first completes the data association judgment and decides on subsequent operations based on whether it matches an existing trajectory: if the match is successful, the target state is considered to exist, and only the corresponding trajectory needs to be updated; if the match fails, and if it is a new trajectory and the confirmation count has exceeded the limit, a new target generation is triggered, and new state nodes and related factors are added to the factor graph. It should be noted that the multi-objective management layer itself does not directly participate in the numerical optimization of continuous variables, but rather controls the dynamic evolution of the factor graph structure through prediction and tracking matching decisions and lifecycle management mechanisms, thereby ensuring the scalability and stability of joint optimization in multi-objective scenarios.

[0230] The core task of the multi-objective state optimization layer is to model the continuity and stability of the target state over time. In this layer, the system first calculates the observation error—the deviation between the predicted target state and the 3D observation—based on the target fusion measurement results at the current moment. Then, the EM algorithm is used to statistically model the observation error, forming a Gaussian mixture model (GMM). This GMM is subsequently introduced into the target state optimization factor graph, acting as observation factors on the target state nodes. By introducing the Gaussian mixture model, the system can adaptively distinguish between normal and abnormal observations during the optimization process, significantly improving the robustness of target state estimation in complex environments. Furthermore, target states at adjacent times are connected through motion factors to characterize the continuous evolution of the target over time. By jointly solving the target state factor graph, the target state estimation result at the current moment can be obtained and fed back to the multi-objective management layer as a tracking result.

[0231] The multi-sensor state optimization layer is a key component of the joint optimization framework. Its core objective is to achieve joint optimization by fusing visual and sensor information to correct sensor states. Within this framework, the system maintains two core state variables: target state, representing the 3D position of a target in the world coordinate system at a given moment; and sensor state, representing the position and attitude of the sensors at a given moment. The goal of joint optimization is to jointly estimate the states of multiple targets and multiple sensors within a sliding time window, minimizing the overall residual under the constraints of the motion model, observation model, and sensor settings.

[0232] Furthermore, this example embodiment also provides a joint optimization system for multiple passive sensors and multiple target states based on a two-factor graph. This system may include:

[0233] The passive detection multi-target measurement solution layer is used to establish a cooperative coordinate system based on all passive sensors, setting the main sensor and auxiliary sensor, with the zero-time position of the Earth-centered Earth-fixed system as the origin. In the cooperative coordinate system, the multi-target identity determination results between any main sensor and auxiliary sensor are combined to assign batch numbers to all targets. Based on the multi-target pixel coordinates of targets with the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined. The direction finding lines of targets with the same batch number are fused by the least squares method and weighted average to obtain a coarse location point set of the multi-target, which is used as the state measurement of the multi-target.

[0234] The multi-target management layer is used to take a coarse set of positioning points of multiple targets as input, and match the current target state measurement with the existing target trajectory through trajectory data association; if the match is successful, the corresponding target trajectory is updated; if the match fails and the number of consecutive occurrences of unmatched measurements exceeds the preset confirmation number, a new target trajectory is generated.

[0235] The multi-objective state optimization layer is used to calculate the observation error based on the coarse location point set of multiple objects; based on the observation error, a three-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the three-dimensional Gaussian mixture model; the parameters of the three-dimensional Gaussian mixture model are used as observation factors to construct observation factor nodes and are introduced into the multi-objective state optimization factor graph to estimate the multi-objective state estimation results at the current time.

[0236] The multi-sensor state optimization layer is used to calculate the reprojection error based on the coarse localization point set of multiple targets. Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed and solved to obtain the parameters of the two-dimensional Gaussian mixture model. The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors to construct reprojection error nodes and are introduced into the sensor state optimization factor graph to estimate the multi-sensor state optimization estimation result at the current time.

[0237] The joint optimization layer is used to perform joint nonlinear least squares optimization on the target state and the sensor state within the same sliding window to obtain the joint optimization result.

[0238] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0239] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for joint optimization of multiple passive sensors and multiple target states based on two-factor graphs, characterized in that, include: Step S1, Passive detection multi-target measurement solution layer solves for multi-target state measurements: Based on all passive sensors, a primary sensor and an auxiliary sensor are set up, and a cooperative coordinate system is constructed with the zero-time position of the Earth-centered Earth-fixed system as the origin. In the cooperative coordinate system, batch numbers are assigned to all targets by combining the multi-target identity determination results between any main sensor and auxiliary sensor. Based on the pixel coordinates of multiple targets of the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined. The direction finding lines of the same batch number are then fused by the least squares method and weighted average to obtain a coarse localization point set of multiple targets, which is used as the state measurement of multiple targets. Step S2: The multi-target management layer performs trajectory matching and management. Using a coarse set of positioning points for multiple targets as input, the current state measurement of the target is matched with the existing target trajectory through trajectory data association; If a match is found, the corresponding target trajectory is updated; If a match fails and the number of consecutive occurrences of an unmatched measurement exceeds the preset confirmation count, a new target trajectory will be generated. Step S3: Solve for the multi-objective state estimation results in the multi-objective state optimization layer. The deviation between the current state prediction and the coarse location points of multiple targets is calculated to obtain the observation error; Based on observation errors, a three-dimensional Gaussian mixture model is constructed using the target optimization EM algorithm, and the parameters of the three-dimensional Gaussian mixture model are obtained by solving the algorithm. The parameters of the three-dimensional Gaussian mixture model are used as observation factors, and observation factor nodes are constructed by combining prior factors, motion model factors and mutually exclusive factors. The observed factor nodes are introduced into the multi-objective state optimization factor graph, and nonlinear minimization is performed on the multi-objective state optimization factor graph within the current window to estimate the multi-objective state; The objective optimization EM algorithm is iterated repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-objective state estimation result at the current time. Step S4: The multi-sensor state optimization layer solves for the multi-sensor state optimization estimation results. The coarse localization points of multiple targets are reprojected onto the multi-sensor imaging plane to obtain the reprojected points of the multiple targets; The reprojection error is obtained based on the deviation between the current multi-target observation point and the corresponding reprojection point; Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed using the sensor-optimized EM algorithm, and the parameters of the two-dimensional Gaussian mixture model are obtained by solving the algorithm. The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors, and combined with sensor factors, reprojection error nodes are constructed. The reprojection error nodes are introduced into the sensor state optimization factor graph, and nonlinear minimization is performed on the sensor state optimization factor graph within the current window to estimate the sensor state. The sensor state EM algorithm iterates repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-sensor state optimization estimation result at the current moment; Step S5: Perform joint optimization of state estimation using multi-sensor, multi-target factor graphs. The target state and the sensor state are jointly optimized using nonlinear least squares optimization within the same sliding window to obtain the joint optimization result.

2. The method for joint optimization of multiple passive sensors and multiple target states based on a two-factor graph according to claim 1, characterized in that, Step S1 specifically includes: Based on all passive sensors, one passive sensor is designated as the master sensor, and the rest of the passive sensors are designated as auxiliary sensors. A collaborative coordinate system for distributed passive sensors is constructed with the origin of the coordinate system at the position point where the main sensor is at a height of 0 at time zero. The position information, attitude information, and target observation information of each auxiliary sensor are uniformly converted into the cooperative coordinate system; The identity of multiple targets in the field of view of different passive sensors is determined, and the one-to-one correspondence between the two-dimensional observation targets of each passive sensor and multiple real targets in space is determined. Based on the multi-target identity determination results, the number of the multi-target number in the imaging plane of the main sensor is used as the basis, and the number is used as the unique batch number of the corresponding real target in space. For any auxiliary sensor, each target in its imaging plane is assigned the target number of the main sensor associated with it through identity determination, which serves as the batch number of the real spatial target represented by that target. By combining the pixel coordinates of multiple targets observed in the two-dimensional imaging plane of any passive sensor, the direction finding lines pointing from the optical centers of different passive sensors to all targets are calculated; By combining direction finding lines pointing to the same batch of targets in pairs, and using the least squares method to calculate the midpoint of the common perpendicular of any combination of direction finding lines, the estimated position of the same target can be obtained. Multiple estimated location points of the same target are fused to obtain the coarse location point of the target; Traverse all two-dimensional observation targets within the main sensor to obtain a coarse set of positioning points for multiple targets.

3. The method for joint optimization of multiple passive sensors and multiple target states based on a two-factor graph according to claim 2, characterized in that, Step S2 specifically includes: Based on the coarse positioning point set of multiple targets and the existing multi-target trajectory set at historical time, calculate the Mahalanobis distance between each state measurement at the current time and the target trajectory at the previous time. A correlation similarity matrix is ​​constructed based on Mahalanobis distance to solve the one-to-one matching relationship between the trajectory at the current time and the trajectory at the previous time. For successfully matched state measurements, update their corresponding target trajectories; For state measurements that fail to match, i.e., unmatched measurements, perform new target determination and initialization: Set the preset number of confirmations Continuous tracking and statistics are performed on unmatched measurements. If a certain unmatched measurement is continuously... If a measurement occurs at all observation times and does not match any existing target trajectory each time, it is determined that the unmatched measurement corresponds to a new real spatial target, triggering the new target generation process, completing the initialization of the new target trajectory, and adding the new target trajectory to the existing trajectory set after initialization.

4. The method for joint optimization of multiple passive sensors and multiple target states based on a two-factor graph according to claim 3, characterized in that, Step S5 specifically includes: Based on the current multi-objective state estimation results and the current multi-sensor state optimization estimation results, a nonlinear least squares optimization problem is constructed with the objective of minimizing the total error of the correlation factor nodes between all sensor states and target state variables within the sliding window. Solve the nonlinear least squares optimization problem to obtain the joint optimized estimate of the state of all targets and all sensor attitude states within the current window; The jointly optimized estimated state obtained at each time step is synchronously fed back to the multi-target management layer to form a closed-loop update, thereby realizing the joint optimization of attitude correction of multiple passive sensors and state estimation of multiple targets.

5. A multi-passive sensor and multi-target state joint optimization system based on a two-factor graph, characterized in that, include: A passive detection multi-target measurement solution layer is used to set the main sensor and auxiliary sensor based on all passive sensors, and to construct a cooperative coordinate system with the zero-time position of the Earth-centered Earth-fixed system as the origin; In the cooperative coordinate system, the batch number is assigned to all targets by combining the multi-target identity determination results between any main sensor and auxiliary sensor; based on the multi-target pixel coordinates of targets with the same batch number in the two-dimensional imaging plane of any two passive sensors, the direction finding lines are calculated and combined with the direction finding lines of targets with the same batch number. The coarse positioning point set of the multi-target is obtained by fusing the least squares method and weighted average, and is used as the state measurement of the multi-target. The multi-target management layer is used to take a coarse set of positioning points of multiple targets as input, and match the current target state measurement with the existing target trajectory through trajectory data association; if the match is successful, the corresponding target trajectory is updated. If a match fails and the number of consecutive occurrences of an unmatched measurement exceeds the preset confirmation count, a new target trajectory will be generated. The multi-objective state optimization layer is used to calculate the deviation between the current state prediction value and the coarse positioning points of multiple objects, thus obtaining the observation error. Based on observation errors, a three-dimensional Gaussian mixture model is constructed using the target optimization EM algorithm, and the parameters of the three-dimensional Gaussian mixture model are obtained by solving. The parameters of the three-dimensional Gaussian mixture model are used as observation factors, and observation factor nodes are constructed by combining prior factors, motion model factors and mutually exclusive factors. The observed factor nodes are introduced into the multi-objective state optimization factor graph, and nonlinear minimization is performed on the multi-objective state optimization factor graph within the current window to estimate the multi-objective state; The objective optimization EM algorithm is iterated repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-objective state estimation result at the current time. The multi-sensor state optimization layer is used to reproject the coarse localization point set of multiple targets onto the multi-sensor imaging plane to obtain the multi-target reprojection points. Based on the deviation between the current multi-target observation points and their corresponding reprojection points, the reprojection error is obtained. Based on the reprojection error, a two-dimensional Gaussian mixture model is constructed using the sensor optimization (EM) algorithm, and the parameters of the two-dimensional Gaussian mixture model are obtained. The parameters of the two-dimensional Gaussian mixture model are used as reprojection factors, combined with sensor factors, to construct reprojection error nodes. The reprojection error nodes are introduced into the sensor state optimization factor graph, and nonlinear minimization is performed on the sensor state optimization factor graph within the current window to estimate the sensor state. The sensor state EM algorithm iterates repeatedly until a set number of iterations or the optimal value is reached, in order to obtain the multi-sensor state optimization estimation result at the current moment; The joint optimization layer is used to perform joint nonlinear least squares optimization on the target state and the sensor state within the same sliding window to obtain the joint optimization result.

Images