Multi-target Fusion Clustering Association Method among Multiple Infrared Sensors Based on Associability

Abstract

The present application discloses a multi-target fusion clustering association method among multiple infrared sensors based on associativity. By decoupling the multi-target association problem among multiple infrared sensors into a combined optimization problem of multiple master-slave sensor pairs, the computational complexity is reduced. By establishing a mathematical model for the multi-target association ability among multiple infrared sensors, the mathematical conditions for the multi-target association ability among infrared sensors are deduced, providing a theoretical basis for optimizing the sensor array. Based on the continuous tracking information of multiple targets in the two-dimensional imaging plane of the sensor, the preliminary association of multiple targets between the master and slave sensors is realized. By fusing the preliminary association results among multiple master-slave sensors, the multi-target association result among multiple infrared sensors is obtained. The present application ensures the accuracy and robustness of multi-target association on the basis of ensuring the data fusion efficiency, thus effectively solving the problem of multi-target association mismatch among sensors in the multi-target cooperative detection of a multi-infrared sensor system.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target association matching and identity determination among multiple infrared sensors, and more particularly to a multi-target fusion clustering association method, device, storage medium and equipment among multiple infrared sensors based on associability. Background Art

[0002] As an important part of the modern perception system, infrared sensing technology has shown extensive application value in civil fields such as urban low-altitude security and forest fire prevention and rescue by virtue of its all-weather working ability, excellent anti-electromagnetic interference performance and good concealment characteristics. Compared with visible light sensing technology, infrared sensors can maintain a stable working state in low-light or even lightless environments, significantly improving the target perception and tracking efficiency in complex scenarios such as night-time UAV navigation, heat source positioning in disaster areas, and wildlife night monitoring. However, a single infrared sensor faces technical bottlenecks such as limited detection field of view, insufficient acquisition of target feature information, and weak anti-occlusion ability in practical applications such as UAV obstacle avoidance in urban dense building areas and thermal imaging monitoring of agricultural pests and diseases. Its target detection and tracking accuracy are easily affected by factors such as thermal radiation interference from high-temperature equipment and signal attenuation caused by vegetation occlusion.

[0003] In view of the inherent defects of single sensors, a multi-infrared sensor collaborative detection scheme has emerged. This scheme realizes more comprehensive information acquisition of target objects through multi-viewpoint observation, effectively improving the reliability and tracking accuracy of single-target detection. At the same time, the information fusion mechanism among multiple sensors can significantly reduce the risk of single-point failure and enhance the overall reliability of the system. However, in the scenario of multi-sensor collaborative detection of multiple targets, such as in high-density target scenarios such as multi-UAV formation scheduling in logistics hubs and unauthorized aircraft control in large-scale concerts, with the increase in target density, the target association mismatch rate among sensors shows an exponential upward trend. A large number of association mismatches will lead to incorrect target positioning, seriously affecting the effective tracking and detection of multiple targets by the system, resulting in serious problems such as UAV path planning conflicts and misjudgment and missed judgment of intrusion targets. Especially when facing formation moving targets, if the sensor deployment configuration is unreasonable, the observed data among different sensors will lack associability, further exacerbating the association mismatch problem. Therefore, how to achieve efficient association of multi-target information obtained by multiple sensors and ensure the accuracy of multi-target identity judgment has become a key topic for improving the performance of multi-infrared sensor collaborative detection systems. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, a multi-target fusion clustering association method between multiple infrared sensors based on associativity is proposed in the first aspect of the present invention, including: when multiple formation infrared sensors cooperate to detect multiple moving space targets, randomly determine one of the multiple formation infrared sensors as the main sensor, and the remaining infrared sensors as auxiliary sensors, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-Centered Earth-Fixed (ECEF) coordinate system as the origin; in the cooperative coordinate system, based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, construct a multi-target associativity matrix between the main sensor and each auxiliary sensor at the current moment; solve the multi-target associativity matrix to obtain the rank of the multi-target associativity matrix. If the rank of the multi-target associativity matrix meets the preset conditions, it is determined that there is associativity between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane, determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm; based on the preliminary association results, determine the center points of the common perpendiculars between the direction lines of the main sensor and each auxiliary sensor for different space targets, determine the set of preliminary association positioning points based on the set of center points, and cluster the set of preliminary association positioning points through a clustering algorithm to obtain the fusion clustering association positioning points of any space target at any moment; compare the distances between the fusion clustering association positioning points of all space targets at any moment and their respective motion tracks with the set distance threshold to determine whether the association is correct, and determine the fusion clustering association results of multiple formation infrared sensors for multiple moving space targets by combining the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate indicators.

[0005] Optionally, define the Earth-Centered Earth-Fixed coordinates of the main sensor at the zero moment as the origin, and use the following formula to calculate the position of the th auxiliary sensor in the cooperative coordinate system at the moment:

[0006]

[0007] In the formula, is the longitude and latitude coordinates of the th auxiliary sensor at the moment, is the Earth-Centered Earth-Fixed coordinate system of the main sensor at the zero moment, is the Earth-Centered Earth-Fixed coordinate system of the main sensor at the zero moment.

[0008] Optionally, constructing a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment based on the pixel coordinate information, respective internal parameter information, and respective position information of multiple spatial targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes includes: obtaining the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor, and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor, and solving the direction lines pointing from the optical centers of different sensors to the first pixel coordinates and the second pixel coordinates based on the first pixel coordinates, the second pixel coordinates, and the optical center coordinates of each sensor; constructing a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment based on each direction line, the internal parameters of each sensor, and its own position information.

[0009] Optionally, the expression of the multi-target association matrix is:

[0010]

[0011] Where:

[0012]

[0013] In the formula, represents the direction vector of the direction line of the th spatial target in the main sensor coordinate system in the main sensor, represents the direction vector of the direction line of the rd target in the th auxiliary sensor in the auxiliary sensing coordinate system, and respectively represent the rotation matrices of the main sensor and the auxiliary sensor at the current moment, represents the direction vector of the direction line of the th target in the main sensor in the cooperative coordinate system, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the cooperative coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the main sensor imaging plane, represents the second pixel coordinate, represents the th central pixel coordinate of the imaging plane of the and respectively represent the pixel size and focal length of the main sensor, and respectively represent the pixel size and focal length of the th auxiliary sensor.

[0014] Optionally, solving the multi-object association matrix to obtain the rank of the multi-object association matrix, and if the rank of the multi-object association matrix meets a preset condition, it is determined that the multiple targets observed by the multi-infrared sensors at the current moment are associable, including: for the multi-object association matrix performing singular value decomposition to obtain the rank of the multi-object association matrix , and if , it is determined that there is an associability between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment, where m and n respectively represent the number of spatial targets observed by the main sensor and the number of spatial targets observed by the auxiliary sensor.

[0015] Optionally, the method for determining the preliminary association results of the main sensor and each auxiliary sensor for different spatial targets by combining the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane may include: in the cooperative coordinate system, based on the unit direction vectors of the main sensor optical center and the auxiliary sensor optical center pointing to each target, calculating the first direction vector of the direction line and the second direction vector of the direction line of the main sensor and the auxiliary sensor optical center pointing to each target respectively; in the cooperative coordinate system, calculating the foot coordinates of the common perpendicular of the first direction vector of the direction line and the second direction vector of the direction line based on the direction vectors of the direction lines; calculating the perpendicular distance based on the foot coordinates of the common perpendicular of the first direction vector of the direction line and the second direction vector of the direction line; calculating the mean and variance of the perpendicular distance within a preset time period; constructing the association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor within the preset time period based on the mean and variance of the perpendicular distance; at the current moment, constructing the preliminary association results of the multiple targets between the main sensor and each auxiliary sensor based on the maximum association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor obtained by solving the Hungarian algorithm.

[0016] Optionally, determining the set of preliminary association positioning points of the main sensor and each auxiliary sensor for different spatial targets based on the preliminary association results, and clustering the set of preliminary association positioning points through a clustering algorithm to obtain the fused clustering association positioning points of any spatial target at any moment, including: calculating the center point coordinates of the common perpendicular based on the intersection coordinates of the common perpendicular of the first direction vector of the direction line and the second direction vector of the direction line of the main sensor and each auxiliary sensor pointing to the same spatial target in any preliminary association result, and obtaining the set of preliminary association positioning points of all spatial targets at the current moment based on all the center point coordinates; clustering the set of preliminary association positioning points of all spatial targets through a clustering algorithm to obtain multiple clustering centers, and using the clustering centers as the fused clustering association positioning points of any spatial target at any moment.

[0017] Optionally, comparing the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective motion tracks with a set distance threshold to determine whether the association is correct, includes: setting a fixed distance threshold, calculating the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective true motion tracks, comparing the distances with the set distance threshold, if the distance is less than the set distance threshold, it is determined as a correct association; otherwise, it is determined as an incorrect association.

[0018] Optionally, determining the fusion clustering association results of multiple formation infrared sensors for multiple moving spatial targets by combining the multi-target fusion clustering association precision and the multi-target fusion clustering association recall rate indicators, includes: obtaining the number of fusion clustering associated positioning points successfully associated with the true motion tracks of the targets , the number of fusion clustering associated positioning points not successfully associated with the true motion tracks of the targets , the number of true motion tracks of the targets that have not been successfully matched with any fusion clustering associated positioning points ; Based on accounting for and to determine the multi-target fusion clustering association precision; based on accounting for and to determine the multi-target fusion clustering association recall rate; if both the fusion clustering association precision and the fusion clustering association recall rate exceed the preset threshold, the association effect is determined to be good, otherwise it is unqualified.

[0019] Based on the above embodiments, the second aspect of the present application further provides a multi-target fusion clustering association device among multiple infrared sensors based on associativity, including: a sensor classification module, which is used to randomly determine one of the multiple formation infrared sensors as the main sensor and the remaining infrared sensors as auxiliary sensors when the multiple formation infrared sensors cooperate to detect multiple moving space targets, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin; an association matrix construction module, which is used to construct a multi-target associativity matrix between the main sensor and each auxiliary sensor at the current moment in the cooperative coordinate system based on the pixel coordinate information of multiple space targets, their respective internal parameter information, and their own position information obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes; an associativity judgment module, which is used to solve the multi-target associativity matrix to obtain the rank of the multi-target associativity matrix, and if the rank of the multi-target associativity matrix meets the preset conditions, it is determined that there is associativity between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; a preliminary association result determination module, which is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the perpendicular distance statistical value of the common perpendicular with the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane; a positioning point determination module, which is used to determine the center point of the common perpendicular between the direction lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association results, determine the preliminary association positioning point set based on the set of center points, and cluster the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment; an association result determination module, which is used to compare the distance between the fusion clustering association positioning point of all space targets at any moment and their respective motion trajectories with a set distance threshold to determine whether the association is correct, and determine the fusion clustering association results of the multiple formation infrared sensors for the multiple moving space targets in combination with the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate indicators.

[0020] An embodiment of the present invention provides a multi-target fusion clustering association method between multiple infrared sensors based on associability. Compared with the prior art, its beneficial effects are as follows: When multiple formation infrared sensors cooperate to detect multiple moving space targets, one of the multiple formation infrared sensors is randomly determined as the main sensor, and the remaining infrared sensors are auxiliary sensors. With the main sensor as the coordinate origin, a cooperative coordinate system is constructed. Thus, the multi-target association problem between multiple infrared sensors is decoupled into a combinatorial optimization problem of multiple main-auxiliary sensor pairs, thereby reducing the computational complexity; In the cooperative coordinate system, based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment is constructed; The multi-target association matrix is solved to obtain the rank of the multi-target association matrix. If the rank of the multi-target association matrix meets the preset conditions, it is determined that the multiple targets observed by the multiple infrared sensors at the current moment have associability. Based on the established mathematical model of the multi-target association ability between multiple infrared sensors, the mathematical conditions for the multi-target association ability between infrared sensors are derived, providing a theoretical basis for optimizing the sensor formation; Based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, a combination of the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets. Based on the continuous tracking information of multiple targets in the two-dimensional imaging plane of the sensor, the preliminary association of multiple targets between the main and auxiliary sensors is realized; Based on the preliminary association results, a set of preliminary association positioning points of the main sensor and each auxiliary sensor for different space targets is determined, and the set of preliminary association positioning points is clustered by a clustering algorithm to obtain the fusion clustering association positioning points of any space target at any moment; Based on the distance threshold between the fusion clustering association positioning points of all space targets at any moment and their respective motion trajectories, the association correctness is evaluated. If the evaluation passes, the correct association results of multiple formation infrared sensors for multiple moving space targets are determined. This application fully exploits the observation information advantages of multiple infrared sensors for multiple targets, ensuring the accuracy and robustness of multi-target association while guaranteeing the data fusion efficiency, thereby effectively solving the problem of multi-target association mismatch between sensors in the multi-target cooperative detection of a multi-infrared sensor system, providing effective support for subsequent multi-infrared sensor cooperative multi-target perception, tracking, and positioning. Brief Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for use in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic diagram of a typical scenario where multiple infrared sensors cooperate to detect multiple targets;

[0023] Figure 2 It is a schematic diagram of the process of the multi-infrared-sensor multi-target fusion clustering and association method based on associability proposed by the present invention;

[0024] Figure 3 It is for combining Figure 1 A schematic diagram of constructing a cooperative coordinate system for a multi-infrared-sensor cooperative detection system in combination with the typical scenario shown;

[0025] Figure 4 It is for combining Figure 1 A schematic diagram of a method for decoupling the multi-target association problem between multiple sensors into multiple multi-target association problems between a master-sensor and a slave-sensor in combination with the typical scenario shown;

[0026] Figure 5 A schematic diagram of a method for judging whether targets between different sensors are associated by combining the perpendicular distance of the least common perpendicular of the direction lines;

[0027] Figure 6 It is a typical scenario where ghost points appear in the process of multi-target association between a master-sensor and a slave-sensor;

[0028] Figure 7 (a), 7 (b), and 7 (c) are three master-sensor and slave-sensor formations that do not meet the associability conditions;

[0029] Figure 8 A schematic diagram of a method for completing the preliminary association of multi-targets between a master-sensor and a slave-sensor by combining the continuous tracking information of multi-targets in the two-dimensional imaging plane of the sensor;

[0030] Figure 9 It is for Figure 1 A schematic diagram of a method for completing the multi-target association between multiple sensors by combining the fusion clustering and association method in the scenario shown; Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0032] This specification provides the method operation steps as described in the embodiments or flowcharts, but may include more or fewer operation steps based on routine or non-creative labor. When the actual system or server product is executed, it can be executed in the method order shown in the embodiments or the drawings or in parallel (e.g., in an environment with parallel processors or multi-threaded processing).

[0033] Currently, the multi-target association methods between dual infrared sensors mainly rely on two mainstays: the geometric constraint theory and the statistical analysis theory. Such methods show relatively good association accuracy in dual-sensor systems, but they cannot fully adapt to the complex scenarios where the number of sensors and the target scale increase simultaneously. In addition, the fundamental reason why dual-sensor association methods are difficult to directly extend to multi-sensor systems is that the increase in the data dimension of multi-sensor systems leads to a significant increase in computational complexity, and the cumulative effect of observation errors between multi-sensors will affect the association accuracy. This prompts researchers to develop more efficient multi-sensor fusion association methods for multi-infrared sensor systems, ensuring the accuracy and robustness of multi-target association while guaranteeing the data fusion efficiency.

[0034] In the scenario of multi-infrared sensor cooperative detection of formation targets, due to the similarity in the motion trends of targets and sensors, the multi-target observation data obtained under the current multi-infrared sensor deployment configuration may not meet the association conditions. However, the existing methods have not fully considered the satisfiable conditions for multi-target association between multi-infrared sensors. Therefore, it is urgent to analyze the associable capabilities between multi-targets based on the multi-target observation information of multi-infrared sensors, and construct a multi-target association matrix for mathematical description. By analyzing the rank of the multi-target association matrix under different multi-infrared sensor deployment configurations, it is judged whether the current sensor deployment configuration meets the association conditions, and then the multi-infrared sensor deployment scheme is optimized to reduce the association mismatches caused by the non-satisfaction of the association conditions, improve the robustness of multi-target association, and ultimately enhance the target perception, tracking, and positioning capabilities of the multi-infrared sensor cooperative detection system in complex environments.

[0035] Based on the above analysis, it is of great theoretical and application value to develop an efficient and association-based multi-target fusion clustering association method between multi-infrared sensors. This method should fully exploit the advantages of the observation information of multi-sensors for multi-targets, and ensure the accuracy and robustness of multi-target association on the basis of guaranteeing the data fusion efficiency. By solving this key technical problem, the accuracy and computational efficiency of multi-target association between multi-infrared sensors in complex scenarios can be significantly improved, providing effective support for subsequent multi-infrared sensor cooperative multi-target perception, tracking, and positioning, and providing core technical support for major livelihood fields such as low-altitude economic safety control and urban emergency response systems.

[0036] To solve the above technical problems, the present invention proposes a multi-target fusion clustering association method between multiple infrared sensors based on relevance. In this embodiment, the complete process steps of the present invention for achieving multi-target fusion clustering association between multiple infrared sensors are described in detail in combination with a typical actual application scenario where four formation infrared sensors cooperate to detect four formation moving targets. As shown in the appendix Figure 1 In the typical scenario of multi-infrared sensor cooperative detection of multiple targets shown, there are four moving infrared sensors. During a period of time, four formation moving targets are observed within the common field of view of the four infrared sensors, and continuous movement trajectories of the four targets are obtained respectively in the two-dimensional imaging planes of each infrared sensor. The present invention assumes that the spatial coordinate positions of the sensor centroid or the sensor optical center coincide. In addition, the number of sensors described in this embodiment is not limited to the four infrared sensors shown in the appendix Figure 1 in the scenario of four infrared sensors cooperating to detect four targets, and can also be extended to multiple infrared sensors cooperating to detect multiple targets.

[0037] Next, in combination with the typical scenario shown in the appendix Figure 1 and according to the specific process steps shown in the appendix Figure 2 , the technical solution of the multi-target fusion clustering association method between multiple infrared sensors based on associability proposed in this application is described in detail.

[0038] Referring to Figure 1 , this application provides a multi-target fusion clustering association method between multiple infrared sensors based on associability. This method can be executed by a processor, and the processor can belong to any terminal or server, which is not limited here. The multi-target fusion clustering association method between multiple infrared sensors based on associability may include:

[0039] S10. When multiple formation infrared sensors cooperate to detect multiple moving space targets, randomly determine one of the multiple formation infrared sensors as the main sensor, and the remaining infrared sensors as auxiliary sensors, and construct a cooperative coordinate system with the initial position of the main sensor at zero moment in the Earth-centered Earth-fixed coordinate system as the origin.

[0040] It should be noted that the number of infrared sensors is at least 3, and the sensor types include but are not limited to infrared sensors.

[0041] In the infrared sensor cooperative detection system, there are multiple infrared sensor nodes. First, the processor takes any one of the infrared sensors as the main sensor and the remaining sensors as auxiliary sensors. As shown in the appendix Figure 3 , shown in the appendix Figure 3 is a schematic diagram of constructing a cooperative coordinate system for a multi-infrared sensor cooperative detection system in combination with the typical scenario shown in the appendix Figure 1 . As shown in Figure 1As shown, the four infrared sensors are respectively designated as the main sensor, auxiliary sensor 1, auxiliary sensor 2, and auxiliary sensor 3. Secondly, taking the position of the main sensor at the zero moment as the origin of the cooperative coordinate system of the multi-infrared sensor cooperative detection system, a cooperative coordinate system that obeys the right-hand system is constructed. The X-axis of the cooperative coordinate system points east, the Y-axis points north, and the Z-axis is perpendicular to the ground and points upward. The purpose of establishing the cooperative coordinate system of the multi-infrared sensor cooperative detection system is to complete tasks such as observation and positioning tracking of multiple sensors in a unified coordinate system, so as to reduce the observation error and positioning error caused by the spatial registration problem, etc.

[0042] The processor takes the main sensor The moment when the target is observed as the zero moment, takes the position of the main sensor at the zero moment as the origin of the cooperative coordinate system, and assumes that the geocentric earth-fixed coordinate of the main sensor at the zero moment is , then the th auxiliary sensor The Position in the cooperative coordinate system at the moment Can be obtained through the following formula:

[0043]

[0044] In the formula Is the th auxiliary sensor at the Latitude and longitude coordinates at the moment, Is the geocentric earth-fixed coordinate of the main sensor at the zero moment.

[0045] After that, the processor needs to decouple the multi-target association problem among multiple infrared sensors in the cooperative coordinate system of the multi-infrared sensor cooperative detection system into the multi-target association problem among multiple master-slave sensor combinations.

[0046] The purpose of multi-target association among multiple infrared sensors is to find the correct one-to-one correspondence of targets in all association combinations, so as to judge whether two targets observed in different sensors come from the same spatial target. The maximum number of multi-target association combinations among multiple sensors Can be expressed by the following formula:

[0047]

[0048] Among them, Represents the number of targets, Represents the number of sensors. The size will expand geometrically with the number of sensors and the number of targets. Therefore, the present invention decouples the multi-target association problem among multiple infrared sensors into a combined optimization problem of multiple master-slave sensor pairs, only considering the multi-target association between each auxiliary sensor and the master sensor, without considering the multi-target association between each auxiliary sensor. Through this method, the number of combinations of multi-target associations among multiple sensors is optimized to:

[0049]

[0050] Based on this, the multi-target association problem among multiple sensors can be mathematically described. Assume that the targets observed in the master sensor are represented as a set , and the targets observed in the th auxiliary sensor are represented as a set , where . Then the multi-target association problem among multiple infrared sensors can be described as:

[0051]

[0052] where, represents a specific target in the set, and this target is determined to be the same target as a certain target in the set. , when , it means that the target detected by the master sensor and the target detected by the th auxiliary sensor are formed by the same target in space. represents the association weight between target and target . The above expression is not only applicable to point targets in single-frame images of different sensors in a certain frame, but also applicable to target trajectories in imaging planes of different sensors over a period of time.

[0053] As shown in the appendix Figure 4 , appendix Figure 4 is a schematic diagram of the method for decoupling the multi-target association problem among multiple infrared sensors into multiple master-slave sensor multi-target association problems in combination with the typical scenario shown in appendix Figure 1 . Among them, represents four targets observed by the master sensor, represents four targets observed by auxiliary sensor 1, represents four targets observed by auxiliary sensor 2, represents four targets observed by auxiliary sensor 3. Figure 4The dashed lines connecting the multi-target elements represent the association combinations of multi-targets among multiple sensors, with a total of 48 association combination methods.

[0054] S20. In the cooperative coordinate system, based on the pixel coordinate information of multiple spatial targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, construct the multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment.

[0055] In the embodiment of the present application, step S20 may include the following execution process:

[0056] S201. Obtain the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor, and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor, and solve the direction lines pointing from the optical centers of different sensors to the first pixel coordinates and the second pixel coordinates based on the first pixel coordinates, the second pixel coordinates, and the optical center coordinates of each sensor.

[0057] S202. Based on each direction line, the internal parameters of each sensor, and its own position information, construct the multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment.

[0058] In the specific execution process, for a certain target in space , suppose at moment, the coordinates of this target in the cooperative coordinate system are , and the pixel coordinates of the target in the sensor imaging plane at the moment are obtained through the following formula: :

[0059]

[0060] In the formula, represents the Z-axis coordinate of the target in the sensor coordinate system. represents the focal length of the sensor, represents the pixel size of the sensor, represents the center coordinates of the sensor imaging plane, represents the translation of the sensor, represents the rotation matrix of the sensor coordinate system with added noise relative to the cooperative coordinate system, which is expressed by the following formula:

[0061]

[0062] In the formula:

[0063]

[0064]

[0065] In the formula, respectively represent the pitch, yaw, and roll angles of the camera. respectively represent the measurement noises that follow a Gaussian distribution obtained from the angular measurement errors of the sensors, , where the variance , represents the angular measurement accuracy of the passive sensor. It means that there is a 99.7% probability that the angular measurement error appears within the angular measurement accuracy range.

[0066] Let the pixel coordinates of the target in the two-dimensional imaging plane of the main sensor obtained at the th moment be , and at the same moment, in the two-dimensional imaging plane of the th auxiliary sensor, the pixel coordinates of the target be .

[0067] To determine whether the targets and come from the same target in space, the processor can combine the pixel coordinates of the target to solve the direction lines pointing from the optical centers of different sensors to this pixel coordinate. As shown in Appendix Figure 5 , Appendix Figure 5 is a schematic diagram of the method for judging whether the targets between different sensors are associated by the perpendicular distance of the least common perpendicular of the direction lines. In the figure, it shows that at the th moment, the direction line of the optical center of the main sensor pointing to the target is , and the direction line of the th auxiliary sensor pointing to the target is . If the targets and come from the same target in space, in the ideal case without any errors, the direction lines and intersect in space, and the intersection point is the three-dimensional position where the target is located. However, due to the resolution and error conditions of the sensors, the distance between the two direction lines is always greater than zero. Therefore, the perpendicular distance of the least common perpendicular between the direction lines can be used as an association criterion to determine the association weight between the targets. The smaller the perpendicular distance of the least common perpendicular, the higher the probability that the current association combination is correct.

[0068] As mentioned above, under ideal conditions, two direction-finding lines from the same target will intersect at a point in a plane in space, which means that the above criterion is based on the coplanarity of the direction-finding lines to determine whether the targets are associated. However, when there are more than one pair of coplanar direction-finding lines, ghost points will appear. Figure 6 As shown, attached Figure 6 This is a typical scenario where ghost points appear during the multi-target association process between the main and auxiliary sensors. Target 1 and target 2 are coplanar with the main sensor and the auxiliary sensor. Therefore, the four direction finding lines pointed to the two targets by different sensors will have four intersections in this plane. The light-colored crosses are ghost points. In this scenario, the two targets between the two sensors are not associable because the sensor formation does not meet the associability condition.

[0069] In the attached Figure 1 In the scenario shown, where the four sensors cooperate to detect four formation targets, since the geometric configurations of multiple targets are usually regular and the movement trends of multiple targets are basically the same, this scenario is more likely to have a sensor formation that does not meet the associativity. Therefore, the present invention combines the observation information of multiple sensors and the internal parameters of the sensors to construct a multi-target associativity matrix of multiple targets between multiple sensors, analyzes the associatability conditions of multiple targets between multiple sensors, and then optimizes the formation of multiple sensors, thereby eliminating the influence of ghost points on the multi-target association effect as much as possible.

[0070] Assume that the main sensor is The set of multiple targets observed at any moment is , No. The auxiliary sensors observed The set of targets is , then combined with the internal parameters of the sensor and the rotation matrix of each sensor, the multi-target correlation matrix between the main sensor and the s-th auxiliary sensor at the current moment can be constructed :

[0071]

[0072] in:

[0073]

[0074] Among them, in the formula, Indicates the main sensor The direction vector of the direction finding line of a space target in the main sensor coordinate system, Indicates Among the auxiliary sensors The direction vector of the direction finding line of the target in the auxiliary sensor coordinate system, and respectively represent the rotation matrices of the primary sensor and the auxiliary sensor at the current moment, represents the th target in the primary sensor, and the direction vector of the direction line of the target in the cooperative coordinate system, represents the th auxiliary sensor, and the th target in the direction vector of the direction line of the target in the cooperative coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the imaging plane of the primary sensor, represents the second pixel coordinate, represents the th auxiliary sensor imaging plane center pixel coordinates, and respectively represent the pixel size and focal length of the primary sensor, and respectively represent the th auxiliary sensor pixel size and focal length.

[0075] The multi-target correlation matrix is only related to the pixel coordinate positions of the targets among the sensors and the internal parameters of each sensor, and characterizes the correlation ability of multiple targets among multiple sensors.

[0076] S30. Solve the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is determined that there is a correlation between the multiple targets observed by the primary sensor and the auxiliary sensor at the current moment.

[0077] In the specific execution process, the processor performs singular value decomposition on the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix . If , it is determined that there is a correlation between the multiple targets observed by the primary sensor and the auxiliary sensor at the current moment, where m and n respectively represent the number of spatial targets observed by the primary sensor and the number of spatial targets observed by the auxiliary sensor.

[0078] It should be noted that if the rank of the multi-target correlation matrix does not meet the preset conditions, it is determined that the multiple targets observed by the multi-infrared sensors at the current moment do not have a correlation.

[0079] The following combines Figure 7 to analyze two cases. Considering the ideal error-free condition and the situation of all targets in the same area observed by both sensors, as shown in the appendix Figure 7 shown, Figure 7 are three primary-auxiliary sensor formations that do not meet the correlation conditions, whereFigure 7 (a) the vertical relevance condition is not met, Figure 7 (b) the horizontal relatability condition is not met, Figure 7 (c) The vertical relatability condition is not met.

[0080] In the attached Figure 7 In (a), at a certain moment, when the positions of the main sensor and the auxiliary sensor differ only in the Y-axis coordinate, if all targets also differ only in the Y-axis coordinate, then the direction-finding lines pointed from all sensors to all targets are in the same inclined plane. Multiple direction-finding lines in the same plane will cause the appearance of ghost points.

[0081] In the attached Figure 7 In (b), at a certain moment, when the positions of the main sensor and the auxiliary sensor differ only in the X-axis coordinate, if all targets also differ only in the X-axis coordinate, then the direction-finding lines pointed by all sensors to all targets are in a vertical plane parallel to the Z-axis. Multiple direction-finding lines in the same plane will cause the appearance of ghost points.

[0082] In the attached Figure 7 In (c), at a certain moment, when the positions of the main sensor and the auxiliary sensor differ only in the Z-axis coordinate, if all targets also differ only in the Z-axis coordinate, then the direction-finding lines pointed from all sensors to all targets are in a vertical plane parallel to the Z-axis. Multiple direction-finding lines in the same plane will cause the appearance of ghost points.

[0083] In summary, for the scenario of multiple infrared sensors collaboratively detecting multiple formation targets, the following multi-infrared sensor formation that meets the relatability conditions as much as possible can be summarized: at any time, all infrared sensors in the multi-infrared sensor collaborative detection system should avoid regular formations, and the three-axis spatial position coordinates of all sensors should be different as much as possible.

[0084] The above analysis only considers the ideal error-free condition and the case where all targets in the same area are observed by both the primary and secondary sensors. When errors are involved or the number of targets observed by the sensors is different, further analysis is required in combination with the sensor error characteristics and the number of targets.

[0085] S40, based on the continuous tracking information of pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, the preliminary association results of the main sensor and each auxiliary sensor for different space targets are determined by combining the vertical distance statistics of the common perpendicular line with the Hungarian algorithm.

[0086] In the embodiment of the present application, step S40 may include the following execution process:

[0087] S401. In the main sensor coordinate system and the auxiliary sensor coordinate system, calculate the unit direction vectors from the optical center of the main sensor and the optical center of the auxiliary sensor to each target respectively.

[0088] S402. In the collaborative coordinate system, based on the unit direction vectors from the optical center of the main sensor and the optical center of the auxiliary sensor pointing to each target, calculate the first direction vector of the direction measurement line and the second direction vector of the direction measurement line from the optical centers of the main sensor and the auxiliary sensor to each target respectively.

[0089] S403. In the collaborative coordinate system, based on the unit direction vectors of the direction measurement lines, calculate the foot coordinates of the common perpendicular of the first direction vector of the direction measurement line and the second direction vector of the direction measurement line.

[0090] S404. Calculate the perpendicular distance based on the foot coordinates of the common perpendicular of the first direction vector of the direction measurement line and the second direction vector of the direction measurement line.

[0091] S405. Calculate the mean and variance of the perpendicular distance within a preset time period.

[0092] S406. Based on the mean and variance of the perpendicular distance, construct the association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor within the preset time period.

[0093] S407. At the current moment, based on the maximum association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor obtained by solving using the Hungarian algorithm, construct the preliminary association result of the multiple targets between the main sensor and each auxiliary sensor.

[0094] Exemplarily, in combination with the observation information of multiple targets in the main - auxiliary sensor two - dimensional imaging, complete the identity determination of the multiple targets observed by the main sensor and the multiple targets observed by the auxiliary sensor, and form a preliminary association result. Suppose at time, targets are observed in the two - dimensional imaging plane of the main sensor. The unit direction vector from the optical center of the main sensor to in the main sensor coordinate system can be expressed in the following form:

[0095]

[0096] where

[0097]

[0098] Similarly, at time, the th auxiliary sensor observes targets in the two - dimensional imaging plane. The unit direction vector from the optical center of the auxiliary sensor to The unit direction vector can be expressed in the following form:

[0099]

[0100] Wherein,

[0101]

[0102] Combined with the above formula, the unit direction vector of the direction-finding line from the optical centers of the main and auxiliary sensors to the target in the collaborative coordinate system can be obtained and The unit direction vector of the direction-finding line can be expressed as:

[0103]

[0104] In the above formula, represents the unit direction vector of the direction-finding line from the optical center of the main sensor to the target in the main sensor coordinate system, represents the unit direction vector of the direction-finding line from the optical center of the th auxiliary sensor to the target in the auxiliary sensor coordinate system. represents the pixel coordinates in the two-dimensional imaging plane of the main sensor, represents the coordinates of the center point of the two-dimensional imaging plane of the main sensor. represents the pixel coordinates in the two-dimensional imaging plane of the th auxiliary sensor, represents the coordinates of the center point of the two-dimensional imaging plane of the th auxiliary sensor. and respectively represent the pixel sizes of the main sensor and the th auxiliary sensor, and respectively represent the focal lengths of the main sensor and the th auxiliary sensor. represents the rotation matrix for transforming the collaborative coordinate system into the sensor coordinate system, represents the rotation matrix for transforming the collaborative coordinate system into the coordinate system of the th auxiliary sensor. represents the unit direction vector of the direction-finding line from the optical center of the main sensor to the target in the collaborative coordinate system, represents the unit direction vector of the direction-finding line from the optical center of the th auxiliary sensor to the target in the collaborative coordinate system.

[0105] The direction-finding line can be calculated by the following formula and the common perpendicular line and the intersection point and in the collaborative coordinate system. Denote them as follows:

[0106]

[0107] where the unknown parameter can be obtained by solving the following system of equations:

[0108]

[0109] Combining the above formula, the perpendicular distance between the direction vector of the direction-finding line and the common perpendicular line is expressed by the following formula:

[0110]

[0111] where represents the coordinates of in the collaborative coordinate system, and represents the coordinates of in the collaborative coordinate system.

[0112] Obviously, when there is no error, if the target in the main sensor and the target in the th auxiliary sensor originate from the same target in space, the perpendicular distance of the common perpendicular line is . However, in actual situations, due to the angular measurement errors of different sensors, etc., the spatial positions of the same target under the observation perspectives of distributed sensors cannot overlap, that is . Nevertheless, the perpendicular distance of the common perpendicular line between two direction-finding lines pointing to the same target is still smaller than that between the misassociated direction-finding lines. At the same time, when considering two targets from the imaging planes of different sensors during association, if they are formed by the same target in space, the perpendicular distance of the common perpendicular line between the direction-finding lines should remain stable or concentrated over a continuous period of time. We use statistical values to measure this concept, using the variance of the perpendicular distance of the common perpendicular line to measure the stability of the perpendicular distance of the common perpendicular line over a period of time, and using the mean value

[0113] of the perpendicular distance of the common perpendicular line to measure the central tendency of the perpendicular distance of the common perpendicular line over a period of time. The above two indicators can be expressed in the following form:

[0113]

[0114] Among them, represents the time variable, represents the start time of a continuous period of time, represents the end time of a continuous period of time, represents the number of statistical frames in this continuous period of time.

[0115] Next, we introduce the Hungarian algorithm and combine the above perpendicular distance statistical value of the common perpendicular with the mean value to construct the target in the main sensor and the th auxiliary sensor The association weight of the target is as follows:

[0116]

[0117] When the target in the main sensor and the th auxiliary sensor

[0118]

[0119] Among them, , when it means that the target in the main sensor and the th target in the auxiliary sensor are formed by the same target in space. This constitutes a typical two-dimensional assignment problem. Through the above method, the preliminary association results between the main sensor and the multi-targets observed by any one of the auxiliary sensors can be constructed. As shown in the appendix Figure 8 shown, appendix Figure 8 is a schematic diagram of the method for completing the preliminary association of multi-targets between the main and auxiliary sensors by combining the continuous tracking information of multi-targets in the two-dimensional imaging plane of the sensor. Among them, within the time period when the main sensor collects five consecutive frames of images, the optical center moves from to , and the optical center of the auxiliary sensor moves from to during this period. The perpendicular distance between the direction lines pointing from the main sensor to the target and the direction lines pointing from the auxiliary sensor to the target at different times during this period are respectively , by combining the perpendicular distances of the common perpendiculars at the above-mentioned multiple moments, the mean and variance of the perpendicular distances of the common perpendiculars can be calculated, thereby constructing the correlation weights of multiple targets. . The Hungarian algorithm is represented in the form of a bipartite graph, which represents the Figure 1 preliminary correlation results obtained by the above method for the multi-target correlation problem in the scenario shown. The results shown represent the in the main sensor and the in the th auxiliary sensor are correlated. The in the main sensor and the th in the th auxiliary sensor are correlated. The in the main sensor

[0120] S50. Based on the preliminary correlation results, determine the center points of the common perpendiculars between the direction lines of the main sensor and each auxiliary sensor for different space targets. Based on the set of center points, determine the set of preliminary correlation positioning points, and cluster the set of preliminary correlation positioning points through a clustering algorithm to obtain the fusion clustering correlation positioning points of any space target at any moment.

[0121] In the embodiment of the present application, step S50 may include the following execution process:

[0122] S501. If the space targets in the main sensor and each auxiliary sensor in the preliminary correlation results are the same target, then based on the intersection coordinates of the common perpendicular of the first direction vector of the direction line and the second direction vector of the direction line pointing to any space target determined to be the same in the preliminary correlation results by the main sensor and each auxiliary sensor, calculate the center point coordinates of the common perpendicular, and based on all the center point coordinates, obtain the set of preliminary correlation positioning points of all space targets at the current moment.

[0123] S502. Cluster the set of preliminary correlation positioning points of all space targets through a clustering algorithm to obtain multiple cluster centers, and use the cluster centers as the fusion clustering correlation positioning points of any space target at any moment.

[0124] Suppose At the moment, the target in the main sensor in the preliminary correlation results and the target in the th , calculate the center point of the common perpendicular line at this moment and define it as the preliminary associated positioning point. The expression is as follows:

[0125]

[0126] Using the above method, calculate the preliminary associated positioning points of multiple master-slave sensors for multiple targets at any moment, and solve the clustering center of the preliminary associated positioning points of multiple targets at any moment through the clustering method as the fusion clustering associated positioning point of multiple targets at a certain moment.

[0127] In the same way, the direction lines of the targets associated with the master sensor can be calculated for different auxiliary sensor directions and the multiple preliminary associated positioning points of the targets in the master sensor can be calculated respectively.

[0128] Suppose the targets preliminarily associated with the rest of the auxiliary sensors are then the set of preliminary associated positioning points obtained by all auxiliary sensors and the master sensor through the preliminary association results at this moment can be calculated Combined with the clustering fusion algorithm, cluster the set of preliminary associated positioning points to obtain the clustering center of the target in the master sensor in three-dimensional space , and the formula is as follows:

[0129]

[0130] where represents the center point of the common perpendicular line between the direction line pointed by the master sensor at the th moment and the direction line pointed by the auxiliary sensor . refers to the set of preliminary associated positioning points of all directions targets obtained by fusing the master sensor and different auxiliary sensors at the th moment. is the fusion clustering associated positioning point of the target in the master sensor. Using the same method, the fusion clustering associated positioning points of any target in the master sensor can be obtained.

[0131] S60. Based on the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective motion tracks, compare with the set distance threshold to determine whether the association is correct, and combine the multi-target fusion clustering association precision and multi-target fusion clustering association recall rate indicators to determine the fusion clustering association results of multiple formation infrared sensors for multiple moving spatial targets.

[0132] In one embodiment of the present application, based on the distances between the fusion clustering associated positioning points of all spatial targets and their respective motion trajectories at any moment, compare with a set distance threshold to determine whether the association is correct, including:

[0133] Set a fixed distance threshold, calculate the distances between the fusion clustering associated positioning points of all spatial targets and their respective true motion trajectories at any moment, compare the distances with the set distance threshold, if the distance is less than the set distance threshold, then judge it as a correct association;

[0134] Otherwise, judge it as an incorrect association.

[0135] Through the above method, the fusion clustering associated positioning points of all targets observed in the main sensor can be obtained. By setting a distance threshold , if the fusion clustering associated positioning point meets the following conditions, then judge it as a correct association.

[0136]

[0137] Wherein represents the true motion trajectory of the target in space at the th moment. The meaning of the above formula indicates that if the distance between the fusion clustering associated positioning point and the true motion trajectory of the target is less than the specified distance threshold, then judge it as a correct association.

[0138] As shown in the appendix Figure 9 , the appendix Figure 9 is a schematic diagram of the method for completing multi-target association between multiple sensors by combining the fusion clustering association method in the scenario shown in the appendix Figure 1 . In Figure 9 , the formations of multiple sensors are the formation distributions optimized through multi-target associability analysis. Multiple sensors are fused to obtain the preliminary associated positioning points for a certain target in the main sensor. Combining the clustering algorithm to fuse multiple sets of preliminary associated positioning points to obtain the clustering center, that is, obtaining the fusion clustering associated positioning point for a certain target in the main sensor. By calculating the distance between the fusion clustering associated positioning point and the true motion trajectory of the target, and comparing it with the distance threshold , if the distance is less than or equal to , then judge it as a correct association.

[0139] Evaluating the association effect of the main sensor and each auxiliary sensor on spatial targets based on two indicators of multi-target fusion clustering association precision rate and multi-target fusion clustering association recall rate can include: obtaining the number of fusion clustering associated positioning points successfully associated with the true motion trajectory of the target , and the number of fusion clustering associated positioning points not successfully associated with the true motion trajectory of the target The number of target true motion traces that have not been successfully matched with any fusion clustering associated positioning points . Based on accounting for the sum of to determine the multi-target fusion clustering association precision rate; based on accounting for the sum of to determine the multi-target fusion clustering association recall rate. If both the fusion clustering association precision rate and the fusion clustering association recall rate exceed the preset threshold, the association effect is determined to be good, otherwise it is unqualified.

[0140] It should be noted that the association effect of the fusion clustering association method proposed by the present invention is judged by two indicators, namely the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate . The definitions of the two indicators are as follows:

[0141]

[0142] Among them, represents the multi-target fusion clustering association precision rate, represents the multi-target fusion clustering association recall rate, represents the number of fusion clustering associated positioning points that have been successfully matched with the target true motion traces. represents the number of fusion clustering associated positioning points that have not been successfully matched with any target true motion traces. represents the number of target true motion traces that have not been successfully matched with any fusion clustering associated positioning points.

[0143] Based on the above method embodiments, the present application further provides a multi-target fusion clustering association device between multiple infrared sensors based on associability, which is used to solve the same technical problems. The device may include a sensor classification module, an association matrix construction module, an associability judgment module, a preliminary association result determination module, a positioning point determination module, and an association determination module. Among them, the sensor classification module is used to randomly determine one of the multiple formation infrared sensors as the main sensor and the remaining infrared sensors as auxiliary sensors when the multiple formation infrared sensors cooperate to detect multiple moving space targets, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed system as the origin; the association matrix construction module is used to construct a multi-target associability matrix between the main sensor and each auxiliary sensor at the current moment in the cooperative coordinate system based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information; the associability judgment module is used to solve the multi-target associability matrix to obtain the rank of the multi-target associability matrix. If the rank of the multi-target associability matrix meets the preset conditions, it is determined that there is associability between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; the preliminary association result determination module is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane; the positioning point determination module is used to determine the center point of the common perpendicular between the direction lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association results, determine the preliminary association positioning point set based on the set of center points, and cluster the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment; the association determination module is used to compare the distance between the fusion clustering association positioning points of all space targets at any moment and their respective motion tracks with a set distance threshold to judge whether the association is correct, and determine the fusion clustering association results of the multiple formation infrared sensors for the multiple moving space targets in combination with the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate indicators.

[0144] Among them, the device embodiment corresponds to the method embodiment. Therefore, the device embodiment has the same technical effects as the method embodiment and will not be elaborated here.

[0145] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0146] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, it is described relatively simply. For the relevant parts, reference can be made to the description of the method embodiment.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multi-target fusion clustering association method between multiple infrared sensors based on associability, characterized in that, Including: When multiple formation infrared sensors cooperate to detect multiple moving space targets, randomly determine one of the multiple formation infrared sensors as the main sensor, and the remaining infrared sensors as auxiliary sensors. Taking the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin, construct a cooperative coordinate system; In the cooperative coordinate system, based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, construct the multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment; Solve the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is determined that there is a correlation between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; Based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, use a combination of the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets; Based on the preliminary association results, determine the center point of the common perpendicular between the direction lines of the main sensor and each auxiliary sensor for different space targets. Based on the set of center points, determine the set of preliminary association positioning points, and cluster the set of preliminary association positioning points through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment; Based on the distances between the fusion clustering association positioning points of all space targets at any moment and their respective motion tracks, compare with the set distance threshold to determine whether the association is correct, and combine the multi-target fusion clustering association precision and multi-target fusion clustering association recall rate indicators to determine the fusion clustering association results of multiple formation infrared sensors for multiple moving space targets.

2. The multi-target fusion clustering association method between multiple infrared sensors based on associability as described in claim 1, wherein Define the geocentric earth-fixed coordinates of the main sensor at the zero moment as the origin, and use the following formula to calculate the position of the th auxiliary sensor in the cooperative coordinate system at the moment: In the formula, is the longitude and latitude coordinates of the th auxiliary sensor at the th moment, is the position of the th auxiliary sensor in the cooperative coordinate system at the th moment, is the geocentric earth fixed coordinate of the master sensor at zero moment.

3. The multi-target fusion clustering and association method between multiple infrared sensors based on associativity as described in claim 1, characterized in that, The constructing the multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information includes: Obtain the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor, and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor. Based on the first pixel coordinates, the second pixel coordinates, and the optical center coordinates of each sensor, solve the direction lines pointing from different sensor optical centers to the first pixel coordinates and the second pixel coordinates; Based on each direction line, the internal parameters of each sensor, and its own position information, construct the multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment.

4. The multi-target fusion clustering and association method between multiple infrared sensors based on associativity according to claim 3, wherein The expression of the multi-target correlation matrix is: Where: In the formula, represents the direction vector of the direction line of the th spatial target in the main sensor coordinate system, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the auxiliary sensing coordinate system, and respectively represent the rotation matrices of the main sensor and the auxiliary sensor at the current moment, represents the direction vector of the direction line of the th target in the cooperation coordinate system of the main sensor, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the cooperation coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the imaging plane of the main sensor, represents the second pixel coordinate, represents the th central pixel coordinate of the imaging plane of the auxiliary sensor, and respectively represent the pixel size and focal length of the main sensor, and respectively represent the pixel size and focal length of the th auxiliary sensor.

5. The multi-target fusion clustering and association method between multiple infrared sensors based on associativity as described in claim 4, wherein The solving the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is determined that there is a correlation between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment includes: For the multi-target association matrix Perform singular value decomposition to obtain the rank of the multi-target correlation matrix , if , then it is determined that there is an associability between the multiple targets observed by the primary sensor and the auxiliary sensor at the current moment, where m and n respectively represent the number of spatial targets observed by the primary sensor and the number of spatial targets observed by the auxiliary sensor.

6. The multi-target fusion clustering and association method between multiple infrared sensors based on associativity as claimed in claim 3, wherein Based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, the preliminary association results of the main sensor and each auxiliary sensor for different space targets are determined by combining the perpendicular distance statistical value and the Hungarian algorithm, including: In the main sensor coordinate system and the auxiliary sensor coordinate system, calculate the unit direction vectors pointing from the optical center of the main sensor and the optical center of the auxiliary sensor to each target respectively; In the collaborative coordinate system, based on the unit direction vectors pointing from the optical center of the main sensor and the optical center of the auxiliary sensor to each target, calculate the first direction vector of the direction-finding line and the second direction vector of the direction-finding line from the optical centers of the main sensor and the auxiliary sensor to each target respectively; In the collaborative coordinate system, calculate the foot coordinates of the perpendicular line between the first direction vector of the direction-finding line and the second direction vector of the direction-finding line based on the unit direction vector of the direction-finding line; Calculate the perpendicular distance based on the foot coordinates of the perpendicular line between the first direction vector of the direction-finding line and the second direction vector of the direction-finding line; Calculate the mean and variance of the perpendicular distance within a preset time period; Based on the mean and variance of the perpendicular distance, construct the association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor within the preset time period; At the current moment, based on the maximum association weights between the multiple targets in the main sensor and the multiple targets in the auxiliary sensor obtained by solving the Hungarian algorithm, construct the preliminary association results of the multiple targets between the main sensor and each auxiliary sensor.

7. The multi-target fusion clustering and association method between multiple infrared sensors based on associativity as claimed in claim 1, wherein Based on the preliminary association results, determine the set of preliminary association positioning points of the main sensor and each auxiliary sensor for different space targets, and cluster the set of preliminary association positioning points through a clustering algorithm to obtain the fusion clustering association positioning points of any space target at any moment, including: Based on the intersection coordinates of the perpendicular lines between the first direction vector of the direction-finding line and the second direction vector of the direction-finding line pointing from the main sensor and each auxiliary sensor to the same space target judged in any preliminary association result, calculate the central point coordinates of the perpendicular line, and obtain the set of preliminary association positioning points of all space targets at the current moment based on all the central point coordinates; Cluster the set of preliminary association positioning points of all space targets through a clustering algorithm to obtain multiple cluster centers, and use the cluster centers as the fusion clustering association positioning points of any space target at any moment.

8. The multi-target fusion clustering association method between multiple infrared sensors based on associability as claimed in claim 1, wherein Based on the distances between the fusion clustering association positioning points of all space targets at any moment and their respective motion trajectories, compare with the set distance threshold to judge whether the association is correct, including: Set a fixed distance threshold, calculate the distances between the fusion clustering association positioning points of all space targets at any moment and their respective true motion trajectories, compare the distances with the set distance threshold, and if the distance is less than the set distance threshold, judge it as a correct association; Otherwise, judge it as an incorrect association.

9. The multi-target fusion clustering association method between multiple infrared sensors based on associability according to claim 1, wherein determining the fusion clustering association results of multiple formation infrared sensors for multiple moving space targets by combining the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate index includes: Obtain the number of fused clustering associated positioning points successfully associated with the target true motion trace , the number of fused clustering associated positioning points not successfully associated with the target true motion trace , the number of target true motion traces not successfully matched with any fused clustering associated positioning points ; Based on Occupying And Determine the multi-objective fusion clustering association precision rate based on the ratio of the sum values; Based on Occupying And Determine the multi-objective fusion clustering association recall rate based on the ratio of the sum values; If both the precision rate of multi-target fusion clustering association and the recall rate of multi-target fusion clustering association exceed the preset threshold, the association effect is determined to be good; otherwise, it is unqualified.

10. A multi-target fusion clustering and association device between multiple infrared sensors based on associability, characterized in that, Including: A sensor classification module, which is used to randomly determine one of the multiple formation infrared sensors as the main sensor and the remaining infrared sensors as auxiliary sensors when the multiple formation infrared sensors cooperate to detect multiple moving space targets, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-Centered Earth-Fixed (ECEF) coordinate system as the origin. An association matrix construction module, which is used to construct a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment in the cooperative coordinate system based on the pixel coordinate information of multiple space targets, their respective internal parameter information, and their own position information obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes. An association judgment module, which is used to solve the multi-target association matrix to obtain the rank of the multi-target association matrix. If the rank of the multi-target association matrix meets the preset conditions, it is determined that there is an associability between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment. A preliminary association result determination module, which is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the statistical value of the perpendicular distance of the common perpendicular line and the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane. A positioning point determination module, which is used to determine the center point of the common perpendicular line between the direction lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association results, determine the set of preliminary association positioning points based on the set of center points, and cluster the set of preliminary association positioning points through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment. An association result determination module, which is used to compare the distance between the fusion clustering association positioning points of all space targets at any moment and their respective motion tracks with the set distance threshold to judge whether the association is correct, and determine the fusion clustering association results of the multiple formation infrared sensors for the multiple moving space targets by combining the precision rate of multi-target fusion clustering association and the recall rate of multi-target fusion clustering association indicators.

Images