Multi-target fusion clustering association method among multiple infrared sensors based on relevance

Abstract

The invention discloses a correlation-based multi-target fusion clustering correlation method among multiple infrared sensors, and the method comprises the steps: decoupling a multi-target correlation problem among the multiple infrared sensors into a combination optimization problem of a plurality of main-auxiliary sensor pairs, thereby reducing the calculation complexity; by establishing a mathematical model of the multi-target association capability among multiple infrared sensors, mathematical conditions of the multi-target association capability among the infrared sensors are deduced, and a theoretical basis is provided for optimizing the sensor formation; based on continuous tracking information of multiple targets in a two-dimensional imaging plane of the sensors, preliminary association of the multiple targets between the main sensor and the auxiliary sensor is achieved; and a multi-target association result among the multiple infrared sensors is obtained by fusing the preliminary association results among the multiple main-auxiliary sensors. According to the method and the device, the accuracy and the robustness of multi-target association are ensured on the basis of ensuring the data fusion efficiency; therefore, the problem of multi-target association mismatching among the sensors of the multi-infrared sensor system in multi-target cooperative detection is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target association matching and identity determination among multiple infrared sensors, and more particularly to a multi-target fusion clustering association method, device, storage medium and equipment among multiple infrared sensors based on associability. Background Art As an important part of the modern perception system, infrared sensing technology has shown extensive application value in civil fields such as urban low-altitude security and forest fire prevention and rescue, relying on its all-weather working ability, excellent anti-electromagnetic interference performance and good concealment characteristics. Compared with visible light sensing technology, infrared sensors can maintain a stable working state in low-light or even lightless environments, significantly improving the target perception and tracking efficiency in complex scenarios such as night-time UAV navigation, heat source location in disaster areas, and wildlife night monitoring. However, single infrared sensors face technical bottlenecks such as limited detection field of view, insufficient acquisition of target feature information, and weak anti-occlusion ability in practical applications such as UAV obstacle avoidance in urban dense building areas and thermal imaging monitoring of agricultural pests and diseases. The accuracy of their target detection and tracking is easily affected by factors such as thermal radiation interference from high-temperature equipment and signal attenuation caused by vegetation occlusion.

[0002] In response to the inherent defects of single sensors, multi-infrared sensor collaborative detection schemes have emerged. This scheme realizes more comprehensive information acquisition of target objects through multi-perspective observations, effectively improving the reliability and tracking accuracy of single target detection. At the same time, the information fusion mechanism between multiple sensors can significantly reduce the risk of single-point failure and enhance the overall reliability of the system. However, in the scenario of multi-sensor collaborative detection of multiple targets, such as in high-density target scenarios such as multi-UAV formation scheduling in logistics hubs and unauthorized aircraft control at large-scale concerts, as the target density increases, the target association mismatch rate between sensors shows an exponential upward trend. A large number of association mismatches will lead to incorrect target positioning, seriously affecting the effective tracking and detection of multiple targets by the system, resulting in serious problems such as UAV path planning conflicts and misjudgment and missed judgment of intrusion targets. Especially when facing formation moving targets, if the sensor deployment configuration is unreasonable, the observed data between different sensors will lack associability, further exacerbating the association mismatch problem. Therefore, how to achieve efficient association of multi-target information obtained by multiple sensors and ensure the accuracy of multi-target identity judgment has become a key issue in improving the performance of multi-infrared sensor collaborative detection systems. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, a multi-target fusion clustering association method between multiple infrared sensors based on associability is proposed in the first aspect of the present invention, including: when multiple formation infrared sensors cooperate to detect multiple moving space targets, randomly determine one of the multiple formation infrared sensors as the main sensor, and the remaining infrared sensors as auxiliary sensors, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin; in the cooperative coordinate system, based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, construct a multi-target associability matrix between the main sensor and each auxiliary sensor at the current moment; solve the multi-target associability matrix to obtain the rank of the multi-target associability matrix. If the rank of the multi-target associability matrix meets the preset conditions, it is determined that there is associability between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane, adopt a method combining the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets; based on the preliminary association results, determine the center point of the common perpendicular between the direction lines of the main sensor and each auxiliary sensor for different space targets, determine the preliminary association positioning point set based on the set of center points, and perform clustering on the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment; based on the distance between the fusion clustering association positioning points of all space targets at any moment and their respective motion trajectories, compare with the set distance threshold to determine whether the association is correct, and combine the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate indicators to determine the fusion clustering association results of multiple formation infrared sensors for multiple moving space targets.

[0004] Optionally, define the Earth-centered Earth-fixed coordinates of the main sensor at the zero moment as the origin, and use the following formula to calculate the position of the th auxiliary sensor in the cooperative coordinate system at the

[0005] In the formula, is the longitude and latitude coordinates of the th auxiliary sensor at the moment, is the Earth-centered Earth-fixed coordinate system coordinates of the main sensor at the zero moment, is the Earth-centered Earth-fixed coordinate system coordinates of the main sensor at the zero moment.

[0006] Optionally, constructing a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment based on the pixel coordinate information, respective internal parameter information, and self-position information of multiple spatial targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes includes: obtaining the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor, and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor, and solving the direction lines pointing from the optical centers of different sensors to the first pixel coordinates and the second pixel coordinates based on the first pixel coordinates, the second pixel coordinates, and the optical center coordinates of each sensor; constructing a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment based on each direction line, the internal parameters of each sensor, and self-position information.

[0007] Optionally, the expression of the multi-target association matrix is:

[0008] Where:

[0009] In the formula, represents the direction vector of the direction line of the th spatial target in the main sensor coordinate system in the main sensor, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the auxiliary sensing coordinate system, and respectively represent the rotation matrices of the main sensor and the auxiliary sensor at the current moment, represents the direction vector of the direction line of the th target in the main sensor in the collaborative coordinate system, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the collaborative coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the main sensor imaging plane, represents the second pixel coordinate, represents the th central pixel coordinate of the auxiliary sensor imaging plane, and respectively represent the pixel size and focal length of the main sensor, and respectively represent the pixel size and focal length of the th auxiliary sensor.

[0010] Optionally, solving the multi-target association matrix to obtain the rank of the multi-target association matrix. If the rank of the multi-target association matrix meets a preset condition, it is determined that multiple targets observed by multiple infrared sensors at the current moment are associable, including: for the multi-target association matrix performing singular value decomposition to obtain the rank of the multi-target association matrix , if , it is determined that there is an associability between multiple targets observed by the main sensor and the auxiliary sensor at the current moment, where m and n respectively represent the number of spatial targets observed by the main sensor and the number of spatial targets observed by the auxiliary sensor.

[0011] Optionally, based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, the method of combining the perpendicular distance statistical value of the common perpendicular line with the Hungarian algorithm to determine the preliminary association results of the main sensor and each auxiliary sensor for different spatial targets may include: in the cooperative coordinate system, based on the unit direction vectors of the main sensor optical center and the auxiliary sensor optical center pointing to each target, respectively calculate the first direction vector of the direction line and the second direction vector of the direction line of the main sensor and the auxiliary sensor optical center pointing to each target; in the cooperative coordinate system, calculate the foot coordinates of the common perpendicular line of the first direction vector of the direction line and the second direction vector of the direction line based on the direction vector of the direction line; calculate the perpendicular distance based on the foot coordinates of the common perpendicular line of the first direction vector of the direction line and the second direction vector of the direction line; calculate the mean and variance of the perpendicular distance within a preset time period; construct the association weights between multiple targets in the main sensor and multiple targets in the auxiliary sensor within the preset time period based on the mean and variance of the perpendicular distance; at the current moment, construct the preliminary association results of multiple targets between the main sensor and each auxiliary sensor based on the maximum association weights between multiple targets in the main sensor and multiple targets in the auxiliary sensor obtained by solving the Hungarian algorithm.

[0012] Optionally, based on the preliminary association results, determining the set of preliminary association positioning points of the main sensor and each auxiliary sensor for different spatial targets, and clustering the set of preliminary association positioning points through a clustering algorithm to obtain the fused clustering association positioning points of any spatial target at any moment, including: based on the intersection coordinates of the common perpendicular line of the first direction vector of the direction line and the second direction vector of the direction line of the main sensor and each auxiliary sensor pointing to the same spatial target judged in any preliminary association result, calculate the center point coordinates of the common perpendicular line, and obtain the set of preliminary association positioning points of all spatial targets at the current moment based on all the center point coordinates; clustering the set of preliminary association positioning points of all spatial targets through a clustering algorithm to obtain multiple clustering centers, and taking the clustering centers as the fused clustering association positioning points of any spatial target at any moment.

[0013] Optionally, comparing the distance between the fusion clustering association positioning points of all spatial targets at any moment and their respective motion tracks with a set distance threshold to determine whether the association is correct, includes: setting a fixed distance threshold, calculating the distance between the fusion clustering association positioning points of all spatial targets at any moment and their respective true motion tracks, comparing the distance with the set distance threshold, if the distance is less than the set distance threshold, it is determined as a correct association; otherwise, it is determined as an incorrect association.

[0014] Optionally, determining the fusion clustering association results of multiple formation infrared sensors for multiple moving spatial targets by combining the multi-target fusion clustering association precision and the multi-target fusion clustering association recall rate indicators, includes: obtaining the number of fusion clustering association positioning points successfully associated with the true motion tracks of the targets , the number of fusion clustering association positioning points not successfully associated with the true motion tracks of the targets , the number of true motion tracks of the targets that have not been successfully matched with any fusion clustering association positioning points ; Based on accounting for and to determine the multi-target fusion clustering association precision; Based on accounting for and to determine the multi-target fusion clustering association recall rate; If both the fusion clustering association precision and the fusion clustering association recall rate exceed the preset threshold, the association effect is determined to be good, otherwise it is unqualified.

[0015] Based on the above embodiments, the second aspect of the present application further provides a multi-target fusion clustering association device among multiple infrared sensors based on associativity, including: a sensor classification module, which is used to randomly determine one of the multiple formation infrared sensors as the main sensor and the remaining infrared sensors as auxiliary sensors when the multiple formation infrared sensors cooperate to detect multiple moving space targets, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin; an association matrix construction module, which is used to construct a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment in the cooperative coordinate system based on the pixel coordinate information of multiple space targets, their respective internal parameter information, and their own position information obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes; an associativity judgment module, which is used to solve the multi-target association matrix to obtain the rank of the multi-target association matrix, and if the rank of the multi-target association matrix meets the preset conditions, it is judged that there is associativity between the multiple targets observed by the main sensor and the auxiliary sensor at the current moment; a preliminary association result determination module, which is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the perpendicular distance statistical value of the common perpendicular line with the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane; a positioning point determination module, which is used to determine the center point of the common perpendicular line between the direction-finding lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association results, determine the preliminary association positioning point set based on the set of center points, and cluster the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment; an association result determination module, which is used to compare the distance between the fusion clustering association positioning point of all space targets at any moment and their respective motion trajectories with a set distance threshold to judge whether the association is correct, and determine the fusion clustering association results of the multiple formation infrared sensors for the multiple moving space targets by combining the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate indicators.

[0016] An embodiment of the present invention provides a multi-target fusion clustering association method between multiple infrared sensors based on associability. Compared with the prior art, its beneficial effects are as follows: When multiple formation infrared sensors cooperate to detect multiple moving space targets, one of the multiple formation infrared sensors is randomly determined as the main sensor, and the remaining infrared sensors are auxiliary sensors. With the main sensor as the coordinate origin, a cooperative coordinate system is constructed. Thus, the multi-target association problem between multiple infrared sensors is decoupled into a combined optimization problem of multiple main-auxiliary sensor pairs, thereby reducing the computational complexity; In the cooperative coordinate system, based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, a multi-target association matrix between the main sensor and each auxiliary sensor at the current moment is constructed; The multi-target association matrix is solved to obtain the rank of the multi-target association matrix. If the rank of the multi-target association matrix meets the preset conditions, it is determined that the multiple targets observed by the multiple infrared sensors at the current moment have associability. Based on the established mathematical model of the multi-target association ability between multiple infrared sensors, the mathematical conditions for the multi-target association ability between infrared sensors are deduced, providing a theoretical basis for optimizing the sensor formation; Based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, a combination of the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets. Based on the continuous tracking information of multiple targets in the two-dimensional imaging plane of the sensor, the preliminary association of multiple targets between the main and auxiliary sensors is realized; Based on the preliminary association results, a set of preliminary association positioning points of the main sensor and each auxiliary sensor for different space targets is determined, and the set of preliminary association positioning points is clustered through a clustering algorithm to obtain the fusion clustering association positioning points of any space target at any moment; Based on the distance threshold between the fusion clustering association positioning points of all space targets at any moment and their respective motion trajectories, the association correctness is evaluated. If the evaluation passes, the correct association results of multiple formation infrared sensors for multiple moving space targets are determined. This application fully exploits the observation information advantages of multiple infrared sensors for multiple targets, ensuring the accuracy and robustness of multi-target association while guaranteeing the data fusion efficiency, thereby effectively solving the problem of multi-target association mismatch between sensors in the multi-target cooperative detection of a multi-infrared sensor system, providing effective support for subsequent multi-infrared sensor cooperative multi-target perception, tracking, and positioning. Brief Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic diagram of a typical scenario where multiple infrared sensors cooperate to detect multiple targets; Figure 2 It is a schematic diagram of the process of the multi - infrared - sensor multi - target fusion clustering and association method based on associability proposed by the present invention; Figure 3 Combined with Figure 1 It is a schematic diagram of constructing a cooperative coordinate system for a multi - infrared - sensor cooperative detection system in the typical scenario shown; Figure 4 Combined with Figure 1 It is a schematic diagram of a method for decoupling the multi - target association problem between multiple sensors into multi - target association problems between multiple master - slave sensors in the typical scenario shown; Figure 5 It is a schematic diagram of a method for judging whether targets between different sensors are associated by the perpendicular distance of the least common perpendicular of the bearing lines; Figure 6 It is a typical scenario where ghost points appear in the multi - target association process between a master - slave sensor; Figure 7 (a), 7(b), 7(c) are three master - slave sensor formations that do not meet the associability conditions; Figure 8 It is a schematic diagram of a method for completing the preliminary association of multi - targets between master - slave sensors by combining the continuous tracking information of multi - targets in the two - dimensional imaging plane of the sensor; Figure 9 For Figure 1 It is a schematic diagram of a method for completing the multi - target association between multiple sensors by combining the fusion clustering and association method in the scenario shown; Detailed implementation manners

[0019] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0020] This specification provides the method operation steps as described in the embodiments or flowcharts, but based on routine or non - creative labor, it may include more or fewer operation steps. When actually executed in a system or server product, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in an environment of parallel processors or multi - thread processing).

[0021] At present, the multi-target association methods between dual infrared sensors mainly rely on two mainstays: geometric constraint theory and statistical analysis theory. Such methods show good association accuracy in dual-sensor systems, but they cannot fully adapt to complex scenarios where the number of sensors and the scale of targets increase synchronously. In addition, the fundamental reason why dual-sensor association methods are difficult to directly extend to multi-sensor systems is that the increase in the data dimension of multi-sensor systems leads to a significant increase in computational complexity, and the cumulative effect of observation errors between multi-sensors will affect the association accuracy. This prompts researchers to develop more efficient multi-sensor fusion association methods for multi-infrared sensor systems, ensuring the efficiency of data fusion while guaranteeing the accuracy and robustness of multi-target association.

[0022] In the scenario of multi-infrared sensor collaborative detection of formation targets, due to the similarity in the movement trends of targets and sensors, the multi-target observation data obtained under the current multi-infrared sensor deployment configuration may not meet the association conditions. However, existing methods have not fully considered the satisfiability conditions of multi-target association between multi-infrared sensors. Therefore, it is urgent to analyze the associability between multi-targets based on the multi-target observation information of multi-infrared sensors, and construct a multi-target association matrix for mathematical description. By analyzing the rank of the multi-target association matrix under different multi-infrared sensor deployment configurations, it is judged whether the current sensor deployment configuration meets the association conditions, and then the multi-infrared sensor deployment plan is optimized to reduce the association mismatches caused by non-satisfaction of the association conditions, improve the robustness of multi-target association, and ultimately enhance the target perception, tracking, and positioning capabilities of the multi-infrared sensor collaborative detection system in complex environments.

[0023] Based on the above analysis, it is of great theoretical and application value to develop an efficient and associability-based multi-target fusion clustering association method between multi-infrared sensors. This method should fully exploit the advantages of multi-sensor observation information of multi-targets, and ensure the accuracy and robustness of multi-target association on the basis of guaranteeing the efficiency of data fusion. By solving this key technical problem, the accuracy and computational efficiency of multi-target association between multi-infrared sensors in complex scenarios can be significantly improved, providing effective support for subsequent multi-infrared sensor collaborative multi-target perception, tracking, and positioning, and providing core technical support for major livelihood fields such as low-altitude economic safety control and urban emergency response systems.

[0024] To solve the above technical problems, the present invention proposes an associability-based multi-target fusion clustering association method between multi-infrared sensors. This embodiment details the complete process steps when the present invention completes multi-target fusion clustering association between multi-infrared sensors in combination with a typical practical application scenario of four formation infrared sensors collaborating to detect four formation moving targets. As attached Figure 1In the typical scenario of cooperative detection of multiple targets by multiple infrared sensors as shown, there are four moving infrared sensors. During a period of time, four formation moving targets are observed within the common field of view of the four infrared sensors, and continuous motion trajectories of the four targets are obtained respectively in the two-dimensional imaging planes of the respective infrared sensors. The present invention assumes that the spatial coordinate positions of the sensor centroid or the sensor optical center coincide. In addition, the number of sensors described in this embodiment is not limited to the attached Figure 1 The four infrared sensors shown in the scenario cooperate to detect four targets, and it can also be extended to the cooperative detection of multiple targets by multiple infrared sensors.

[0025] Next, in combination with the attached Figure 1 The typical scenario shown, according to the specific flow steps shown in the attached Figure 2 The technical solution of the multi-target fusion clustering association method between multiple infrared sensors based on associability proposed in this application will be described in detail.

[0026] Refer to Figure 1 , this application provides a multi-target fusion clustering association method between multiple infrared sensors based on associability. This method can be executed by a processor, and the processor can belong to any terminal or server, without excessive limitation here. The multi-target fusion clustering association method between multiple infrared sensors based on associability may include: S10. When multiple formation infrared sensors cooperate to detect multiple moving space targets, randomly determine one of the multiple formation infrared sensors as the main sensor, and the remaining infrared sensors as auxiliary sensors. Taking the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin, construct a cooperative coordinate system.

[0027] It should be noted that the number of infrared sensors is at least 3, and the sensor types include but are not limited to infrared sensors.

[0028] In the infrared sensor cooperative detection system, there are multiple infrared sensor nodes. First, the processor takes any one of the infrared sensors as the main sensor, and the remaining sensors as auxiliary sensors. As shown in the attached Figure 3 Shown, the attached Figure 3 For combining the attached Figure 1 Schematic diagram of constructing a cooperative coordinate system for a multi-infrared sensor cooperative detection system in the typical scenario shown. As Figure 1As shown, the four infrared sensors are respectively designated as the main sensor, auxiliary sensor 1, auxiliary sensor 2, and auxiliary sensor 3. Secondly, the position of the main sensor at the zero moment is taken as the origin of the cooperative coordinate system of the multi-infrared sensor cooperative detection system, and a cooperative coordinate system that obeys the right-hand system is constructed. The X-axis of the cooperative coordinate system points east, the Y-axis points north, and the Z-axis is perpendicular to the ground and points upward. The purpose of establishing the cooperative coordinate system of the multi-infrared sensor cooperative detection system is to complete tasks such as observation and positioning tracking of multiple sensors in a unified coordinate system, so as to reduce the observation error and positioning error caused by the spatial registration problem, etc.

[0029] The processor takes the main sensor The moment when the target is observed as the zero moment, takes the position of the main sensor at the zero moment as the origin of the cooperative coordinate system, and assumes that the geocentric earth-fixed coordinate of the main sensor at the zero moment is , then the th auxiliary sensor The Position in the cooperative coordinate system at the moment Can be obtained through the following formula:

[0030] In the formula Is the th auxiliary sensor at the Latitude and longitude coordinates at the moment, Is the geocentric earth-fixed coordinate of the main sensor at the zero moment.

[0031] After that, the processor needs to decouple the multi-target association problem among multiple infrared sensors in the cooperative coordinate system of the multi-infrared sensor cooperative detection system into the multi-target association problem among multiple main-auxiliary sensor combinations.

[0032] The purpose of multi-target association among multiple infrared sensors is to find the correct one-to-one correspondence of targets in all association combinations, so as to judge whether two targets observed in different sensors come from the same spatial target. The maximum number of multi-target association combinations among multiple sensors Can be expressed by the following formula:

[0033] Among them, Represents the number of targets, Represents the number of sensors. The size will increase exponentially with the number of sensors and the number of targets. Therefore, the present invention decouples the multi-target association problem among multiple infrared sensors into a combinatorial optimization problem of multiple master-slave sensor pairs, only considering the multi-target association between each auxiliary sensor and the master sensor, without considering the multi-target association between each auxiliary sensor. Through this method, the number of multi-target association combinations among multiple sensors is optimized to:

[0034] Based on this, the multi-target association problem among multiple sensors can be mathematically described. Assuming that the targets observed in the master sensor are represented as a set , and the targets observed in the th auxiliary sensor are represented as a set , where . Then the multi-target association problem among multiple infrared sensors can be described as:

[0035] Among them, represents a specific target in the set, and this target is determined to be the same target as a certain target in the set. , when , it means that the target detected by the master sensor and the th target detected by the th auxiliary sensor are formed by the same target in space. represents the association weight between target and target . The above expression is not only applicable to point targets in single-frame images of different sensors in a certain frame, but also applicable to target trajectories in imaging planes of different sensors over a period of time.

[0036] As shown in the appendix Figure 4 , appendix Figure 4 is a schematic diagram of the method for decoupling the multi-target association problem among multiple infrared sensors into multi-target association problems among multiple master-slave sensors in combination with the typical scenario shown in appendix Figure 1 . Among them, represents four targets observed by the master sensor, represents four targets observed by auxiliary sensor 1, represents four targets observed by auxiliary sensor 2, represents four targets observed by auxiliary sensor 3. Figure 4 The dotted lines connecting the multi-target elements in represent the association combinations of multi-targets among multiple sensors, with a total of 48 association combination methods.

[0037] In the collaborative coordinate system, based on the pixel coordinate information of multiple spatial targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal parameter information, and their own position information, construct a multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment.

[0038] In the embodiment of the present application, step S20 may include the following execution process: S201. Obtain the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor, and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor, and solve the direction lines pointing from the optical centers of different sensors to the first pixel coordinates and the second pixel coordinates based on the first pixel coordinates, the second pixel coordinates, and the optical center coordinates of each sensor.

[0039] S202. Based on each direction line, the internal parameters of each sensor, and its own position information, construct a multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment.

[0040] In the specific execution process, for a certain target in space , assume that At time, the coordinates of the target in the collaborative coordinate system are , and at the At time, The pixel coordinates of the target in the sensor imaging plane are obtained through the following formula:

[0041] In the formula, represents the Z-axis coordinate of the target in the sensor coordinate system. represents the focal length of the sensor, represents the pixel size of the sensor, represents the center coordinate of the sensor imaging plane, represents the translation of the sensor, represents the rotation matrix of the sensor coordinate system with added noise relative to the collaborative coordinate system, which is expressed by the following formula:

[0042] In the formula:

[0043]

[0044] In the formula, respectively represent the pitch, yaw, and roll angles of the camera. respectively represent the measurement noise obeying the Gaussian distribution obtained from the angular measurement error of the sensor, , where the variance , represents the angular measurement accuracy of the passive sensor. It means that there is a 99.7% probability that the angular measurement error appears within the angular measurement accuracy range.

[0045] Let the pixel coordinates of the target in the two-dimensional imaging plane of the main sensor obtained at the th moment be , and in the two-dimensional imaging plane of the th auxiliary sensor at the same moment, the pixel coordinates of the target be .

[0046] To determine whether the targets and come from the same target in space, the processor can combine the pixel coordinates of the target to solve the direction lines pointing from the optical centers of different sensors to this pixel coordinate. As shown in Appendix Figure 5 , Appendix Figure 5 is a schematic diagram of the method for judging whether the targets between different sensors are associated by the perpendicular distance of the least common perpendicular of the direction lines. In the figure, it shows that at the th moment, the direction line from the optical center of the main sensor to the target is , and the direction line from the th auxiliary sensor to the target is . If the targets and come from the same target in space, under the ideal condition without any error, the direction lines and intersect in space, and the intersection point is the three-dimensional position where the target is located. However, due to the resolution and error conditions of the sensor, the distance between the two direction lines is always greater than zero. Therefore, the perpendicular distance of the common perpendicular between the direction lines can be used as the association criterion to determine the association weight between the targets. The smaller the perpendicular distance of the common perpendicular, the higher the probability that the current association combination is correct.

[0047] As described above, the two direction lines from the same target will intersect at a point in a certain plane in space under ideal conditions, which means that the above criterion is based on the coplanarity of the direction lines to judge whether the targets are associated. However, when there are more than one pair of coplanar direction lines, ghost points will appear. As shown in Appendix Figure 6 , Appendix Figure 6This is a typical scenario where ghost points occur during the multi-target association process between the primary and auxiliary sensors. Among them, Target 1 and Target 2 are coplanar with both the primary sensor and the auxiliary sensor. Therefore, the four direction lines pointing from different sensors to these two targets will have four intersection points in this plane. The light-colored cross marks are ghost points. In this scenario, the two targets between the two sensors are not associable, because the sensor array does not meet the associability conditions.

[0048] In the scenario where four sensors cooperate to detect four formation targets as shown in the appendix Figure 1 Since the geometric configurations of multiple targets are usually regular and the motion trends of multiple targets are basically the same, it is easier to have a sensor array that does not meet the associability conditions in this scenario. Therefore, the present invention will combine the observation information of multiple sensors and the internal parameters of the sensors to construct a multi-target association matrix for multi-targets between multiple sensors, analyze the associable conditions of multi-targets between multiple sensors, and then optimize the arrays of multiple sensors, so as to eliminate the influence of ghost points on the multi-target association effect as much as possible.

[0049] Let the set of multiple targets observed by the primary sensor at moment be , and the set of targets observed by the th auxiliary sensor be . Then, by combining the internal parameters of the sensors and the rotation matrices of each sensor, the multi-target association matrix between the primary sensor and the

[0050] s-th auxiliary sensor at the current moment can be constructed:

[0051] where: represents the direction vector of the direction line of the th spatial target in the primary sensor coordinate system in the primary sensor, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the auxiliary sensor coordinate system, and represent the rotation matrices of the primary sensor and the auxiliary sensor at the current moment respectively, represents the direction vector of the direction line of the th target in the primary sensor in the cooperative coordinate system, represents the direction vector of the direction line of the th target in the th auxiliary sensor in the cooperative coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the main sensor imaging plane, represents the second pixel coordinate, represents the central pixel coordinate of the imaging plane of the th auxiliary sensor, and represent the pixel size and focal length of the main sensor respectively, and

[0052] represent the pixel size and focal length of the

[0053] th

[0054] auxiliary sensor respectively. This multi-target correlation matrix is only related to the pixel coordinate positions of the targets among various sensors and the internal parameters of each sensor, and characterizes the correlation ability of multiple targets among multiple sensors. If S30, solve the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is determined that there is a correlation between the multiple targets observed by the main sensor and the auxiliary sensor at the current moment. m and n represent the number of spatial targets observed by the main sensor and the number of spatial targets observed by the auxiliary sensor respectively.

[0055] It should be noted that if the rank of the multi-target correlation matrix does not meet the preset conditions, it is determined that there is no correlation between the multiple targets observed by the multi-infrared sensors at the current moment.

[0056] The following Figure 7 analyzes two cases in combination with Figure 7 . Considering the case of ideal error-free conditions and all targets in the same area observed by both sensors, as shown in Appendix Figure 7 There are three main-auxiliary sensor formations that do not meet the correlation conditions, where Figure 7 (a) does not meet the longitudinal correlation condition, Figure 7 (b) does not meet the transverse correlation condition, Figure 7 (c) does not meet the vertical correlation condition.

[0057] In Appendix Figure 7In (a), at a certain moment, when there is only a difference in the Y-axis coordinate between the main sensor and the auxiliary sensor, if all the targets also have only a difference in the Y-axis coordinate, then the direction lines from all the sensors to all the targets are all in the same inclined plane. The presence of multiple direction lines in the same plane will cause the appearance of ghost points.

[0058] In the appendix Figure 7 In (b), at a certain moment, when there is only a difference in the X-axis coordinate between the main sensor and the auxiliary sensor, if all the targets also have only a difference in the X-axis coordinate, then the direction lines from all the sensors to all the targets are all in a vertical plane parallel to the Z-axis. The presence of multiple direction lines in the same plane will cause the appearance of ghost points.

[0059] In the appendix Figure 7 In (c), at a certain moment, when there is only a difference in the Z-axis coordinate between the main sensor and the auxiliary sensor, if all the targets also have only a difference in the Z-axis coordinate, then the direction lines from all the sensors to all the targets are all in a vertical plane parallel to the Z-axis. The presence of multiple direction lines in the same plane will cause the appearance of ghost points.

[0060] In summary, for the scenario of multiple infrared sensors jointly detecting multiple formation targets, the following multi-infrared sensor formations that best meet the condition of associability can be summarized: At any moment, all the infrared sensors in the multi-infrared sensor joint detection system should avoid regular formations and, as much as possible, satisfy that the three-axis spatial position coordinates of all the sensors are different.

[0061] The above analysis only considers the ideal error-free condition and the situation where all the targets in the same area are observed by the main and auxiliary sensors. When there are errors or the number of targets observed by the sensors is different, it is necessary to further analyze in combination with the error characteristics of the sensors and the number of targets.

[0062] S40. Based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, a method combining the perpendicular distance statistical value of the common perpendicular and the Hungarian algorithm is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different spatial targets.

[0063] In the embodiment of the present application, step S40 may include the following execution process: S401. In the main sensor coordinate system and the auxiliary sensor coordinate system, calculate the unit direction vectors from the optical center of the main sensor and the optical center of the auxiliary sensor to each target respectively.

[0064] S402. In the cooperative coordinate system, based on the unit direction vectors from the optical center of the main sensor and the optical center of the auxiliary sensor to each target, calculate the first direction vector of the direction line and the second direction vector of the direction line from the optical center of the main sensor and the optical center of the auxiliary sensor to each target respectively.

[0065] S403. Calculate the foot coordinates of the common perpendicular line of the first direction vector of the direction-finding line and the second direction vector of the direction-finding line based on the unit direction vector of the direction-finding line in the collaborative coordinate system.

[0066] S404. Calculate the perpendicular distance based on the foot coordinates of the common perpendicular line of the first direction vector of the direction-finding line and the second direction vector of the direction-finding line.

[0067] S405. Calculate the mean and variance of the perpendicular distance within a preset time period.

[0068] S406. Construct the correlation weight between the multi-targets in the main sensor and the multi-targets in the auxiliary sensor within a preset time period based on the mean and variance of the perpendicular distance.

[0069] S407. At the current moment, based on the maximum correlation weight between the multi-targets in the main sensor and the multi-targets in the auxiliary sensor obtained by solving using the Hungarian algorithm, construct the preliminary correlation result of the multi-targets between the main sensor and each auxiliary sensor.

[0070] Exemplarily, combining the observation information of multiple targets in the main-auxiliary sensor two-dimensional imaging, complete the identity determination of the multiple targets observed by the main sensor and the multiple targets observed by the auxiliary sensor, and form a preliminary correlation result. Suppose at moment, in the two-dimensional imaging plane of the main sensor, targets are observed. The unit direction vector from the optical center of the main sensor to in the main sensor coordinate system can be expressed in the following form:

[0071] where,

[0072] Similarly, at moment, the th auxiliary sensor observes targets in the two-dimensional imaging plane. The unit direction vector from the optical center of the auxiliary sensor to in the auxiliary sensor coordinate system can be expressed in the following form:

[0073] where,

[0074] Combining the above formula, the unit direction vector of the direction-finding line from the optical centers of the main and auxiliary sensors to the targets and in the collaborative coordinate system can be expressed as:

[0075] In the above formulas, represents the unit direction vector of the direction line pointing from the optical center of the main sensor to the target in the main sensor coordinate system, represents the unit direction vector of the direction line pointing from the optical center of the th auxiliary sensor to the target in the auxiliary sensor coordinate system. represents the pixel coordinates in the two-dimensional imaging plane of the main sensor, and represents the coordinates of the center point of the two-dimensional imaging plane of the main sensor. represents the pixel coordinates in the two-dimensional imaging plane of the th auxiliary sensor, and represents the coordinates of the center point of the two-dimensional imaging plane of the th auxiliary sensor. and respectively represent the pixel sizes of the main sensor and the th auxiliary sensor, and respectively represent the focal lengths of the main sensor and the th auxiliary sensor. represents the rotation matrix for transforming from the cooperative coordinate system to the sensor coordinate system, and represents the rotation matrix for transforming from the cooperative coordinate system to the coordinate system of the th auxiliary sensor. represents the unit direction vector of the direction line pointing from the optical center of the main sensor to the target in the cooperative coordinate system, represents the unit direction vector of the direction line pointing from the optical center of the th auxiliary sensor to the target

[0076] The perpendicular bisector and of the direction lines and in the cooperative coordinate system can be calculated through the following formula, and the intersection point of and

[0077] Among them, the unknown parameter can be obtained by solving the following system of equations:

[0078] Combined with the above formula, the direction vector of the direction-finding line is obtained and The perpendicular distance of the common perpendicular line between them is expressed by the following formula:

[0079] where represents The coordinates in the collaborative coordinate system, represents The coordinates in the collaborative coordinate system.

[0080] Obviously, when there is no error, if the target in the main sensor and the target in the th auxiliary sensor are from the same target in space, the perpendicular distance of the common perpendicular line . However, in actual situations, due to the influence of angle measurement errors of different sensors, etc., the spatial positions of the same target under the observation perspectives of distributed sensors cannot overlap, that is . Nevertheless, the perpendicular distance of the common perpendicular line between two direction-finding lines pointing to the same target is still smaller than the perpendicular distance of the common perpendicular line between misassociated direction-finding lines. At the same time, when considering two targets from the imaging planes of different sensors during association, if they are formed by the same target in space, the perpendicular distance of the common perpendicular line between the direction-finding lines should remain stable or concentrated over a continuous period of time. We use statistical values to measure this concept, using the variance of the perpendicular distance of the common perpendicular line to measure the stability of the perpendicular distance of the common perpendicular line over a period of time, and using the mean value of the perpendicular distance of the common perpendicular line to measure the central tendency of the perpendicular distance of the common perpendicular line over a period of time. The above two indicators can be expressed in the following form:

[0081] where represents the time variable, represents the start time of a continuous period of time, represents the end time of a continuous period of time, represents the statistical number of frames of this continuous period of time.

[0082] Next, we introduce the Hungarian algorithm and combine the above statistical value of the perpendicular distance of the common perpendicular line with the mean value to construct the association weight of the target in the main sensor and the target in the th auxiliary sensor, which is expressed as follows:

[0083] When the target in the main sensor and the target in the nth auxiliary sensor come from the same target, the greater the above weight. Then we can use the Hungarian algorithm to find the maximum-weight association between the trajectories observed by the two sensors and record it as the globally optimal association model, which can be expressed in the following form:

[0084] where, , when it means that the target in the main sensor and the nth target in the auxiliary sensor are formed by the same target in space. This constitutes a typical two-dimensional assignment problem. Through the above method, the preliminary association results between the multi-targets observed by the main sensor and any one of the auxiliary sensors can be constructed. As shown in the appendix Figure 8 shown, the appendix Figure 8 is a schematic diagram of the method for completing the preliminary multi-target association between the main and auxiliary sensors by combining the continuous tracking information of the multi-targets in the two-dimensional imaging plane of the sensors. Among them, within the time period when the main sensor collects five consecutive frames of images, the optical center moves from to , and the optical center of the auxiliary sensor moves from to during this period. The perpendicular distances between the direction lines pointing from the main sensor to the target and from the auxiliary sensor to the target at different times during this period are respectively . By combining the perpendicular distances of the above-mentioned multiple moments, the mean and variance of the perpendicular distances can be calculated, thereby constructing the association weight of the multi-targets. The Hungarian algorithm is represented in the form of a bipartite graph, and this bipartite graph form represents the preliminary association results obtained by the above method for the multi-target association problem in the scenario shown in the appendix Figure 1 shown. The results shown indicate that in the main sensor is associated with in the nth auxiliary sensor, in the main sensor is associated with in the nth auxiliary sensor, in the main sensor is associated with in the nth auxiliary sensor, in the main sensor is associated with in the Association

[0085] S50. Determine the center point of the common perpendicular line between the direction lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association result. Determine the preliminary association positioning point set based on the set of center points, and cluster the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning point of any space target at any moment.

[0086] In the embodiment of the present application, step S50 may include the following execution process: S501. If the space targets in the main sensor and each auxiliary sensor in the preliminary association result are the same target, calculate the center point coordinates of the common perpendicular line based on the intersection coordinates of the direction vectors of the first direction line and the second direction line of the main sensor and each auxiliary sensor pointing to the same space target determined in any preliminary association result. Obtain the preliminary association positioning point set of all space targets at the current moment based on all the center point coordinates.

[0087] S502. Cluster the preliminary association positioning point set of all space targets through a clustering algorithm to obtain multiple cluster centers, and use the cluster centers as the fusion clustering association positioning points of any space target at any moment.

[0088] Let At a certain moment, the target in the main sensor in the preliminary association result And the th The target in the auxiliary sensor comes from the same space target. Combine the intersection points And Of the common perpendicular line with the first direction line vector and the second direction line vector at this moment, calculate the center point Of the common perpendicular line at this moment and define it as the preliminary association positioning point. The expression is as follows:

[0089] Use the above method to calculate the preliminary association positioning points of multiple main-auxiliary sensor pairs for multiple targets at any moment, and solve the cluster centers of the preliminary association positioning points of multiple targets at any moment through the clustering method as the fusion clustering association positioning points of multiple targets at a certain moment.

[0090] In the same way, the direction lines of different auxiliary sensors pointing to the targets associated with the main sensor can be calculated, and multiple preliminary association positioning points of the target in the main sensor Can be calculated respectively. Of the target in the main sensor.

[0091] Let the targets preliminarily associated with In the remaining auxiliary sensors be , the preliminary associated positioning point set obtained by all auxiliary sensors and the main sensor at this moment through the preliminary association result can be calculated , combined with the clustering fusion algorithm, cluster the preliminary associated positioning point set to obtain the cluster center of the target in the three-dimensional space in the main sensor at this moment , the formula is as follows:

[0092] Among them, represents the time, the perpendicular bisector center point between the direction lines pointed by the main sensor and the auxiliary sensor . refers to all the directions obtained by fusing the main sensor and different auxiliary sensors at the time set of preliminary associated positioning points of the target. is the fusion clustering associated positioning point of the target in the main sensor. Using the same method, the fusion clustering associated positioning points of any target in the main sensor can be obtained.

[0093] S60. Based on the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective motion tracks, compare with the set distance threshold to determine whether the association is correct, and combine the multi-target fusion clustering association precision and multi-target fusion clustering association recall rate indicators to determine the fusion clustering association results of multiple formation infrared sensors for multiple moving spatial targets.

[0094] In an embodiment of the present application, based on the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective motion tracks, compare with the set distance threshold to determine whether the association is correct, including: Set a fixed distance threshold, calculate the distances between the fusion clustering associated positioning points of all spatial targets at any moment and their respective true motion tracks, compare the distances with the set distance threshold, if the distance is less than the set distance threshold, then judge it as a correct association; Otherwise, judge it as an incorrect association.

[0095] Through the above method, the fusion clustering associated positioning points of all targets observed in the main sensor can be obtained. By setting the distance threshold , if the fusion clustering associated positioning point meets the following conditions, it is judged as a correct association.

[0096]

[0097] Among them represents At the moment, the true moving point trace of the target in space. The meaning of the above formula indicates that if the distance between the fused clustering associated positioning point and the true moving point trace of the target is less than the specified distance threshold, it is judged as a correct association.

[0098] As shown in the appendix Figure 9 shown, the appendix Figure 9 is Figure 1 a schematic diagram of the method for completing multi-target association among multiple sensors by combining the fused clustering association method in the scenario shown. In Figure 9 , the formations of multiple sensors are the formation distributions optimized through multi-target associability analysis. Multiple sensors are fused to obtain the preliminary associated positioning points for a certain target in the main sensor. Combining the clustering algorithm, multiple sets of preliminary associated positioning points are fused to obtain the clustering center, that is, the fused clustering associated positioning point for a certain target in the main sensor. By calculating the distance between the fused clustering associated positioning point and the true moving point trace of the target, and comparing it with the distance threshold , if the distance is less than or equal to , it is judged as a correct association.

[0099] Evaluating the association effect of the main sensor and each auxiliary sensor on space targets based on two indicators, the multi-target fused clustering association precision rate and the multi-target fused clustering association recall rate, can include: obtaining the number of fused clustering associated positioning points successfully associated with the true moving point trace of the target , the number of fused clustering associated positioning points not successfully associated with the true moving point trace of the target , the number of true moving point traces of the target not successfully matched with any fused clustering associated positioning points . Based on accounting for the proportion of the sum of and to determine the multi-target fused clustering association precision rate; based on accounting for the proportion of the sum of and to determine the multi-target fused clustering association recall rate. If both the fused clustering association precision rate and the fused clustering association recall rate exceed the preset threshold, the association effect is determined to be good, otherwise it is unqualified.

[0100] It should be noted that the association effect of the fused clustering association method proposed by the present invention is evaluated through two indicators, the multi-target fused clustering association precision rate and the multi-target fused clustering association recall rate . The definitions of the two indicators are as follows:

[0101] Among them, Indicates the precision rate of multi-object fusion clustering association, Indicates the recall rate of multi-object fusion clustering association, Indicates the number of fusion clustering association positioning points that are successfully matched with the true moving traces of the targets. Indicates the number of fusion clustering association positioning points that are not successfully matched with any true moving traces of the targets. Indicates the number of true moving traces of the targets that are not successfully matched with any fusion clustering association positioning points.

[0102] Based on the above method embodiments, the present application further provides a multi-object fusion clustering association device between multiple infrared sensors based on associability, which is used to solve the same technical problems. The device may include a sensor classification module, an association matrix construction module, an associability judgment module, a preliminary association result determination module, a positioning point determination module, and an association determination module. Among them, the sensor classification module is used to randomly determine one of the multiple formation infrared sensors as the main sensor and the remaining infrared sensors as auxiliary sensors when the multiple formation infrared sensors cooperate to detect multiple moving space targets, and construct a cooperative coordinate system with the initial position of the main sensor at the zero moment in the Earth-centered Earth-fixed coordinate system as the origin; the association matrix construction module is used to construct a multi-object associability matrix between the main sensor and each auxiliary sensor at the current moment based on the pixel coordinate information of multiple space targets, their respective internal parameter information, and their own position information obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes in the cooperative coordinate system; the associability judgment module is used to solve the multi-object associability matrix to obtain the rank of the multi-object associability matrix. If the rank of the multi-object associability matrix meets the preset conditions, it is determined that there is associability between the multiple targets observed by the main sensor and the auxiliary sensors at the current moment; the preliminary association result determination module is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the perpendicular distance statistical value of the common perpendicular line with the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of the multiple targets obtained in the two-dimensional imaging plane; the positioning point determination module is used to determine the center point of the common perpendicular line between the direction finding lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association results, determine the preliminary association positioning point set based on the set of center points, and cluster the preliminary association positioning point set through a clustering algorithm to obtain the fusion clustering association positioning points of any space target at any moment; the association determination module is used to compare the distance between the fusion clustering association positioning points of all space targets at any moment and their respective moving traces with a set distance threshold to determine whether the association is correct, and combine the multi-object fusion clustering association precision rate and the multi-object fusion clustering association recall rate indicators to determine the fusion clustering association results of the multiple formation infrared sensors for the multiple moving space targets.

[0103] Among them, the device embodiments correspond to the method embodiments. Therefore, the device embodiments have the same technical effects as the method embodiments and will not be elaborated herein.

[0104] It should be noted that, in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0105] Each embodiment in this specification is described in a related manner. The same or similar parts among the embodiments can be referred to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the system embodiments, since they are basically similar to the method embodiments, they are described relatively simply, and the relevant parts can be referred to the description of the method embodiments.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A multi-target fusion clustering association method among multiple infrared sensors based on relatability, characterized in that: include: When multiple formation infrared sensors cooperate to detect multiple mobile space targets, one of the multiple formation infrared sensors is randomly determined to be the main sensor, and the remaining infrared sensors are auxiliary sensors, and the initial position of the main sensor at time zero in the Earth-centered Earth-fixed system is taken as the origin to construct a collaborative coordinate system; In the collaborative coordinate system, based on the pixel coordinate information of multiple spatial targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal reference information and their own position information, a multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment is constructed; The multi-target correlation matrix is ​​solved to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is judged that the multiple targets observed by the main sensor and the auxiliary sensor at the current moment are correlated; Based on the continuous tracking information of pixel coordinates of multiple targets obtained in the two-dimensional imaging plane, the statistical value of the vertical distance of the common vertical line and the Hungarian algorithm are combined to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets. Based on the preliminary association results, the center point of the common perpendicular line between the direction finding lines of the main sensor and each auxiliary sensor for different space targets is determined, and based on the set of center points, a set of preliminary associated positioning points is determined, and the set of preliminary associated positioning points is clustered by a clustering algorithm to obtain a fused clustered associated positioning point of any space target at any time. The distance between the fusion clustering association positioning points and their respective motion traces of all space targets at any moment is compared with the set distance threshold to determine whether the association is correct. The fusion clustering association results of multiple formation infrared sensors for multiple mobile space targets are determined by combining the multi-target fusion clustering association precision and multi-target fusion clustering association recall indicators.

2. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 1, characterized in that: Define the Earth-fixed coordinates of the main sensor at time zero as the origin, and use the following formula to calculate the Auxiliary sensors in The collaborative coordinate system position at the moment: In the formula, For the The auxiliary sensor is in the The latitude and longitude coordinates of the time, For the The auxiliary sensor is in the The collaborative coordinate system position at the time, The Earth-centered Earth-fixed coordinates of the primary sensor at time zero.

3. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 1, characterized in that: The method of constructing a multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal reference information and their own position information includes: Acquire the first pixel coordinates of each target in the two-dimensional imaging plane of the main sensor and the second pixel coordinates of each target in the two-dimensional imaging plane of the auxiliary sensor, and solve the direction-finding lines from the optical centers of different sensors to the first pixel coordinates and the second pixel coordinates based on the first pixel coordinates and the second pixel coordinates and the coordinates of the optical centers of each sensor; Based on each direction-finding line, the internal parameters of each sensor and its own position information, a multi-target correlation matrix between the main sensor and the auxiliary sensors at the current moment is constructed.

4. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 3, characterized in that: The expression of the multi-objective correlation matrix is: in: In the formula, Indicates the main sensor The direction vector of the direction finding line of a space target in the main sensor coordinate system, Indicates Among the auxiliary sensors The direction vector of the direction finding line of the target in the auxiliary sensor coordinate system, and Respectively represent the rotation matrices of the main sensor and auxiliary sensor at the current moment, Indicates the main sensor The direction vector of the direction finding line of each target in the collaborative coordinate system, Indicates Among the auxiliary sensors The direction vector of the direction finding line of each target in the collaborative coordinate system, represents the first pixel coordinate, represents the central pixel coordinate of the main sensor imaging plane, represents the second pixel coordinate, Indicates The central pixel coordinates of the auxiliary sensor imaging plane, and Respectively represent the pixel size and focal length of the main sensor, and Respectively represent The pixel size and focal length of the auxiliary sensor.

5. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 4, characterized in that: Solving the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix, if the rank of the multi-target correlation matrix satisfies a preset condition, then judging whether the multiple targets observed by the main sensor and the auxiliary sensor at the current moment are correlated may include: Multi-target correlation matrix Perform singular value decomposition to obtain the rank of the multi-objective correlation matrix ,like , then it is judged that the multiple targets observed by the main sensor and the auxiliary sensor at the current moment are relatable, where m and n They represent the number of space targets observed by the main sensor and the number of space targets observed by the auxiliary sensor respectively.

6. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 3, characterized in that: The pixel coordinate continuous tracking information of multiple targets obtained in the two-dimensional imaging plane is based on the combination of the common vertical line vertical distance statistics and the Hungarian algorithm to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets, including: In the main sensor coordinate system and the auxiliary sensor coordinate system, the unit direction vectors from the main sensor optical center and the auxiliary sensor optical center to each target are calculated respectively; In the collaborative coordinate system, based on the unit direction vectors of the optical center of the main sensor and the optical center of the auxiliary sensor pointing to each target, the first direction finding line direction vector and the second direction finding line direction vector of the optical center of the main sensor and the auxiliary sensor pointing to each target are calculated respectively; In the collaborative coordinate system, the foot coordinate of the common perpendicular line between the first direction finding line direction vector and the second direction finding line direction vector is calculated based on the direction finding line unit direction vector; Calculate the vertical distance based on the foot coordinates of the common perpendicular line between the first direction finding line direction vector and the second direction finding line direction vector; Calculate the mean and variance of the vertical distance within a preset time period; Building association weights between multiple targets in the main sensor and multiple targets in the auxiliary sensor within a preset time period based on the mean and variance of the vertical distances; At the current moment, based on the maximum correlation weight between the multiple targets in the main sensor and the multiple targets in the auxiliary sensors solved by the Hungarian algorithm, the preliminary correlation results of the multiple targets between the main sensor and each auxiliary sensor are constructed.

7. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 1, characterized in that: The method of determining a set of preliminary associated positioning points of the main sensor and each auxiliary sensor for different space targets based on the preliminary association result, and clustering the set of preliminary associated positioning points by a clustering algorithm to obtain a fused clustered associated positioning point of any space target at any time includes: Based on the coordinates of the intersection of the common perpendicular line of the first direction-finding line direction vector and the second direction-finding line direction vector that are judged to be the same space target in any preliminary association result pointed by the main sensor and each auxiliary sensor, the coordinates of the center point of the common perpendicular line are calculated, and based on the coordinates of all the center points, a preliminary association positioning point set of all space targets at the current moment is obtained; The preliminary associated positioning point sets of all space targets are clustered by clustering algorithm to obtain multiple cluster centers, and the cluster centers are used as the fused cluster associated positioning points of any space target at any time.

8. The method for multi-target fusion clustering association among multiple infrared sensors based on associatability as claimed in claim 1, characterized in that: The distance between the fusion clustering associated positioning points and their respective motion traces at any time based on all space targets is compared with a set distance threshold to determine whether the association is correct, including: A fixed distance threshold is set, and the distance between the fusion clustering associated positioning points of all space targets at any time and their respective real motion traces is calculated, and the distance is compared with the set distance threshold. If the distance is less than the set distance threshold, it is judged to be correctly associated; Otherwise, it is judged as an incorrect association.

9. The method for multi-target fusion clustering association among multiple infrared sensors based on associability as claimed in claim 1, wherein the combination of the multi-target fusion clustering association precision rate and the multi-target fusion clustering association recall rate index determines the fusion clustering association results of multiple formation infrared sensors for multiple mobile space targets, including: Get the number of fusion clustering associated positioning points that are successfully associated with the target's real motion traces , the number of fusion clustering associated positioning points that are not successfully associated with the target's true motion traces , the number of target real motion points that are not successfully matched with any fusion cluster associated positioning points ; based on occupy and The ratio of the sum of the values ​​determines the multi-target fusion clustering association accuracy; based on occupy and The ratio of the sum of the values ​​determines the multi-target fusion clustering association recall rate; If the multi-target fusion clustering association precision and the multi-target fusion clustering association recall both exceed the preset threshold, the association effect is judged to be good, otherwise it is unqualified.

10. A multi-target fusion clustering association device between multiple infrared sensors based on associatability, characterized in that: include: A sensor classification module is used to randomly determine one of the multiple infrared sensors in the formation as a main sensor and the other infrared sensors as auxiliary sensors when multiple infrared sensors in the formation cooperate to detect multiple mobile space targets, and to construct a cooperative coordinate system with the initial position of the main sensor at time zero in the Earth-centered Earth-fixed system as the origin; The correlation matrix construction module is used to construct a multi-target correlation matrix between the main sensor and each auxiliary sensor at the current moment in the collaborative coordinate system based on the pixel coordinate information of multiple space targets obtained by the main sensor and each auxiliary sensor in their respective two-dimensional imaging planes, their respective internal reference information and their own position information; A correlation judgment module is used to solve the multi-target correlation matrix to obtain the rank of the multi-target correlation matrix. If the rank of the multi-target correlation matrix meets the preset conditions, it is judged that the multiple targets observed by the main sensor and the auxiliary sensor at the current moment are correlated; A preliminary association result determination module is used to determine the preliminary association results of the main sensor and each auxiliary sensor for different space targets by combining the vertical distance statistics of the common vertical line with the Hungarian algorithm based on the continuous tracking information of the pixel coordinates of multiple targets obtained in the two-dimensional imaging plane; A positioning point determination module is used to determine the center point of the common perpendicular line between the direction finding lines of the main sensor and each auxiliary sensor for different space targets based on the preliminary association result, determine the preliminary associated positioning point set based on the set of center points, and cluster the preliminary associated positioning point set through a clustering algorithm to obtain the fused clustered associated positioning point of any space target at any time; The association result determination module is used to compare the distance between the fusion clustering association positioning points and their respective motion traces of all space targets at any time with the set distance threshold to determine whether the association is correct, and to determine the fusion clustering association results of multiple formation infrared sensors for multiple mobile space targets in combination with the multi-target fusion clustering association precision and multi-target fusion clustering association recall indicators.

Images