A radar photoelectric radio laser integrated target tracking monitoring system

Abstract

This invention relates to the field of radar tracking technology, specifically to a radar-electro-optical-radio-laser integrated target tracking and monitoring system. The system includes a memory and a processor. The processor executes a computer program stored in the memory to perform the following steps: using a three-dimensional Doppler velocity vector to compensate for the time difference between radar warning and camera exposure; combining the initial three-dimensional coordinates of the radar warning target and the two-dimensional position of the suspected target to predict the remaining time of image separation; obtaining the confidence level of each suspected target based on the deviation distance between the rays of the suspected target and the radar; obtaining the membership degree of the tower components of each suspected target using a global probability transfer iterative model based on the confidence level and the point cloud background occlusion distance; and determining the true target and tracking it by combining the deviation distance between the rays of each suspected target and the radar, the membership degree of the tower components, and the remaining time of image separation. This invention improves the target recognition accuracy and tracking efficiency of radar-electro-optical linkage systems.

Description

Technical Field

[0001] This invention relates to the field of radar tracking technology, and more specifically to a radar, optoelectronic, radio, and laser integrated target tracking and monitoring system. Background Technology

[0002] In routine monitoring of power transmission line corridors, industry practice typically involves using radar detection equipment for wide-area early warning, followed by guiding the pan-tilt-zoom (PTZ) camera of an electro-optical camera to the warning area for image capture, and finally using a coaxial laser rangefinder for single-point ranging comparison of suspected targets. This integrated hardware system combining radar, electro-optical, and laser technologies aims to combine the breadth of radio detection with the accuracy of laser ranging.

[0003] However, in practical engineering applications, this system faces two key technical problems: First, after the radar outputs the warning coordinates, the mechanical deflection of the servo motor inside the photoelectric gimbal has a physical delay of tens to hundreds of milliseconds. During this period, maneuvering targets in flight (such as drones and birds) have already deviated from the initial warning position. If the initial radar coordinates are directly used to map the image, a significant spatial coordinate misalignment will occur. Second, under sunlight, the insulator strings and vibration dampers mounted on the high-voltage tower will split into a large number of clustered, bright reflective patches in the camera's image recognition algorithm. Birds hovering or flying low near the tower in pure three-dimensional space have a point cloud depth that is very similar to the depth of the tower's components. Conventional distance detection logic cannot distinguish between the two. If the laser rangefinder mechanically rotates and illuminates the massive number of reflective insulator patches one by one without any focus, it will not only consume a large amount of single-point ranging hardware resources, but also cause real maneuvering intrusion targets to fly out of the camera's monitoring field of view during the long hardware detection window. Summary of the Invention

[0004] To address the issue of low target identification accuracy in existing radar-electro-optical linkage systems during target tracking in complex backgrounds, this invention aims to provide an integrated radar-electro-optical-radio-laser target tracking and monitoring system. The specific technical solution adopted is as follows: This invention provides a radar, optoelectronic, radio, and laser integrated target tracking and monitoring system. The system includes a memory and a processor. The processor executes a computer program stored in the memory to achieve the following steps: The initial three-dimensional coordinates and three-dimensional Doppler velocity vectors of radar early warning targets are obtained based on radar detection equipment, and two-dimensional images containing multiple suspected targets are acquired using a camera. Based on the coordinates of each suspected target in the two-dimensional image, the point cloud background occlusion distance corresponding to each suspected target is determined; the displacement compensation between the radar warning and the camera exposure is performed using the three-dimensional Doppler velocity vector, and the aligned three-dimensional coordinates of the radar warning target at the time of camera exposure are determined by combining the initial three-dimensional coordinates; based on the aligned three-dimensional coordinates and the position of each suspected target in the two-dimensional image, the remaining time of image separation of each suspected target is predicted. Based on the deviation distance between the rays of each suspected target and the radar, the confidence level of each suspected target is obtained; based on the confidence level and the deep adjacency matrix, the membership degree of the tower components of each suspected target is obtained by a global probability transfer iterative model. The deep adjacency matrix is ​​constructed based on the background occlusion distance of the point cloud; by combining the deviation distance between the rays of each suspected target and the radar, the membership degree of the tower components, and the remaining time of image separation, the ranging priority score of each suspected target is obtained. Based on the ranging priority scores of all suspected targets, the true target is identified and tracked.

[0005] Preferably, determining the point cloud background occlusion distance corresponding to each suspected target based on the coordinates of each suspected target in the two-dimensional image includes: The coordinates of the candidate target in the two-dimensional image are back-projected into a three-dimensional spatial ray. The intersection of the three-dimensional spatial ray with the pre-stored static three-dimensional point cloud of the power transmission corridor is obtained. The distance from the intersection point closest to the optical center of the camera to the optical center of the camera is taken as the background occlusion distance of the point cloud corresponding to the candidate target. If there is no intersection point, the background occlusion distance of the point cloud corresponding to the candidate target is set to a preset distance value. The candidate target is any suspected target.

[0006] Preferably, the step of using the three-dimensional Doppler velocity vector to compensate for the displacement between the radar warning and the camera exposure, and combining the initial three-dimensional coordinates to determine the aligned three-dimensional coordinates of the radar warning target at the time of camera exposure, includes: The time difference between the camera exposure time and the radar warning time is used as the time delay difference; The displacement compensation vector is obtained by multiplying the three-dimensional Doppler velocity vector by the time delay difference. The displacement compensation vector is then added to the initial three-dimensional coordinates of the radar warning target to obtain the aligned three-dimensional coordinates of the radar warning target at the camera exposure time.

[0007] Preferably, the step of predicting the remaining time of image separation for each suspected target based on the aligned three-dimensional coordinates and the position of each suspected target in the two-dimensional image includes: The aligned three-dimensional coordinates are extended along the direction of the three-dimensional Doppler velocity vector by a predetermined length to obtain the future three-dimensional coordinates; the aligned three-dimensional coordinates and the future three-dimensional coordinates are projected onto the image plane to obtain the current projected pixel coordinates and the future projected pixel coordinates, and the difference between the future projected pixel coordinates and the current projected pixel coordinates is used as the two-dimensional pixel velocity vector; When the magnitude of the two-dimensional pixel velocity vector is less than the preset pixel tolerance, the remaining duration of the image will be removed and assigned the preset time parameter. When the magnitude of the two-dimensional pixel velocity vector is greater than or equal to the preset pixel tolerance, for a candidate target, the two-dimensional coordinates of the center point of the candidate target are taken as the starting point, and the two-dimensional pixel velocity vector is extended outward. The first distance between the intersection of the extension line and the border of the first frame image and the starting point is obtained. The ratio of the first distance to the magnitude of the two-dimensional pixel velocity vector is taken as the remaining time of the candidate target's image departure.

[0008] Preferably, the acquisition of the deviation distance between the candidate target's ray and the radar includes: The shortest vertical line segment length of the 3D spatial ray corresponding to the aligned 3D coordinates of the candidate target is taken as the deviation distance between the ray of the candidate target and the radar.

[0009] Preferably, the step of obtaining the confidence level of each suspected target based on the deviation distance between the ray and the radar of each suspected target includes: By performing a negative correlation mapping between the ray of each suspected target and the deviation distance from the radar, the confidence level of each suspected target is obtained.

[0010] Preferably, obtaining the depth adjacency matrix includes: For any two suspected targets: If at least one of the point cloud background occlusion distances of any two suspected targets is a preset distance value, then the depth similarity weight of any two suspected targets is set to 0; if neither of the point cloud background occlusion distances of any two suspected targets is a preset distance value, then the depth similarity weight of any two suspected targets is determined based on the difference in the point cloud background occlusion distances of any two suspected targets. A deep adjacency matrix is ​​generated based on the deep similarity weights of each pair of suspected targets.

[0011] Preferably, the step of obtaining the membership degree of each suspected target tower component using a global probability transitive iterative model based on the confidence level and the depth adjacency matrix includes: Construct an initial preference vector using the confidence scores of all suspected targets; The initial preference vector and the depth adjacency matrix are substituted into the global probability transfer iterative equation for iterative calculation until the state distribution vector converges. The components corresponding to each suspected target are extracted from the converged state distribution vector as the membership degree of the tower components of each suspected target.

[0012] Preferably, the ranging priority score for each suspected target is obtained by combining the deviation distance between the rays and the radar of each suspected target, the membership degree of the tower components, and the remaining time of image separation, including: The ranging priority score of the candidate target is obtained based on the negative correlation mapping results of the deviation distance between the ray and the radar of the candidate target, the negative correlation mapping results of the remaining time of the candidate target's image detachment, and the membership degree of the candidate target's tower components.

[0013] Preferably, determining the true target based on the ranging priority scores of all suspected targets includes: The ranging priority scores of all suspected targets are sorted in descending order to generate a ranging priority score data sequence. According to the ranging priority score data sequence, the two-dimensional pixel coordinates of the suspected target corresponding to each ranging priority score in the ranging priority score data sequence are extracted sequentially. The laser rangefinder is driven to align with the spatial straight line direction corresponding to the two-dimensional pixel coordinates, and a laser pulse is released and the laser firing timestamp and laser echo distance are recorded. Obtain the radar detection range of the two nearest neighbor frames before and after the laser firing timestamp, and obtain the radar observation range estimate of the laser firing timestamp based on the time weight and the radar detection range. The first difference between the laser echo distance and the radar observation distance estimate is obtained. If the first difference is less than or equal to the radar ranging error limit, the corresponding suspected target is determined to be the real target.

[0014] The present invention has at least the following beneficial effects: The radar-electro-optical-radio-laser integrated target tracking and monitoring system provided by this invention acquires the initial three-dimensional coordinates and three-dimensional Doppler velocity vectors of the early warning target through radar detection equipment. It then combines this with two-dimensional images containing multiple suspected targets acquired by a camera to achieve collaborative processing of multi-source heterogeneous sensor data. Based on the coordinates of the suspected targets in the two-dimensional image, it determines the point cloud background occlusion distance and uses the three-dimensional Doppler velocity vector to compensate for the displacement between the radar warning and camera exposure, thereby determining the aligned three-dimensional coordinates of the radar warning target at the moment of camera exposure. This overcomes the target spatial coordinate misalignment problem caused by electromechanical delay. Furthermore, it predicts the image detachment of each suspected target from the remaining... The system calculates the range priority score by combining the deviation distance between the rays of each suspected target and the radar, and further obtains the membership degree of the tower components of each suspected target through a global probability transfer iterative model based on the confidence score and the deep adjacency matrix. This effectively distinguishes real targets from background interference in three-dimensional space. Finally, the system integrates the ray deviation distance, the membership degree of the tower components, and the remaining time of image separation to obtain the range priority score. Based on the score, the real target is determined for tracking. This improves the target recognition accuracy and tracking efficiency of the radar-electro-optical linkage system in complex backgrounds, avoids the occupation of limited range resources by invalid background targets, and ensures the timely acquisition and continuous tracking of high-priority targets. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the method performed by a radar-optoelectronic-radio-laser integrated target tracking and monitoring system provided in an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of a radar-photoelectric-radio-laser integrated target tracking and monitoring system proposed according to the present invention is provided in conjunction with the accompanying drawings and preferred embodiments.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The following description, in conjunction with the accompanying drawings, details a specific scheme for an integrated radar, optoelectronic, radio, and laser target tracking and monitoring system provided by the present invention.

[0020] An embodiment of a radar-electro-optical-radio-laser integrated target tracking and monitoring system: This embodiment proposes an integrated radar, optoelectronic, radio, and laser target tracking and monitoring system, including a memory and a processor. The processor executes a computer program stored in the memory to achieve, for example... Figure 1 The steps shown are as follows: Step S1: Based on the radar detection equipment, obtain the initial three-dimensional coordinates and three-dimensional Doppler velocity vector of the radar warning target, and use the camera to acquire two-dimensional images containing multiple suspected targets.

[0021] First, obtain the internal parameter matrix of the photoelectric camera. This matrix is... The matrix is ​​constructed such that its main diagonal elements represent the horizontal and vertical focal lengths of the camera, and the translation elements represent the horizontal and vertical pixel coordinates of the physical intersection of the camera lens optical axis and the camera sensor plane. The spatial translational and rotational deviation values ​​of the gimbal mounting base of the photoelectric camera relative to the transmission center point of the radar detection equipment are extracted. After obtaining these values, the external parameter matrix of the photoelectric camera is generated based on the coordinate system transformation datum. Specifically, the spatial translational deviation values ​​(i.e., the three-dimensional physical offset distances of the X, Y, and Z axes) are used as translation vectors; the rotational deviation values ​​(i.e., yaw, pitch, and roll angles) are substituted into the trigonometric function formulas of the rotation matrix for the corresponding coordinate axes, and the rotation matrices of the three coordinate axes are multiplied to obtain the final rotation matrix; finally, the rotation matrix and the translation vectors are combined and concatenated to form the external parameter matrix. .

[0022] In the process of using radar detection equipment to monitor and provide early warning for power transmission lines, if the radar detection equipment emits an initial alarm signal, it indicates that the radar detection equipment has detected a radar warning target. At this time, the moment when the initial alarm signal was emitted is obtained and recorded as the radar warning moment. The three-dimensional coordinates and three-dimensional Doppler velocity vector of the radar warning target at the radar warning moment are extracted from the data packet of the initial alarm signal. This three-dimensional coordinate is taken as the initial three-dimensional coordinate. Then, using this initial three-dimensional coordinates... Calculate the yaw and pitch angles required to drive the opto-gimbal motor. Specifically, calculate the initial three-dimensional coordinates. The relative three-dimensional coordinate differences between the gimbal and the geometric center point are denoted as X, Y, and Z vector components, respectively. The yaw angle required to drive the gimbal motor is calculated by taking the arctangent function value of the Y-axis component compared to the X-axis component. The pitch angle is calculated by taking the arctangent function value of the Z-axis component compared to the square root of the sum of the horizontal squares of the X and Y axes. After calculating the required yaw and pitch angles to drive the gimbal motor, the system sends a drive control command to control the gimbal to yaw and align to the corresponding angles. After aligning to the corresponding angles, the photoelectric camera mounted on the gimbal performs a shutter exposure operation, capturing the first frame image of the current field of view. This frame image is a two-dimensional image, meaning a two-dimensional image of the field of view has been acquired. Simultaneously, the time of capturing the first frame image is recorded as the camera exposure time.

[0023] After acquiring the first frame image, the image is input into the visual target detection algorithm to obtain rectangular bounding boxes of multiple suspected targets. For each rectangular bounding box, the intersection of the two diagonals of the bounding box is taken as the center point of the corresponding suspected target. A two-dimensional rectangular coordinate system is constructed with the top left corner of the first frame image as the origin, the horizontal rightward direction as the positive X-axis, and the vertical downward direction as the positive Y-axis. The two-dimensional coordinates of the center point of each suspected target in the two-dimensional rectangular coordinate system are obtained.

[0024] It should be noted that if no rectangular bounding box of a suspected target is detected, no further processing will be performed.

[0025] Step S2: Based on the coordinates of each suspected target in the two-dimensional image, determine the point cloud background occlusion distance corresponding to each suspected target; use the three-dimensional Doppler velocity vector to perform displacement compensation for the time difference between radar warning and camera exposure, and combine the initial three-dimensional coordinates to determine the aligned three-dimensional coordinates of the radar warning target at the time of camera exposure; based on the aligned three-dimensional coordinates and the position of each suspected target in the two-dimensional image, predict the remaining time of image separation of each suspected target.

[0026] Considering that photoelectric cameras may exhibit reflection when capturing two-dimensional images, it is impossible to distinguish between a stationary reflection attached to an insulator on a tower and a solitary flying object suspended in front of the tower based solely on two-dimensional images. Therefore, it is necessary to reverse the two-dimensional pixel coordinates into rays that penetrate virtual three-dimensional space and allow them to physically collide with the point cloud to extract depth values ​​that reflect the distance to real physical obstacles.

[0027] Specifically, the rotation matrix R and translation vector of the external parameter matrix obtained in step S1 are obtained. Then, using the algebraic matrix transformation formula The calculation is performed, and the result is used as the three-dimensional coordinates of the optical center of the photoelectric camera in the world coordinate system. This indicates transpose.

[0028] The following explanation uses a suspected target as an example. Other suspected targets can be processed using the method provided in this embodiment.

[0029] Any suspected target is recorded as a candidate target, and the well-known three-dimensional back projection equation is used. Construct a mapping relationship to represent the coordinates of candidate targets in a two-dimensional image. Substituting into the back projection equation, a three-dimensional spatial ray is calculated, originating from the position of the optical center of the photoelectric camera in the three-dimensional coordinate system and penetrating along the pixel direction. ,in, This represents the perpendicular distance from the candidate target to the camera's optical center plane. This represents the internal parameter matrix of the photoelectric camera. Represents the external parameter matrix. These represent the coordinate values ​​of a point in space along the X, Y, and Z axes in the coordinate system, respectively. It should be noted that in this embodiment, the coordinates of the suspected target in the 2D image are the same as the coordinates of the center point of the suspected target in the 2D image.

[0030] Furthermore, obtain three-dimensional spatial rays. The intersection point with the pre-stored static 3D point cloud of the power transmission corridor is used. The distance from the closest intersection point to the camera's optical center is taken as the background occlusion distance of the point cloud corresponding to the candidate target. If no intersection point exists, i.e., a 3D spatial ray... If no collision points are found with the static 3D point cloud of the power transmission corridor, it is determined that there is no physical obstruction behind the pixel in that direction. In this case, the background occlusion distance of the point cloud corresponding to the candidate target is set to a preset distance value. This preset distance value needs to be a relatively large value. In this embodiment, the preset distance value is [value missing]. The point cloud background occlusion distance is used to characterize the exact depth scale of physical obstructions located behind corresponding image pixels in the three-dimensional corridor space. The pre-stored static three-dimensional point cloud of the power transmission corridor is generated by the UAV inspection of the power transmission corridor before radar target tracking.

[0031] Within the tens to hundreds of milliseconds of gimbal electromechanical drive delay, the maneuver trajectory of a moving target can be reasonably approximated as first-order linear uniform motion. Next, displacement compensation extrapolation will be performed using the instantaneous Doppler velocity captured by radar to eliminate the misalignment error between radar coordinates and imaging timing caused by the electromechanical delay. Furthermore, this will be projected onto the image plane to determine the remaining buffer time before each suspected target flies out of the camera's field of view, providing a basis for subsequent ranking of ranging urgency.

[0032] Because the mechanical drive of the internal motors of the opto-gimbal takes time, there is a short time difference between the radar warning time and the actual time the camera captures the image. If this time difference is ignored and outdated radar coordinates are forcibly used for image area searching, the mapping points of maneuvering targets such as insects and aircraft will deviate from the actual imaging points in the image. This embodiment compensates for this by multiplying the three-dimensional Doppler velocity characteristics by the time delay, forcibly flattening the target position to the time slice of the camera's accurate exposure.

[0033] Specifically, the time interval between the camera exposure moment and the radar warning moment is used as the time delay difference. Utilizing the short-time linear motion assumption—that the target maintains a uniform velocity for hundreds of milliseconds—the three-dimensional Doppler velocity vector is multiplied by this time delay difference to obtain a displacement compensation vector. This displacement compensation vector is then added to the initial three-dimensional coordinates of the radar warning target to obtain the aligned three-dimensional coordinates of the radar warning target at the camera exposure moment. .

[0034] Considering that some intrusion targets may be located at the edge of the camera's field of view or possess high two-dimensional visual motion speed in the captured image, if these easily escapeable targets are not identified in advance, the limited rangefinder resources may be wasted on other stationary reflective objects in the center of the image, causing the truly dangerous targets to completely disappear from the picture. Therefore, perspective projection is needed to determine the visual motion trend of each suspected target and the countdown time before it leaves the field of view.

[0035] Specifically, the aligned three-dimensional coordinates of the radar warning target at the camera exposure time. The future three-dimensional coordinates are obtained by extending the length corresponding to a preset time along the direction of the three-dimensional Doppler velocity vector; where the preset time can be set to 1 second, and the length corresponding to the preset time is the spatial displacement span of the target in real physical space within 1 second. The internal parameter matrix of the photoelectric camera is then used. Multiply by the external parameter matrix Obtain the perspective projection matrix. Multiply the aligned 3D coordinates and future 3D coordinates of the radar warning target at the camera exposure time by this perspective projection matrix to obtain the current and future projected pixel coordinates. Project these coordinates onto the image plane to obtain the current and future projected pixel coordinates. Subtract the current projected pixel coordinates from the future projected pixel coordinates to obtain the two-dimensional pixel velocity vector.

[0036] When the magnitude of the 2D pixel velocity vector is less than the preset pixel tolerance, it indicates that the target is tending to be stationary or purely moving in depth within the image viewpoint and will not detach from the image boundaries in the short term. In this case, the remaining time of image detachment is assigned as a preset time parameter; where the preset time parameter is a relatively long time length, in this embodiment the preset time parameter is 999 seconds. When the magnitude of the 2D pixel velocity vector is greater than or equal to the preset pixel tolerance, it indicates that the target has undergone significant translation in the 2D image. In this case, for the candidate target, the 2D coordinates of the center point of the candidate target are taken as the starting point, and the extension is made outward along the direction of the 2D pixel velocity vector. The distance between the intersection of the extended line and the first frame image border (the outermost boundary line of the image) and the above starting point is obtained. This distance is recorded as the first distance, and the ratio of the first distance to the magnitude of the 2D pixel velocity vector is taken as the remaining time of image detachment for the candidate target. Through this method, the remaining time of image detachment for each suspected target can be obtained. The purpose of the preset pixel tolerance is to filter out minute changes in image displacement. When the length of the calculated 2D pixel velocity vector is too small, the target is determined to be stationary in the image viewpoint, thus avoiding division by zero errors when calculating the remaining time of image separation. The preset pixel tolerance can be 2 pixels. The shorter the remaining time of image separation, the closer the suspected target is to the edge of the image and the more violently it moves outward.

[0037] Step S3: Based on the deviation distance between the rays of each suspected target and the radar, obtain the confidence level of each suspected target; based on the confidence level and the deep adjacency matrix, use a global probability transfer iterative model to obtain the membership degree of the tower components of each suspected target. The deep adjacency matrix is ​​constructed based on the point cloud background occlusion distance; combine the deviation distance between the rays of each suspected target and the radar, the membership degree of the tower components, and the remaining time of image separation to obtain the ranging priority score of each suspected target.

[0038] In actual power transmission line monitoring scenarios, birds perched or flying low near towers (intrusion targets) and the reflections from the tower's insulators may have the same point cloud background occlusion distance in the 3D point cloud model. Therefore, the point cloud background occlusion distance alone cannot accurately distinguish between these two situations. Considering that the reflective patches of insulators and other tower components exhibit a chain-like clustering feature in the image—numerous but not connected end-to-end, and of similar depth—while birds appear as isolated and scattered, a deep adjacency graph network structure is constructed based on this. This allows local depth features to be interconnected and transmitted globally, thereby evaluating the probability of each suspected target's tower component belonging to the radar and the degree of deviation, serving as data evidence for subsequent removal of background reflective objects.

[0039] For candidate targets, the aligned three-dimensional coordinates of the radar warning target at the camera exposure time are used. The shortest vertical line segment length of the ray in three-dimensional space corresponding to the candidate target is taken as the deviation distance between the candidate target's ray and the radar. This method allows the deviation distance between the ray and the radar for each suspected target to be obtained. A negative correlation mapping is then performed on the deviation distances between the ray and the radar for each suspected target to obtain the confidence level of each suspected target.

[0040] As a specific implementation method, a specific formula for calculating the confidence level is given. The confidence level of the i-th suspected target can be expressed as: in, Let represent the confidence level of the i-th suspected target. This represents the deviation distance between the ray of the i-th suspected target and the radar. This represents the divergence tolerance constant.

[0041] The divergence tolerance constant is set based on the radar hardware beamwidth, and its example value can be 5.0 meters. The closer the deviation distance between the ray of the suspected target and the radar, the greater the confidence level of the suspected target.

[0042] Using the above method, the confidence level of each suspected target can be obtained. Further, the sum of the confidence levels of all suspected targets is calculated. The ratio of the confidence level of each suspected target to this sum is used as the confidence factor for each suspected target. The confidence factors of all suspected targets are then arranged according to their index numbers to obtain an initial preference vector. That is, the first element of the initial preference vector is the confidence factor of the first suspected target, the second element is the confidence factor of the second suspected target, the third element is the confidence factor of the third suspected target, and so on.

[0043] The following explanation uses two suspected targets as an example. The method provided in this embodiment can be used to process other pairs of suspected targets.

[0044] Specifically, for the i-th and j-th suspected targets: if at least one of the point cloud background occlusion distances of these two suspected targets is a preset distance value, then the depth similarity weight of these two suspected targets is set to 0; if neither of the point cloud background occlusion distances of these two suspected targets is a preset distance value, then the depth similarity weight of these two suspected targets is determined based on the difference in the point cloud background occlusion distances of these two suspected targets. As a specific implementation method, a specific calculation formula for the depth similarity weight in this case is given, and the depth similarity weight between the i-th and j-th suspected targets can be expressed as: in, Let represent the depth similarity weight between the i-th suspected target and the j-th suspected target. This represents the point cloud background occlusion distance of the i-th suspected target. This represents the point cloud background occlusion distance of the j-th suspected target. Indicates the absolute value sign. This indicates the preset reference value for the steel thickness of the iron tower. This represents an exponential function with the natural constant as its base.

[0045] The preset steel thickness benchmark value for the tower is used to provide an environmental tolerance scale, and its value can be set to 2.0 meters. This represents the difference in the point cloud background occlusion distance between the i-th suspected target and the j-th suspected target. The larger this difference is, the less similar the features of the i-th suspected target and the j-th suspected target are, that is, the smaller the depth similarity weight between the i-th suspected target and the j-th suspected target is.

[0046] The smaller the difference in depth between the actual occluders behind the reflective patches in two independent images, that is, the smaller the difference in the occlusion distance between the point cloud backgrounds of the two suspected targets, the greater the depth similarity weight, and the more likely the bounding boxes of the two suspected targets are attached to the same continuous physical facade of the tower.

[0047] Using the above method, we can obtain the deep similarity weights of pairwise suspected targets, constructing a dimension of... The matrix is ​​defined as follows: N represents the number of suspected targets. Then, the depth similarity weights of each pair of suspected targets are assigned to the corresponding elements in the matrix. The element in the i-th row and j-th column of the matrix represents the depth similarity weight between the i-th and j-th suspected targets. Further, for each column of the matrix, the sum of all elements in that column is calculated. The ratio between each element in that column and the sum of all elements in its column is used as the new element at the corresponding position, thus normalizing the elements in the matrix. The matrix obtained after this normalization operation is denoted as the depth adjacency matrix. Specifically, if the sum of all elements in a column is 0, meaning the denominator of the normalization formula is 0, then the result of the normalization operation on all elements in that column is directly set to 0.

[0048] After obtaining the initial preference vector and the deep adjacency matrix, the initial preference vector and the deep adjacency matrix are further combined, and iterative loops are used to encourage targets with clustering characteristics to mutually increase their scores in order to obtain accurate evaluation results.

[0049] Specifically, the initial preference vector and depth adjacency matrix are substituted into the global probability transfer iterative equation for iterative calculation until the state distribution vector converges. Before the first iteration, the system sets all elements in the initial state distribution vector to 1 / N, resulting in a state distribution vector with N elements (i.e., N vectors of 1 / N). The global probability transfer iterative equation can be expressed as: in, This represents the state distribution vector after iteration. Represents the initial preference vector. This represents the state distribution vector before the iteration. Indicates the damping coefficient. This represents the depth adjacency matrix.

[0050] The damping coefficient is used to adjust the ratio of local injection to global diffusion, and its value can be set to 0.85, a common constant in graph theory algorithms. This represents the basic independent confidence probability that the system forcibly supplements in each iteration based on the degree to which the target ray approaches the radar predicted coordinates. This indicates that the probability of the current round is distributed proportionally through the matrix edges to the locations of neighboring targets with similar depths.

[0051] The state distribution vector reflects the global probability convergence result. Under the control of this iterative equation, multiple iterations are performed. In each round of computation, the magnitude changes of the state distribution vector before and after the update are compared. When the difference between the magnitudes of the two state distribution vectors before and after the update is less than a preset difference threshold for the first time, it is determined that the probability propagation of the network in this round of iteration has reached a Markov convergence steady state, and the loop iteration is terminated. The state distribution vector after this round of iteration is taken as the converged state distribution vector. The components corresponding to each suspected target are extracted from the converged state distribution vector as the membership degree of the tower components of each suspected target. The preset difference threshold can be set to... .

[0052] Considering the time consumed by the single electromechanical turning and laser transmission and reception of the coaxial laser rangefinder, the system cannot simultaneously measure a large number of suspected reflective targets in the image. Next, based on the deviation distance between the rays of the suspected targets and the radar, the membership degree of the tower components, and the remaining time of image separation, a comprehensive evaluation of the priority of resource acquisition for all disordered targets in the image is performed to obtain the ranging priority score of each suspected target.

[0053] Next, we will take the candidate target as an example to illustrate the following. Based on the negative correlation mapping results of the deviation distance between the candidate target's ray and the radar, the negative correlation mapping results of the remaining time of the candidate target's image separation, and the membership degree of the candidate target's tower components, we obtain the ranging priority score of the candidate target.

[0054] As a specific implementation method, a specific formula for calculating the ranging priority score is given. The ranging priority score of the i-th suspected target can be expressed as: in, This represents the ranging priority score of the i-th suspected target. This represents the deviation distance between the ray of the i-th suspected target and the radar. This represents the membership degree of the tower component representing the i-th suspected target. This indicates that the parameters are being adjusted. This indicates the remaining time after the image of the i-th suspected target leaves the frame; This represents system constant parameters.

[0055] The system constant parameter represents the average time of the baseline hardware operation consumed in a single laser deflection and ranging operation; its value can be set as an empirical value. Seconds. The adjusted parameter is a very small safety constant used to prevent the computer from throwing a zero-denominator error when performing division operations; its value is set to [value missing]. .

[0056] The negative correlation mapping result represents the deviation distance between the ray of the i-th suspected target and the radar. The closer the direction of the back projection ray of the i-th suspected target matches the radar warning direction, the larger its value and the higher the basic score. The graph theory network is introduced to calculate the background cluster reflection. When the i-th suspected target is in a cluster of bright interference such as tower insulators, the membership degree of the tower components of the i-th suspected target is relatively large. Through division operation, the ranging priority score of the i-th suspected target is reduced, thereby depriving this type of background interference of the opportunity to preempt the ranging hardware. The negative correlation mapping result represents the remaining time of the image of the i-th suspected target leaving the frame. The shorter the remaining time of the image of the i-th suspected target leaving the frame, the less the numerical decay of this negative exponential term is, and the closer the negative correlation mapping result is to 1. The introduction of this term forces intervention in the system ranking: when a bird or aircraft about to escape the field of view appears at the edge of the frame, the system can give it a higher ranging priority score.

[0057] Using the above methods, the ranging priority score for each suspected target can be obtained.

[0058] Step S4: Based on the ranging priority scores of all suspected targets, determine the real target and begin tracking.

[0059] In step S3 of this embodiment, the ranging priority score of each suspected target is obtained. In this step, the ranging priority scores of all suspected targets are sorted in descending order to generate a ranging priority score data sequence.

[0060] Laser rangefinders can only verify whether an object objectively exists at a certain distance in the indicated direction. To definitively confirm that the object is an intruding radar tracking target, it is necessary to compare the depth of the emitted laser with the radar depth. Since radar detection data is output in a low-frequency, periodic manner, it is impossible to directly obtain radar detection values ​​that perfectly coincide with the microsecond-level instantaneous output of the laser. To eliminate comparison errors caused by clock misalignment in heterogeneous sensor hardware, this embodiment constructs a historical data interpolation mechanism to implement synchronous verification.

[0061] Specifically, the system sequentially extracts the two-dimensional pixel coordinates of the suspected target corresponding to each ranging priority score in the ranging priority score data sequence. It then calculates the angle control command required for the beam to deflect towards the spatial line containing that two-dimensional pixel. This command is sent to the servo drive motor of the coaxial laser rangefinder, controlling the rangefinder's emitting end face to rotate and align with the line. After alignment, the rangefinder is triggered to release a single-beam detection laser pulse, and the timestamp of this pulse release is recorded as the laser firing timestamp. The coaxial laser rangefinder receives the photon echo signal reflected back from the surface of an object and calculates the laser echo distance of the corresponding verification object based on the speed of light and the round-trip time difference. .

[0062] Then, the system obtains the laser firing timestamp. The radar detection range (physical distance detection value) of the two nearest neighboring frames is used as the basis for the time weighting and the radar detection range of these two frames to obtain the estimated radar observation range value of the laser firing timestamp. Specifically, the timestamp and radar detection range of the previous radar frame, as well as the timestamp and radar detection range of the subsequent radar frame, are extracted. The first time difference is obtained by subtracting the timestamp of the previous radar frame from the laser firing timestamp. The second time difference is obtained by subtracting the timestamp of the previous radar frame from the timestamp of the subsequent radar frame. The time weight is calculated by dividing the first time difference by the second time difference. The radar detection range of the previous radar frame is subtracted from the radar detection range of the subsequent radar frame, and the resulting distance difference is multiplied by the time weight. This product is then summed with the radar detection range of the previous radar frame to obtain the estimated radar observation range. The laser echo range is then calculated. Estimated radar observation range The absolute value of the difference between the two values ​​is recorded as the first difference. If the first difference is greater than the radar ranging error limit, it indicates that the current two-dimensional pixel position does not meet the depth coincidence condition, and its true identity is an accidental leaf interference or visual false alarm. The above judgment is made on the two-dimensional pixel coordinates of the suspected target corresponding to the next ranging priority score in the ranging priority score data sequence. If the first difference is less than or equal to the radar ranging error limit, it indicates that the hardware reflector actually hit by the laser pulse is in a three-dimensional depth position that matches the maneuvering object captured by radio. At this time, the corresponding suspected target is determined to be the real target. The system immediately sends a forced interrupt command to the servo drive motor to terminate the analysis of the two-dimensional pixel coordinates of the suspected targets corresponding to all remaining ranging priority scores in the ranging priority score data sequence. The system inputs the maximum focal length execution signal to the zoom control mechanism of the photoelectric camera. The zoom control mechanism drives the lens group to shrink the lens field of view to the telephoto tracking state. The system assigns the two-dimensional pixel coordinates of the real target to the visual target detection module in the computer and controls the module to continuously and stably output the target tracking bounding box in the telephoto view. The radar ranging error limit can be set to 3.5 meters.

[0063] Thus, the radar-optical-radio-laser integrated target tracking and monitoring system provided in this embodiment has achieved target tracking and monitoring.

[0064] The radar-electro-optical-radio-laser integrated target tracking and monitoring system provided in this embodiment acquires the initial three-dimensional coordinates and three-dimensional Doppler velocity vectors of the early warning target through radar detection equipment. It then combines this with two-dimensional images containing multiple suspected targets acquired by a camera to achieve collaborative processing of multi-source heterogeneous sensor data. Based on the coordinates of the suspected targets in the two-dimensional images, it determines the point cloud background occlusion distance and uses the three-dimensional Doppler velocity vectors to compensate for the time difference between radar warning and camera exposure, thereby determining the aligned three-dimensional coordinates of the radar warning target at the moment of camera exposure. This overcomes the target spatial coordinate misalignment problem caused by electromechanical delay. Furthermore, it predicts the image detachment of each suspected target from the remaining... The system calculates the range priority score by combining the deviation distance between the rays of each suspected target and the radar, and further obtains the membership degree of the tower components of each suspected target through a global probability transfer iterative model based on the confidence score and the deep adjacency matrix. This effectively distinguishes real targets from background interference in three-dimensional space. Finally, the system integrates the ray deviation distance, the membership degree of the tower components, and the remaining time of image separation to obtain the range priority score. Based on the score, the real target is determined for tracking. This improves the target recognition accuracy and tracking efficiency of the radar-electro-optical linkage system in complex backgrounds, avoids the occupation of limited range resources by invalid background targets, and ensures the timely acquisition and continuous tracking of high-priority targets.

[0065] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A radar-electro-optical-radio-laser integrated target tracking and monitoring system, comprising a memory and a processor, characterized in that, The processor executes the computer program stored in the memory to perform the following steps: The initial three-dimensional coordinates and three-dimensional Doppler velocity vectors of radar early warning targets are obtained based on radar detection equipment, and two-dimensional images containing multiple suspected targets are acquired using a camera. Based on the coordinates of each suspected target in the two-dimensional image, the point cloud background occlusion distance corresponding to each suspected target is determined; The displacement compensation between the time difference between radar warning and camera exposure is performed by using the three-dimensional Doppler velocity vector, and the aligned three-dimensional coordinates of the radar warning target at the time of camera exposure are determined by combining the initial three-dimensional coordinates. Based on the aligned 3D coordinates and the positions of each suspected target in the 2D image, the remaining time of the suspected target's departure from the scene is predicted. Based on the deviation distance between the rays of each suspected target and the radar, the confidence level of each suspected target is obtained; based on the confidence level and the deep adjacency matrix, the membership degree of the tower components of each suspected target is obtained by a global probability transfer iterative model. The deep adjacency matrix is ​​constructed based on the background occlusion distance of the point cloud; by combining the deviation distance between the rays of each suspected target and the radar, the membership degree of the tower components, and the remaining time of image separation, the ranging priority score of each suspected target is obtained. Based on the ranging priority scores of all suspected targets, the true target is identified and tracked.

2. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 1, characterized in that, The step of determining the point cloud background occlusion distance corresponding to each suspected target based on the coordinates of each suspected target in the two-dimensional image includes: The coordinates of the candidate target in the two-dimensional image are back-projected into a three-dimensional spatial ray. The intersection of the three-dimensional spatial ray with the pre-stored static three-dimensional point cloud of the power transmission corridor is obtained. The distance from the intersection point closest to the optical center of the camera to the optical center of the camera is taken as the background occlusion distance of the point cloud corresponding to the candidate target. If there is no intersection point, the background occlusion distance of the point cloud corresponding to the candidate target is set to a preset distance value. The candidate target is any suspected target.

3. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 1, characterized in that, The method of using three-dimensional Doppler velocity vectors to compensate for the time difference between radar warning and camera exposure, and combining the initial three-dimensional coordinates to determine the aligned three-dimensional coordinates of the radar warning target at the moment of camera exposure, includes: The time difference between the camera exposure time and the radar warning time is used as the time delay difference; The displacement compensation vector is obtained by multiplying the three-dimensional Doppler velocity vector by the time delay difference. The displacement compensation vector is then added to the initial three-dimensional coordinates of the radar warning target to obtain the aligned three-dimensional coordinates of the radar warning target at the camera exposure time.

4. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 2, characterized in that, The method of predicting the remaining time of image separation for each suspected target based on aligned 3D coordinates and the position of each suspected target in the 2D image includes: The aligned three-dimensional coordinates are extended along the direction of the three-dimensional Doppler velocity vector by a predetermined length to obtain the future three-dimensional coordinates; the aligned three-dimensional coordinates and the future three-dimensional coordinates are projected onto the image plane to obtain the current projected pixel coordinates and the future projected pixel coordinates, and the difference between the future projected pixel coordinates and the current projected pixel coordinates is used as the two-dimensional pixel velocity vector; When the magnitude of the two-dimensional pixel velocity vector is less than the preset pixel tolerance, the remaining duration of the image will be removed and assigned the preset time parameter. When the magnitude of the two-dimensional pixel velocity vector is greater than or equal to the preset pixel tolerance, for a candidate target, the two-dimensional coordinates of the center point of the candidate target are taken as the starting point, and the two-dimensional pixel velocity vector is extended outward. The first distance between the intersection of the extension line and the border of the first frame image and the starting point is obtained. The ratio of the first distance to the magnitude of the two-dimensional pixel velocity vector is taken as the remaining time of the candidate target's image departure.

5. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 2, characterized in that, The acquisition of the deviation distance between the candidate target's ray and the radar includes: The shortest vertical line segment length of the 3D spatial ray corresponding to the aligned 3D coordinates of the candidate target is taken as the deviation distance between the ray of the candidate target and the radar.

6. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 1, characterized in that, The confidence level of each suspected target is obtained based on the deviation distance between the ray and the radar for each suspected target, including: By performing a negative correlation mapping between the ray of each suspected target and the deviation distance from the radar, the confidence level of each suspected target is obtained.

7. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 2, characterized in that, The acquisition of the depth adjacency matrix includes: For any two suspected targets: If at least one of the point cloud background occlusion distances of any two suspected targets is a preset distance value, then the depth similarity weight of any two suspected targets is set to 0; if neither of the point cloud background occlusion distances of any two suspected targets is a preset distance value, then the depth similarity weight of any two suspected targets is determined based on the difference in the point cloud background occlusion distances of any two suspected targets. A deep adjacency matrix is ​​generated based on the deep similarity weights of each pair of suspected targets.

8. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 1, characterized in that, The step of obtaining the membership degree of each suspected target tower component using a global probability transitive iterative model based on confidence level and depth adjacency matrix includes: Construct an initial preference vector using the confidence scores of all suspected targets; The initial preference vector and the depth adjacency matrix are substituted into the global probability transfer iterative equation for iterative calculation until the state distribution vector converges. The components corresponding to each suspected target are extracted from the converged state distribution vector as the membership degree of the tower components of each suspected target.

9. A radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 2, characterized in that, The ranging priority score for each suspected target is obtained by combining the deviation distance between the rays and the radar, the membership degree of the tower components, and the remaining time of image separation. This score includes: The ranging priority score of the candidate target is obtained based on the negative correlation mapping results of the deviation distance between the ray and the radar of the candidate target, the negative correlation mapping results of the remaining time of the candidate target's image detachment, and the membership degree of the candidate target's tower components.

10. The radar-optoelectronic-radio-laser integrated target tracking and monitoring system according to claim 1, characterized in that, The process of determining the true target based on the ranging priority scores of all suspected targets includes: The ranging priority scores of all suspected targets are sorted in descending order to generate a ranging priority score data sequence. According to the ranging priority score data sequence, the two-dimensional pixel coordinates of the suspected target corresponding to each ranging priority score in the ranging priority score data sequence are extracted sequentially. The laser rangefinder is driven to align with the spatial straight line direction corresponding to the two-dimensional pixel coordinates, and a laser pulse is released and the laser firing timestamp and laser echo distance are recorded. Obtain the radar detection range of the two nearest neighbor frames before and after the laser firing timestamp, and obtain the radar observation range estimate of the laser firing timestamp based on the time weight and the radar detection range. The first difference between the laser echo distance and the radar observation distance estimate is obtained. If the first difference is less than or equal to the radar ranging error limit, the corresponding suspected target is determined to be the real target.

Images