Multi-sensor target trajectory fusion modeling method, fusion method, device and storage medium

Abstract

The application discloses a multi-sensor target trajectory fusion modeling method, a fusion method, equipment and a storage medium. The modeling method comprises the following steps: acquiring a plurality of trajectory pairs and a target real trajectory corresponding to each trajectory pair; performing angle expansion on a radar target trajectory and an optoelectronic target trajectory of each trajectory pair; constructing a trajectory fusion model, wherein the trajectory fusion model comprises a first self-attention mechanism encoder, a second self-attention mechanism encoder, an adaptive weighted fusion module and a self-attention mechanism decoder; taking the trajectory pair as input and taking the target real trajectory as a label, training and verifying the trajectory fusion model to obtain a target fusion model; in the training process, calculating a multi-order difference weighted loss according to a predicted fusion trajectory output by the trajectory fusion model and the target real trajectory, and adjusting parameters of the trajectory fusion model according to the multi-order difference weighted loss. The application can significantly improve the correctness and accuracy of trajectory tracking in a multi-target complex scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sensor target tracking, and particularly relates to a multi-sensor target trajectory fusion modeling method based on space-time cooperation and neural network, a fusion method, equipment and a storage medium. BACKGROUND

[0002] The existing multi-target tracking algorithm based on images usually adopts the following technical path: first, the target position is obtained in the pixel coordinate system through a target detection algorithm, then multi-target tracking is implemented, and finally the tracking result is converted to the spherical coordinate system. This scheme basically meets the demand in the conventional camera application scene, but when dealing with complex working conditions such as wide dynamic adjustment range of focal length and drastic change of rotation angle of large gimbal cameras, it cannot efficiently handle the camera rotation and focal length change, and there are prominent problems such as difficulty in re-association after target loss and significant decrease in tracking accuracy.

[0003] For unmanned aerial vehicle tracking, track association is an important step in multi-sensor information fusion, and its purpose is to find the track set of the same target in the information from different sensors. Due to the influence of device errors, noise, obstacles and other factors in the process of collecting target information by sensors, there is a certain difference between the detection value and the true value of the target, which reduces the target tracking accuracy. Through the track association technology, the multiple observation values of the same target can be found by integrating the data of multiple sensors, and then the relationship between the sensor detection value and the true value of the target can be determined, so the track association is an important prerequisite for track fusion.

[0004] The existing track association methods can be mainly divided into two categories: one is the statistical method, and the other is the fuzzy mathematics method. In terms of the statistical method, weighted statistical distance test method, modified weighted statistical distance test method, nearest neighbor method, etc. are usually used to process track association problems; in terms of the fuzzy mathematics method, it usually includes fuzzy double threshold algorithm, fuzzy comprehensive function algorithm and fuzzy comprehensive decision algorithm, etc. These track association methods still have some problems:

[0005] Firstly, the statistical distance test rule or the fuzzy mathematics membership function is artificially preset, the feature engineering design and the threshold setting need to be based on prior experience, and it is difficult to adapt to the association requirements of dynamic scenes;

[0006] Secondly, when the targets are densely distributed or the tracks intersect and branch, the association logic based on instantaneous state or local segment lacks the global modeling ability of historical time sequence information, resulting in failure to capture the continuous motion mode and obvious decrease in effect;

[0007] Thirdly, asynchronous track processing needs to rely on time registration operation, which not only increases the calculation complexity, but also introduces additional errors due to interpolation or extrapolation process, reducing the association reliability;

[0008] Fourthly, the algorithm is not robust to systematic bias of sensor system, high-density target motion conflict and other nonlinear interference, and deviation of statistical assumptions (such as noise distribution) can easily cause false association. The complex optimization strategy required to improve accuracy often accompanies a sharp increase in computational load, making it difficult to balance real-time and accuracy requirements.

[0009] Track fusion can obtain more accurate and reliable global estimation results than any single sensor by synthesizing a series of multi-source tracks of the same target, and can achieve higher precision tracking and measurement of the target, providing a technical basis for unmanned aerial vehicle tracking.

[0010] There are mainly three types of track fusion algorithms: weighted fusion-based fusion algorithm, Kalman filter-based fusion algorithm and data-driven fusion algorithm. The weighted fusion-based track fusion algorithm sets different weights for different local tracks and generates a fused track by weighted summation. There are many ways to calculate the weight, for example, variance weighting uses the measurement error variance of the sensor to generate the weight of the local track. When the variance estimation is accurate, the variance weighting fusion accuracy reaches the upper limit of the weighted fusion algorithm. Although the accuracy is improved by assigning weights through variance, its performance is highly dependent on the prior accuracy of the sensor measurement error variance, and the static weight mechanism cannot adapt to the target dynamic motion or sudden change of sensor characteristics, resulting in strong parameter sensitivity and weak robustness of the fusion result.

[0011] The Kalman filter-based fusion algorithm uses the powerful state estimation capability of Kalman filter to estimate the fused track as a state variable, thereby realizing track fusion. The Kalman filter-based fusion algorithm mainly includes two types: (1) by merging multi-sensor data, increasing the dimension of the observation matrix of the Kalman filter, thereby realizing the fusion of multi-sensor track; (2) first fuse multi-source measurement data based on the least mean square error criterion, and then use this fused measurement to estimate the state vector, the dimension of the observation matrix remains unchanged. However, when implementing track fusion through Kalman filter, it is necessary to accurately define parameters such as target motion model, system noise covariance and measurement noise covariance, otherwise it will greatly affect the performance of the algorithm; in actual application, model mismatch (such as non-Gaussian noise, nonlinear motion) can easily cause state estimation bias accumulation or even divergence problem, and the expansion of the dimension of the multi-sensor observation matrix or the measurement pre-fusion strategy can further increase the computational complexity and error transmission risk.

[0012] Data-driven track fusion mainly uses deep learning methods to achieve. In recent years, deep learning has been deeply researched and applied in the fields of image processing, speech recognition, text processing and filter optimization design, and in some application scenarios, it has basically surpassed traditional algorithms, but it is less applied in track association and fusion. And the conventional convolution-based feature extraction network is limited by the linear hypothesis and the local optimal criterion, and lacks the modeling ability of long-range time sequence association, nonlinear noise interference and deep coupling relationship of multi-source heterogeneous data, especially in the dense target intersection, track fragment missing or asynchronous heterogeneous sensor cooperation scene, it is difficult to achieve global optimal estimation through artificial rules. SUMMARY

[0013] The purpose of the present application is to provide a multi-sensor target track fusion modeling method, a fusion method, a device and a storage medium, to solve the problems of insufficient track accuracy and weak anti-interference ability caused by the lack of single sensor data dimension in complex environments, and the problems of poor generalization in complex scenes of traditional track association and fusion algorithms relying on the latest target track state, and the need for accurate estimation of target motion model, system noise and measurement noise covariance.

[0014] The present application solves the above technical problems through the following technical solutions: a multi-sensor target track fusion modeling method, comprising:

[0015] obtaining a plurality of track pairs and a target real track corresponding to each track pair; wherein each track pair comprises an associated radar target track and an associated photoelectric target track;

[0016] angle-expanding the radar target track and the photoelectric target track of each track pair respectively;

[0017] constructing a track fusion model; wherein the track fusion model comprises a first self-attention mechanism encoder, a second self-attention mechanism encoder, an adaptive weighted fusion module and a self-attention mechanism decoder; the first self-attention mechanism encoder is used for feature extraction of the angle-expanded radar target track to obtain radar target high-dimensional track features; the second self-attention mechanism encoder is used for feature extraction of the angle-expanded photoelectric target track to obtain photoelectric target high-dimensional track features; the adaptive weighted fusion module is used for adaptive weighted fusion of the radar target high-dimensional track features and the photoelectric target high-dimensional track features to obtain weighted fusion features; the self-attention mechanism decoder is used for decoding the weighted fusion features to obtain a predicted fusion track;

[0018] The trajectory fusion model is trained and verified by taking the trajectory pair as input and taking the corresponding target real trajectory as label to obtain a target fusion model; in the training process, a multi-order difference weighted loss is calculated according to the predicted fusion trajectory output by the trajectory fusion model and the target real trajectory, and the parameters of the trajectory fusion model are adjusted according to the multi-order difference weighted loss.

[0019] Further, the specific process of obtaining the plurality of trajectory pairs comprises:

[0020] obtaining a radar target trajectory and an optoelectronic target trajectory;

[0021] respectively pre-processing the radar target trajectory and the optoelectronic target trajectory;

[0022] respectively extracting features of the pre-processed radar target trajectory and the optoelectronic target trajectory by using a first self-attention mechanism backbone network and a second self-attention mechanism backbone network sharing parameters to obtain a radar target feature vector and an optoelectronic target feature vector;

[0023] calculating the similarity of the radar target feature vector and the optoelectronic target feature vector to obtain a similarity matrix;

[0024] determining the trajectory pair according to the similarity matrix.

[0025] Further, the specific process of obtaining the radar target trajectory comprises:

[0026] obtaining radar target point data, the radar target point data comprising azimuth angle, elevation angle, distance and speed of the radar target point;

[0027] pre-processing and filtering the radar target point data, and converting the radar target point data in a spherical coordinate system to a Cartesian coordinate system;

[0028] generating a current state vector according to the radar target point data in the Cartesian coordinate system; wherein the current state vector comprises the position and speed of the radar target point;

[0029] designing a Kalman filter; wherein the state transition model of the Kalman filter adopts a uniform motion model, and the observation model is used to observe the position of the radar target point;

[0030] taking the current state vector as the input of the Kalman filter, and performing state estimation by using the Kalman filter to obtain an estimated position of the radar target point;

[0031] converting the estimated position of the radar target point to a spherical coordinate system to obtain the azimuth angle, elevation angle and distance of the radar target point at the next time, and further obtaining the radar target trajectory.

[0032] Further, the specific acquisition process of the photoelectric target trajectory is as follows:

[0033] Acquire a photoelectric target point detection box;

[0034] Convert the photoelectric target point detection box in the pixel coordinate system to the spherical coordinate system to obtain angle information of the photoelectric target point; wherein the angle information of the photoelectric target point includes an azimuth angle and a pitch angle of the photoelectric target point in the spherical coordinate system;

[0035] Take the azimuth angle, the pitch angle, the azimuth angle velocity and the pitch angle velocity of the photoelectric target point as inputs of a photoelectric target point tracking algorithm, take an angle distance as a distance measurement of the photoelectric target point tracking algorithm, and obtain the azimuth angle and the pitch angle of the photoelectric target point in the spherical coordinate system at the next moment by using the photoelectric target point tracking algorithm, and further obtain the photoelectric target trajectory.

[0036] Further, the specific calculation formula of the angle distance is as follows:

[0037] ;

[0038] wherein, denotes an angle distance between the i th photoelectric target point and the j th photoelectric target point, ; and denote the azimuth angle and the pitch angle of the i th photoelectric target point respectively; and denote the azimuth angle and the pitch angle of the j th photoelectric target point respectively.

[0039] Further, the radar target trajectory and the photoelectric target trajectory are respectively preprocessed, and the specific preprocessing includes:

[0040] Convert the azimuth angle and the pitch angle in the radar target trajectory into radians respectively, then calculate the sine and cosine of each radian respectively, and form a radar target input feature by the sine and cosine of each radian;

[0041] Convert the azimuth angle and the pitch angle in the photoelectric target trajectory into radians respectively, then calculate the sine and cosine of each radian respectively, and form a photoelectric target input feature by the sine and cosine of each radian.

[0042] Further, the specific calculation formula of the multi-order difference weighted loss is as follows:

[0043] ;

[0044] ;

[0045] , , ;

[0046] , , ;

[0047] wherein, represents a multi-order differential weighted loss; respectively represent weight coefficients of position loss , first-order differential loss , and second-order differential loss ; N represents the number of target points of a predicted fusion trajectory or a target real trajectory; represents the kth angle information of the predicted fusion trajectory; represents the kth angle information of the target real trajectory; represents the angle distance between the kth angle information of the predicted fusion trajectory and the kth angle information of the target real trajectory; represents the differential result of the k+1th angle information and the kth angle information of the predicted fusion trajectory, that is, the kth first-order differential result of the predicted fusion trajectory; represents the differential result of the k+1th angle information and the kth angle information of the target real trajectory, that is, the kth first-order differential result of the target real trajectory; represents the angle distance between the kth first-order differential result of the predicted fusion trajectory and the kth first-order differential result of the target real trajectory; represents the differential result between the k+1th first-order differential result and the kth first-order differential result of the predicted fusion trajectory, that is, the kth second-order differential result of the predicted fusion trajectory; represents the differential result between the k+1th first-order differential result and the kth first-order differential result of the target real trajectory, that is, the kth second-order differential result of the target real trajectory; represents the angle distance between the kth second-order differential result of the predicted fusion trajectory and the kth second-order differential result of the target real trajectory.

[0048] Based on the same concept, the present application also provides a multi-sensor target trajectory fusion method, comprising:

[0049] obtaining a pair of trajectories to be fused; wherein the pair of trajectories to be fused comprises one radar target trajectory and one photoelectric target trajectory;

[0050] calling a target fusion model; wherein the target fusion model is constructed by using the multi-sensor target trajectory fusion modeling method as described above;

[0051] performing fusion processing on the pair of trajectories to be fused by using the target fusion model to obtain a fused trajectory.

[0052] Based on the same concept, the present application also provides an electronic device comprising a memory, a processor and a computer program / instructions stored on the memory, the processor executing the computer program / instructions to implement the multi-sensor target trajectory fusion modeling method or the multi-sensor target trajectory fusion method as described above.

[0053] Based on the same concept, the present application also provides a computer-readable storage medium having stored thereon a computer program / instructions, which, when executed by a processor, implements the multi-sensor target trajectory fusion modeling method or the multi-sensor target trajectory fusion method as described above.

[0054] Compared with the prior art, the present application has the following beneficial effects:

[0055] The present application correlates and fuses radar target trajectories and photoelectric target trajectories, solving the problems of insufficient trajectory accuracy and weak anti-interference ability caused by the lack of data dimension of a single sensor in a complex environment.

[0056] The present application constructs a trajectory fusion model based on a self-attention mechanism, directly modeling long-distance dependencies at any distance, which is more effective than the convolution method; global feature extraction is performed on the radar target trajectory and the photoelectric target trajectory in the trajectory pair, so that even if the radar target trajectory and the photoelectric target trajectory are not strictly time-aligned, spatio-temporal fusion can still be performed through global features.

[0057] The present application uses multi-order differential weighted loss to adjust the parameters of the model, and through multi-order differential constraints, the trajectory output by the model is more in line with the target motion characteristics, avoiding unreasonable sudden changes in trajectory, while being able to respond more quickly to dynamic changes; the multi-order differential weighted loss can also effectively suppress low-frequency noise and short-term fluctuations, making the loss function pay more attention to meaningful motion trends rather than local jitter.

[0058] The present application significantly overcomes the inherent bottlenecks of traditional fusion algorithms through the intelligent architecture of deep learning, effectively eliminating the influence of the accuracy of artificial prior knowledge, while having excellent generalization ability; in a multi-target complex scene, the present application can significantly improve the accuracy and precision of trajectory tracking, effectively resisting the interference of the environment on a single sensor through a multi-sensor cooperative mechanism, and is particularly suitable for technical needs in the field of low-altitude unmanned aerial vehicle monitoring and security management. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0060] Figure 1 is a multi-sensor target trajectory fusion method flow chart in the embodiment of the present application;

[0061] Figure 2 is a radar target trajectory and photoelectric target trajectory association flow chart in the embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the present application will be described clearly and completely in the embodiments of the present application in combination with the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0063] The technical solutions of the present application will be described in detail in the following specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0064] Embodiment one

[0065] The UAV tracking algorithm is mainly divided into a visible light-based tracking algorithm and a radar and visible light fusion (i.e. radar and visible light fusion)-based tracking algorithm: the visible light-based tracking algorithm realizes UAV tracking through target detection and tracking algorithm, which is simple to implement, but cannot obtain the distance information of the UAV, cannot realize effective tracking in a three-dimensional coordinate system, and is easily disturbed by the environment; the radar and visible light fusion-based tracking algorithm realizes UAV tracking through information collected by multiple sensors of radar and visible light, and the distance information detected by radar can make up for the defect that the visible light sensor cannot obtain the distance information of the UAV, which realizes the fusion of information of multiple sensors, but the traditional track association and fusion algorithm depends on the latest target trajectory state, and needs to accurately estimate the target motion model, system noise and measurement noise covariance, and has poor generalization in complex scenes.

[0066] To solve the above technical problems, the present embodiment takes a UAV as an example to provide a multi-sensor target trajectory fusion method, which comprises the following steps:

[0067] Step 1: obtaining a radar target trajectory and a photoelectric target trajectory.

[0068] The radar target point data is processed by using a radar target point tracking algorithm, so as to generate a radar target trajectory. The main purpose of the radar target point tracking algorithm is to accurately track and identify the target detected by radar and give a unique ID.

[0069] In the specific embodiment of the present application, the specific acquisition process of the radar target trajectory is:

[0070] Step A1: Obtain radar target point data.

[0071] The radar target point refers to the target point detected by radar. The radar target point data includes the azimuth angle, elevation angle, distance and speed of the radar target point, and the radar target trajectory is output after processing by steps A2 to A5. Taking a drone as an example, the radar target point data includes the azimuth angle, elevation angle, distance and speed of the drone.

[0072] Step A2: Preprocess and filter the radar target point data, and convert the radar target point data in the spherical coordinate system to the Cartesian coordinate system (i.e. three-dimensional coordinate system).

[0073] This embodiment sets the signal-to-noise ratio, speed threshold and distance threshold to filter the obviously abnormal radar target points.

[0074] This embodiment uses a Kalman filter to track the radar target point. In order to achieve better linear characteristics of radar target point tracking, the radar target point data in the spherical coordinate system is converted to the Cartesian coordinate system, i.e. the azimuth angle, elevation angle and distance of the radar target point are converted to the position of the radar target point in the Cartesian coordinate system.

[0075] Step A3: Generate a current state vector according to the radar target point data in the Cartesian coordinate system.

[0076] The radar target point data in the Cartesian coordinate system is the position and speed of each radar target point, so the state vector can be expressed as wherein, represents the coordinate position of the radar target point, represents the speed component of the radar target point on the X, Y and Z coordinate axes, respectively.

[0077] Step A4: Design a Kalman filter; wherein the state transition model of the Kalman filter adopts a uniform motion model, and the observation model is used to observe the position of the radar target point.

[0078] Step A5: Take the current state vector as the input of the Kalman filter, and use the Kalman filter to perform state estimation to obtain the estimated position of the radar target point.

[0079] The Kalman filter performs state estimation by alternating prediction and update: according to the current state vector and the uniform motion model, the state vector at the next time is predicted; the predicted state vector at the next time is corrected according to the new observation value output by the observation model.

[0080] For each new radar target point, the Euclidean distance between it and the predicted radar target point position is calculated first, a distance cost matrix is constructed, and then a matching algorithm is used for optimal matching to assign the new radar target point to the optimally matched radar target track. The unmatched new radar target point forms a new radar target track, and the radar target track that has not been updated for a long time is terminated.

[0081] Step A5: convert the radar target point estimated position to the spherical coordinate system to obtain the azimuth angle, elevation angle and distance of the radar target point at the next time, and further obtain the radar target track.

[0082] Convert the radar target point position estimated by the Kalman filter in the Cartesian coordinate system (i.e., the radar target point estimated position) to the spherical coordinate system, i.e., obtain the azimuth angle, elevation angle and distance of the radar target point at the next time, and form the radar target track from the radar target points at different times.

[0083] The optical-electrical target tracking algorithm is used to process the optical-electrical target point data, and the optical-electrical target track is generated. The main purpose of the optical-electrical target tracking algorithm is to accurately track and identify the target detected by the optical-electrical sensor. The optical-electrical target point refers to the target point monitored by the optical-electrical sensor. The optical-electrical sensor of the embodiment is an infrared camera and a visible light camera, for example, a model DS-2TD9167-230ZK2FL / W pan-tilt camera, the visible light camera of which has a zoom range of 1~100 times, the infrared camera has a zoom range of 1~10 times, and the motion range of the pan-tilt camera is horizontal 0~360°, and pitch -45°~+45°.

[0084] In the specific embodiment of the present application, the specific acquisition process of the optical-electrical target track is as follows:

[0085] Step B1: obtain the current optical-electrical target point detection frame.

[0086] Target recognition is performed on the image collected by the optical-electrical sensor, and the optical-electrical target point detection frame is obtained.

[0087] Step B2: convert the optical-electrical target point detection frame in the pixel coordinate system to the spherical coordinate system according to the horizontal and vertical field of view angle range of the optical-electrical sensor to obtain the angle information of the optical-electrical target point.

[0088] The angle information of the optical-electrical target point includes the azimuth angle (or horizontal angle) and the elevation angle of the optical-electrical target point in the spherical coordinate system, and the specific conversion formula is as follows:

[0089] (1)

[0090] (2)

[0091] (3)

[0092] (4)

[0093] wherein, and denote the azimuth and elevation angle of the photoelectric target point in the spherical coordinate system, respectively; and denote the azimuth and elevation angle of the photoelectric sensor at present, respectively; denotes the north-bias angle of the photoelectric sensor; and denote the azimuth and elevation angle offset of the photoelectric target point relative to the optical center, respectively; and denote the horizontal and vertical coordinates of the photoelectric target point in the pixel coordinate system; , ) denotes the center point of the photoelectric target point detection frame in the pixel coordinate system; and denote the width and height of the image, respectively; and denote the horizontal and vertical field of view angles of the photoelectric sensor, respectively.

[0094] Step B3: taking the azimuth angle, the elevation angle, the azimuth angle velocity and the elevation angle velocity of the photoelectric target point as the input of the photoelectric target point tracking algorithm, taking the angular distance as the distance measurement of the photoelectric target point tracking algorithm, and using the photoelectric target point tracking algorithm to obtain the azimuth angle and the elevation angle of the photoelectric target point in the spherical coordinate system at the next moment, and then obtaining the photoelectric target trajectory.

[0095] taking the azimuth angle, the elevation angle, the azimuth angle velocity and the elevation angle velocity of the photoelectric target point as the input of the photoelectric target point tracking algorithm, taking the angular distance as the distance measurement of the photoelectric target point tracking algorithm, and using the photoelectric target point tracking algorithm to obtain the azimuth angle and the elevation angle of the photoelectric target point in the spherical coordinate system at the next moment, and then obtaining the photoelectric target trajectory.

[0096] In order to effectively deal with the periodicity problem of angle, the embodiment takes the angular distance as the distance measurement of the photoelectric target point tracking algorithm, and the specific calculation formula is:

[0097] (5)

[0098] wherein, denotes the angular distance between the i-th photoelectric target point and the j-th photoelectric target point, ; and denote the azimuth and elevation angle of the i-th photoelectric target point, respectively; and respectively represent the azimuth angle and the elevation angle of the jth photoelectric target point.

[0099] The present application replaces the traditional intersection and ratio calculation by introducing the great circle distance to effectively deal with the periodicity problem of angle. The calculation method based on the great circle distance not only considers the comprehensive influence of the azimuth angle and the elevation angle, but also can avoid the influence of angle periodicity on distance calculation; compared with the traditional Mahalanobis distance, it can more accurately reflect the actual distance between any two photoelectric target points in the spherical coordinate system.

[0100] Considering the characteristics of low-altitude unmanned aerial vehicle target, such as small volume, high mobility and complex background, the embodiment of the present application designs an angle-based multi-target tracking algorithm, which cascades and matches the photoelectric target point detection frame with the photoelectric target point detection frame predicted by the last frame Kalman filter, performs secondary matching on the photoelectric target point detection frame and the predicted photoelectric target point detection frame which fail to match, and updates the photoelectric target point state or creates a new photoelectric target track according to the matching result. The cascade matching constructs a cost matrix by integrating the spatial distance and the appearance feature similarity, and uses the Hungarian algorithm to realize the optimal matching, so as to effectively associate the newly detected photoelectric target point with the existing photoelectric target track.

[0101] The radar target track obtained by the radar target tracking algorithm and the photoelectric target track obtained by the photoelectric target tracking algorithm are in the same spherical coordinate system, so as to facilitate subsequent association and fusion of the radar target track and the photoelectric target track.

[0102] Step 2: associate the radar target track and the photoelectric target track to obtain a plurality of track pairs.

[0103] Track association is a key step of multi-sensor track fusion, and the purpose is to find the track set belonging to the same target from the information of different sensors. The present application proposes a track association method based on self-attention mechanism, which can determine whether asynchronous tracks are associated without high-precision time registration, allows tracks with different frame rates and short time misalignment to be associated, and effectively solves the limitations of traditional methods in dealing with complex scenes. In the specific embodiments of the present application, such as Figure 2 , the association of the radar target track and the photoelectric target track comprises:

[0104] Step 2.1: pre-process the radar target track and the photoelectric target track respectively.

[0105] Each radar target corresponds to a radar target track, each radar target track is generated by radar target points at different times, and each radar target point data includes the azimuth angle, the elevation angle and the ID number of the radar target point; each photoelectric target corresponds to a photoelectric target track, each photoelectric target track is generated by photoelectric target points at different times, and each photoelectric target point data includes the azimuth angle, the elevation angle and the ID number of the photoelectric target point.

[0106] In order to avoid the error caused by the angle periodicity, the present application performs trigonometric function normalization processing on the azimuth angle and the elevation angle in the radar target trajectory and the photoelectric target trajectory. Specifically, the azimuth angle and the elevation angle in the radar target trajectory are converted into radians respectively to obtain the azimuth angle radian and the elevation angle radian; then the sine and cosine of the azimuth angle radian and the sine and cosine of the elevation angle radian are calculated, and the sine and cosine of the azimuth angle radian and the sine and cosine of the elevation angle radian constitute the radar target input feature.

[0107] The azimuth angle and the elevation angle in the photoelectric target trajectory are converted into radians respectively to obtain the azimuth angle radian and the elevation angle radian; then the sine and cosine of the azimuth angle radian and the sine and cosine of the elevation angle radian are calculated, and the sine and cosine of the azimuth angle radian and the sine and cosine of the elevation angle radian constitute the photoelectric target input feature.

[0108] For example, for a radar target point or a photoelectric target point with an azimuth angle of 30° and an elevation angle of 160°, after trigonometric function normalization processing, (sin30°, cos30°, sin160°, cos160°) is obtained, that is, a four-dimensional input feature of (0.5, 0.87, 0.34, -0.94) is obtained.

[0109] When the radar target trajectory and the photoelectric target trajectory are associated, the radar target trajectory and the photoelectric target trajectory are preprocessed by using the trigonometric function normalization method to preprocess the two-dimensional input (azimuth angle and elevation angle) into a four-dimensional input feature, which not only can well preserve the periodicity of the angle and avoid boundary jumping, but also can more fully represent the trajectory feature, which is conducive to trajectory association; at the same time, only the feature needs to be extracted during the trajectory association, and no inverse normalization processing is needed.

[0110] Step 2.2: The first self-attention mechanism backbone network and the second self-attention mechanism backbone network are used to extract features from the preprocessed radar target trajectory and photoelectric target trajectory respectively to obtain a radar target feature vector and a photoelectric target feature vector.

[0111] The first self-attention mechanism backbone network and the second self-attention mechanism backbone network share weight parameters, and the self-attention mechanism backbone network is used to extract the global feature of the preprocessed radar target trajectory or photoelectric target trajectory. In the present embodiment, the first self-attention mechanism backbone network and the second self-attention mechanism backbone network are both Encoder modules in the transformer network model.

[0112] The first self-attention mechanism backbone network extracts features of each preprocessed radar target track to obtain a 128-dimensional radar target feature vector.

[0113] Step 2.3: Calculate the cosine similarity of the radar target feature vector and the photoelectric target feature vector to obtain a similarity matrix.

[0114] Step 2.4: Determine the track pair according to the similarity matrix, each track pair including one associated radar target track and one photoelectric target track.

[0115] Based on the similarity matrix between any two radar target tracks and photoelectric target tracks, the association relationship between the radar target tracks and the photoelectric target tracks is calculated through a matching algorithm, and then the track pair is determined. In this embodiment, the Hungarian matching algorithm is selected as the matching algorithm. The Hungarian matching algorithm constructs a cost matrix according to the similarity matrix, and through row and column transformation and zero element marking, it iteratively finds the method of covering all zero elements with the smallest number, and finally finds the track pair with the smallest total cost in the bipartite graph. Each track pair includes one associated radar target track and one photoelectric target track. The target real track corresponding to each track pair can be generated according to the target path characteristics, or can be obtained by actual collection, for example, the unmanned aerial vehicle target. The simulation data generated according to the flight characteristics of the unmanned aerial vehicle can be used as the target real track, or the real track of the unmanned aerial vehicle can be obtained through the GPS system of the unmanned aerial vehicle, and the target real track is used as the label.

[0116] After determining the track pair, track fusion is performed. Track fusion aims to integrate multiple source tracks of the same target to obtain more reliable global estimation results than single sensors, so the input of the track fusion algorithm is the track pair, and the output is the fused track, i.e. the fusion track.

[0117] Step 3: Perform angle expansion on the radar target track and the photoelectric target track of each track pair.

[0118] In order to solve the jump problem caused by the periodic characteristics of the angle data of each track pair, the present application performs angle expansion on the radar target track and the photoelectric target track of each track pair, linearizes the angle data, ensures smooth transition when continuously changing, and effectively avoids the negative impact of periodic jumps on subsequent fusion analysis. The specific formula of angle expansion is:

[0119] (6)

[0120] wherein, represents the original angle, i.e. the azimuth angle or the pitch angle in the radar target track and the photoelectric target track; Indicates the angle after unfolding; Integer factor representing the adjustment period.

[0121] For example, for a radar target trajectory or an electro-optical target trajectory, the azimuth angles of multiple consecutive target points are [355°, 358°, 2°, 4°]. Since this crosses the 360° boundary, directly performing differential calculations yields [3°, -356°, 2°], which is not realistic. However, if the azimuth angles are expanded to [355°, 358°, 362°, 364°], and then differential calculations yield [3°, 4°, 2°], the correct differential result can be obtained. Therefore, to ensure the correctness of subsequent differential calculations, the radar target trajectory and electro-optical target trajectory in the trajectory pair are first subjected to angle expansion processing.

[0122] When fusing radar target trajectories and electro-optical target trajectories, the angle unfolding method is directly used to preprocess the radar target trajectories and electro-optical target trajectories. This not only eliminates the influence of periodicity and achieves linear continuous change, but also directly calculates the multi-order difference weighted loss based on the output results during trajectory fusion model training, thus reducing computational complexity.

[0123] Step 4: Construct a trajectory fusion model.

[0124] In a specific embodiment of the present invention, the trajectory fusion model adopts an encoder-decoder structure based on a self-attention mechanism, specifically including a first self-attention mechanism encoder, a second self-attention mechanism encoder, an adaptive weighted fusion module, and a self-attention mechanism decoder; the first self-attention mechanism encoder is used to extract features from the radar target trajectory after angular unwrapping to obtain high-dimensional trajectory features of the radar target; the second self-attention mechanism encoder is used to extract features from the photoelectric target trajectory after angular unwrapping to obtain high-dimensional trajectory features of the photoelectric target; the adaptive weighted fusion module is used to adaptively weight and fuse the high-dimensional trajectory features of the radar target and the photoelectric target to obtain weighted fused features; the self-attention mechanism decoder is used to decode the weighted fused features to obtain the predicted fused trajectory.

[0125] The first and second self-attention encoders project the input target trajectory (i.e., radar or electro-optical target trajectory) onto a unified feature dimension through a linear projection layer. Then, position encoding is added to enhance the temporal information of the sequence. Finally, a multi-head self-attention mechanism captures long-range dependencies within the sequence, yielding high-dimensional target trajectory features. The adaptive weighted fusion module uses an adaptive weighting method, adding independently learnable weights to the high-dimensional trajectory features of each target trajectory. It adaptively learns the importance of the radar and electro-optical target trajectories, thereby fusing the two high-dimensional trajectory features. This allows the trajectory fusion model to dynamically adjust the importance of different sensors based on data characteristics, improving fusion accuracy. The self-attention decoder decodes the weighted fusion features layer by layer, converting the high-dimensional features into the same trajectory sequence as the input.

[0126] Step 5: Using trajectory pairs as input and the corresponding target real trajectories as labels, train and validate the trajectory fusion model to obtain the target fusion model.

[0127] During training, a multi-order difference weighted loss is calculated based on the predicted fused trajectory output by the trajectory fusion model and the target's true trajectory. The parameters of the trajectory fusion model are then adjusted based on this multi-order difference weighted loss to achieve iterative training. In a specific embodiment of this invention, the specific formula for calculating the multi-order difference weighted loss is as follows:

[0128] (7)

[0129] (8)

[0130] , , (9)

[0131] , , (10)

[0132] in, This represents the multi-order difference weighted loss; They represent position loss respectively. First-order difference loss Second-order difference loss The weighting coefficients; N represents the number of target points in the predicted fusion trajectory or the actual target trajectory; This represents the k-th angle information of the predicted fused trajectory; This represents the k-th angle information of the target's true trajectory; This represents the angular distance between the k-th angular information of the predicted fused trajectory and the k-th angular information of the target's true trajectory; This represents the difference between the (k+1)th angle information and the kth angle information in the predicted fusion trajectory, i.e., the kth first-order difference result in the predicted fusion trajectory. This represents the difference between the (k+1)th angle information and the kth angle information of the target's true trajectory, i.e., the kth first-order difference result of the target's true trajectory; This represents the angular distance between the k-th first-order difference result of the predicted fused trajectory and the k-th first-order difference result of the target true trajectory; This represents the difference between the (k+1)th first-order difference result and the kth first-order difference result in the predicted fusion trajectory, which is the kth second-order difference result in the predicted fusion trajectory. This represents the difference between the (k+1)th first-order difference result and the kth first-order difference result of the target's true trajectory, which is the kth second-order difference result of the target's true trajectory. This represents the angular distance between the k-th second-order difference result of the predicted fused trajectory and the k-th second-order difference result of the target's true trajectory.

[0133] angular distance , , The calculation is shown in formula (5). The first-order difference and the second-order difference can be approximated as the first-order derivative and the second-order derivative. In the loss calculation, the first-order difference loss is used to measure the target's velocity information, and the second-order difference loss is used to measure the target's acceleration information. The target's motion characteristics are measured by the first-order difference loss and the second-order difference loss. This invention adopts multi-order difference weighted loss, which makes the model comprehensively consider the trajectory position, velocity and acceleration. Through the constraints of the first-order difference and the second-order difference, the trajectory output by the model is more in line with the target's motion law (such as inertia and smoothness), avoiding unreasonable sudden trajectory changes. For maneuvering targets (such as sudden turns, acceleration / deceleration), relying solely on position loss may lead to lag or oversmoothing, but after adding velocity and acceleration constraints, the model can respond to dynamic changes more quickly. At the same time, for difference operations (especially high-order differences), low-frequency noise and short-term fluctuations (such as sensor drift) can be well suppressed, making the loss function focus more on meaningful motion trends rather than local jitter.

[0134] To improve the training efficiency of the trajectory fusion model, this invention employs transfer learning. First, a large amount of simulated data generated based on the characteristics of the target path is used as the target's real trajectory for pre-training the trajectory fusion model. Then, a small amount of real data is collected to fine-tune the pre-trained trajectory fusion model. During training, data augmentation is performed through random flipping and random pruning to enhance the model's generalization ability.

[0135] This invention employs a trajectory fusion model based on a self-attention mechanism, which can directly model long-range dependencies at arbitrary distances, making it more efficient than convolutional methods. Through global feature extraction, even if the radar target trajectory and the photoelectric target trajectory in a trajectory pair are not strictly time-aligned, they can still be fused using global features. Convolutional neural network models, on the other hand, cannot efficiently extract global features and require strict time alignment to ensure fusion effectiveness. This invention effectively combines the advantages of different sensors, improving tracking accuracy.

[0136] Example 2

[0137] The multi-sensor target trajectory fusion method provided in this embodiment of the invention includes the following steps:

[0138] Step 1: Obtain the trajectory pairs to be merged.

[0139] The trajectory pair to be fused includes a radar target trajectory and an electro-optical target trajectory.

[0140] Step 2: Invoke the target fusion model.

[0141] The target fusion model is constructed using the multi-sensor target trajectory fusion modeling method in Embodiment 1 of this invention.

[0142] Step 3: Use the target fusion model to fuse the trajectory pairs to obtain the fused trajectory.

[0143] Example 3

[0144] This invention also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory. The processor executes the computer program / instructions to implement the multi-sensor target trajectory fusion modeling method or the multi-sensor target trajectory fusion method of this invention.

[0145] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes based on programs and / or data stored in read-only memory (ROM) or loaded from a storage portion into random access memory (RAM). The processor can be a multi-core processor or may contain multiple processors. In some embodiments, the processor may include a general-purpose main processor and one or more specialized coprocessors, such as a central processing unit, graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), etc. Various programs and data required for device operation are also stored in RAM. The processor, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0146] The processor and memory described above are used together to execute programs / instructions stored in the memory. When the program / instructions are executed by the computer, they can implement the methods, steps, or functions described in the above embodiments.

[0147] Although not shown, embodiments of the present invention also provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the multi-sensor target trajectory fusion modeling method or the multi-sensor target trajectory fusion method of the present invention.

[0148] Readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0149] The above description only discloses specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or modifications that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-sensor target trajectory fusion modeling method, characterized in that, The modeling method includes: Multiple trajectory pairs and the target's actual trajectory corresponding to each trajectory pair are acquired; wherein each trajectory pair includes a successfully associated radar target trajectory and an optoelectronic target trajectory; For each of the aforementioned trajectory pairs, the radar target trajectory and the photoelectric target trajectory are respectively expanded by angle; A trajectory fusion model is constructed, comprising a first self-attention mechanism encoder, a second self-attention mechanism encoder, an adaptive weighted fusion module, and a self-attention mechanism decoder. The first self-attention mechanism encoder extracts features from the radar target trajectory after angular unwrapping to obtain high-dimensional trajectory features of the radar target. The second self-attention mechanism encoder extracts features from the photoelectric target trajectory after angular unwrapping to obtain high-dimensional trajectory features of the photoelectric target. The adaptive weighted fusion module performs adaptive weighted fusion of the high-dimensional trajectory features of the radar target and the high-dimensional trajectory features of the photoelectric target to obtain weighted fused features. The self-attention mechanism decoder decodes the weighted fused features to obtain a predicted fused trajectory. Using the trajectory pairs as input and the corresponding target real trajectories as labels, the trajectory fusion model is trained and validated to obtain a target fusion model. During the training process, a multi-order difference weighted loss is calculated based on the predicted fused trajectory and the target real trajectory output by the trajectory fusion model, and the parameters of the trajectory fusion model are adjusted based on the multi-order difference weighted loss. The specific process for acquiring the radar target trajectory is as follows: Acquire radar target point data, which includes the azimuth, elevation, range, and velocity of the radar target point; The radar target point data is preprocessed and filtered, and the radar target point data in the spherical coordinate system is transformed to the Cartesian coordinate system; A current state vector is generated based on radar target point data in Cartesian coordinates; wherein, the current state vector includes the radar target point position and velocity; Design a Kalman filter; wherein, the state transition model of the Kalman filter adopts a uniform motion model, and the observation model is used to observe the position of the radar target point; Using the current state vector as the input to the Kalman filter, state estimation is performed using the Kalman filter to obtain the estimated position of the radar target point; The estimated position of the radar target point is transformed into a spherical coordinate system to obtain the azimuth, elevation, and range of the radar target point at the next moment, and thus the radar target trajectory is obtained.

2. The multi-sensor target trajectory fusion modeling method according to claim 1, characterized in that, The specific process of obtaining multiple trajectory pairs includes: Acquire radar target trajectories and electro-optical target trajectories; The radar target trajectory and the photoelectric target trajectory are preprocessed respectively; The first self-attention mechanism backbone network and the second self-attention mechanism backbone network with parameter sharing are used to extract features from the preprocessed radar target trajectory and photoelectric target trajectory, respectively, to obtain radar target feature vector and photoelectric target feature vector; Calculate the similarity between the radar target feature vector and the electro-optical target feature vector to obtain the similarity matrix; Trajectory pairs are determined based on the similarity matrix.

3. The multi-sensor target trajectory fusion modeling method according to claim 2, characterized in that, The specific process for acquiring the photoelectric target trajectory is as follows: Obtain the photoelectric target point detection frame; The photoelectric target point detection box in the pixel coordinate system is transformed to the spherical coordinate system to obtain the angle information of the photoelectric target point; wherein, the angle information of the photoelectric target point includes the azimuth and pitch angle of the photoelectric target point in the spherical coordinate system; The azimuth, elevation, azimuth angular velocity, and elevation angular velocity of the photoelectric target point are used as inputs to the photoelectric target point tracking algorithm. The angular distance is used as the distance metric of the photoelectric target point tracking algorithm. The photoelectric target point tracking algorithm is used to obtain the azimuth and elevation angles of the photoelectric target point in the spherical coordinate system at the next moment, and then the trajectory of the photoelectric target is obtained.

4. The multi-sensor target trajectory fusion modeling method according to claim 3, characterized in that, The specific formula for calculating the angular distance is as follows: ； in, This represents the angular distance between the i-th photoelectric target point and the j-th photoelectric target point. ; and Let represent the azimuth and elevation angles of the i-th photoelectric target point, respectively; and Let represent the azimuth and elevation angles of the j-th photoelectric target point, respectively.

5. The multi-sensor target trajectory fusion modeling method according to claim 2, characterized in that, The radar target trajectory and the photoelectric target trajectory are preprocessed respectively, specifically including: The azimuth and elevation angles in the radar target trajectory are converted into radians, and then the sine and cosine of each radian are calculated. The sine and cosine of each radian constitute the radar target input characteristics. The azimuth and elevation angles in the photoelectric target trajectory are converted into radians, and then the sine and cosine of each radian are calculated. The sine and cosine of each radian constitute the input features of the photoelectric target.

6. The multi-sensor target trajectory fusion modeling method according to any one of claims 1 to 5, characterized in that, The specific formula for calculating the multi-order difference weighted loss is as follows: ； ； ， ， ； ， ， ； in, This represents the multi-order difference weighted loss; They represent position loss respectively. First-order difference loss Second-order difference loss The weighting coefficients; N represents the number of target points in the predicted fusion trajectory or the actual target trajectory; This represents the k-th angle information of the predicted fused trajectory; This represents the k-th angle information of the target's true trajectory; This represents the angular distance between the k-th angular information of the predicted fused trajectory and the k-th angular information of the target's true trajectory; This represents the difference between the (k+1)th angle information and the kth angle information in the predicted fusion trajectory, i.e., the kth first-order difference result in the predicted fusion trajectory. This represents the difference between the (k+1)th angle information and the kth angle information of the target's true trajectory, i.e., the kth first-order difference result of the target's true trajectory; This represents the angular distance between the k-th first-order difference result of the predicted fused trajectory and the k-th first-order difference result of the target true trajectory; This represents the difference between the (k+1)th first-order difference result and the kth first-order difference result in the predicted fusion trajectory, which is the kth second-order difference result in the predicted fusion trajectory. This represents the difference between the (k+1)th first-order difference result and the kth first-order difference result of the target's true trajectory, which is the kth second-order difference result of the target's true trajectory. This represents the angular distance between the k-th second-order difference result of the predicted fused trajectory and the k-th second-order difference result of the target's true trajectory.

7. A multi-sensor target trajectory fusion method, characterized in that, The fusion method includes: Obtain the trajectory pair to be fused; wherein, the trajectory pair to be fused includes a radar target trajectory and an electro-optical target trajectory; The target fusion model is invoked; wherein the target fusion model is constructed using the multi-sensor target trajectory fusion modeling method as described in any one of claims 1 to 6; The target fusion model is used to fuse the trajectory pairs to be fused, resulting in the fused trajectory.

8. An electronic device comprising a memory, a processor, and a computer program / instructions stored in the memory, characterized in that, The processor executes the computer program / instructions to implement the multi-sensor target trajectory fusion modeling method as described in any one of claims 1 to 6 or the multi-sensor target trajectory fusion method as described in claim 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the multi-sensor target trajectory fusion modeling method as described in any one of claims 1 to 6 or the multi-sensor target trajectory fusion method as described in claim 7.

Images