Multi-target vehicle trajectory extraction method based on pixel-level image fusion

Abstract

The application provides a multi-target vehicle trajectory extraction method based on pixel-level image fusion, comprising the following steps: S1, acquiring heterogeneous image sequence data and road environment information; S2, image preprocessing; S3, dual-source image fusion; S4, dual-source image fusion; S5, trajectory post-processing. The application improves the anti-halation ability and target definition of the fused image, improves the target detection effect in the fused image, accelerates the running speed of the multi-target tracking algorithm in the fused image, can automatically extract complete and high-precision vehicle trajectories in the fused image, and is beneficial to increasing the traffic data acquisition scene and improving the traffic data accuracy.

Description

Technical Field

[0001] This invention relates to the field of image fusion and target detection and tracking technology, specifically to a method for extracting the trajectory of multiple targets vehicles based on pixel-level image fusion. Background Technology

[0002] As one of the main research contents of intelligent transportation systems, image-based traffic information perception technology uses image sensors deployed from different perspectives such as roads, vehicles, and the air to collect traffic data. It can detect road conditions and traffic environment in real time and obtain accurate traffic parameter information such as speed, traffic flow, density, and capacity, as well as traffic event and traffic condition information, providing solid data support for traffic model calibration, signal control, driving behavior and other traffic flow theory research.

[0003] Most existing image-based traffic information perception systems use visible light sensors as the data source to acquire road traffic information. The widespread use of visible light sensors is due to several factors, such as their imaging similarity to human vision, intuitive data, rich semantic features including target textures, lane lines, and traffic signs, high resolution, and low deployment cost. However, visible light sensors perform poorly in low light or inclement weather conditions. Infrared image sensors, based on thermal imaging, detect the thermal radiation of objects and possess several advantages: strong anti-interference capabilities, the ability to operate in harsh environments, all-weather operation, and good thermal target detection capabilities due to their temperature sensitivity. However, the resulting images are generally blurry and have low resolution, making them unsuitable for observation and understanding. Considering the imaging characteristics and complementary properties of infrared and visible light images, utilizing their collaborative imaging and fusion can yield fused images with prominent targets and rich background information, even at night or in inclement weather. This improves the accuracy of target recognition, detection, and tracking, and expands the application scenarios of image-based traffic information perception technology.

[0004] Image fusion based on visible light and infrared is a feasible and effective image processing technique. Existing research includes Chinese patent CN202111352352.X, which discloses an infrared and visible light fusion technique based on multi-scale features, and Chinese patent CN202110901665.X, which discloses an infrared and visible light image fusion method based on frequency domain rules. However, these existing methods require manual calibration for specific scenes, resulting in a large computational load and limited algorithm universality. Furthermore, in typical traffic scenes containing halos, significant information loss occurs in single-source images, negatively impacting the quality of the fused image. Existing fusion algorithms rarely consider the loss of single-source information, leading to darker fused images or loss of target details within halos in such scenarios, resulting in poor fusion performance.

[0005] Vehicle-based target detection and tracking is a feasible and effective traffic information perception technology. Existing research includes Chinese patent CN202110061609.X, which discloses a traffic flow statistics algorithm based on vehicle detection and multi-target tracking. This algorithm uses the YOLOv5 algorithm for target vehicle detection and Kalman filtering and the Hungarian algorithm for multi-target correlation tracking. Chinese patent CN201910550286.3 discloses a traffic vehicle information acquisition method based on Mask R-CNN target detection and uses the SORT method to track vehicles within the ROI. However, most target detection and tracking algorithms only provide perception results from visible light images. The imaging mechanism of fused images differs from that of visible light images, and image information such as background texture and target detail contours are also different. It is necessary to explore the application effect of information perception algorithms in fused images. Due to a lack of focus on targets in fused images, most image fusion research has not further realized the application of fused images in target detection, tracking, and other information perception algorithms. The lack of key work such as target-based image fusion and fused image detection and tracking results in a lack of a complete framework for extracting target trajectories from fused images in traffic scenarios. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a multi-target vehicle trajectory extraction method based on pixel-level image fusion. This technique first performs image preprocessing, scaling, cropping, and image registration of the source video images to create conditions for image fusion. Next, it performs dual-source image fusion, designing an FT-GAN dual-discriminator generative adversarial network to fuse the two source images. A target-background network group is designed to divide the source images into target and background image blocks, which are then fed into their respective networks for training. A dual-discriminator generative adversarial network structure is adopted. In the adversarial network generator, a U-Net structure is used to enhance target detail information in the target network, while a tensor addition structure is used in the background network to neutralize and improve the image's anti-halation capability. In the discriminator network, a weighted discriminator loss function is designed using bilateral filtering weight coefficients to increase the similarity between the fused image and the source image with more information, thus enhancing the network fusion effect. Finally, target detection and tracking of the fused image are implemented using YOL. The Ov5 target detection algorithm acquires vehicle trajectories, and a shape-position discrimination and association algorithm is designed to achieve multi-target vehicle tracking in fused images. Sequence background compensation and ROI region extraction are added to improve trajectory extraction efficiency. Finally, a trajectory post-processing step is performed, constraining the trajectory curve based on the stimulus-response model, completing and recovering fragmented trajectories, and calculating the true motion information of the target in the geodetic coordinate system through coordinate transformation. Finally, high-precision target trajectories and traffic parameters are output. This technology improves the problems of unclear targets and poor fusion quality in poor lighting conditions in existing fusion algorithms, enhances the anti-halation ability and target clarity of fused images, improves the target detection effect in fused images, accelerates the running speed of multi-target tracking algorithms in fused images, and can automatically extract complete high-precision vehicle trajectories from fused images. This is beneficial for increasing traffic data acquisition scenarios, improving the accuracy of traffic data, and providing better data support for traffic theory research, traffic management and control, which has extremely significant practical significance.

[0007] Technical solution: To solve the above technical problems, the present invention adopts the following technical solution:

[0008] A method for extracting the trajectory of multiple targets vehicles based on pixel-level image fusion includes the following steps:

[0009] S1. Acquire heterogeneous image sequence data and road environment information. The shooting scene should be a common traffic scene monitoring perspective, and the image source should be visible light and infrared images. Collect the same scene image data at the same time and place.

[0010] S2. Perform image preprocessing operations, cropping and scaling the medium-to-high resolution image (generally a visible light image); use the Shi-Tomasi operator for corner detection, use the KNN algorithm for feature point association, and perform dual-source image registration.

[0011] S3. Dual-source image fusion: Design a dual-discriminator generative adversarial network FT-GAN to implement pixel-level fusion of dual-source images, thereby enhancing the target details and anti-halo capability of the fused image.

[0012] S4. Dual-source image fusion: YOLOv5 algorithm is used for target detection in the fused image. A shape and position discrimination multi-target tracking algorithm is designed to realize the association of detection boxes and trajectory generation from multiple angles of kinematics and target contour, thereby improving the running speed of the target detection and tracking part.

[0013] S5. Perform trajectory post-processing operations, design a trajectory classification process to retain complete trajectories, delete ghost trajectories, filter fragmented trajectories, and complete missing trajectory data based on stimulus-response model to improve the rationality of vehicle trajectories; convert vehicle trajectories from image coordinates to geodetic coordinates, and use the EEMD algorithm to denoise the trajectories to improve the overall accuracy of the trajectories; finally, output high-precision vehicle trajectory and traffic parameter data.

[0014] Furthermore, in S1, the image source is visible light and infrared images, which are the same scene image data collected at the same time and place.

[0015] Furthermore, the specific steps of the improved FAST neighborhood pixel feature point matching strategy in S2 are as follows:

[0016] S201. Use the Shi-Tomasi operator to find the feature corner points in the background region;

[0017] S202, For the feature corner point p in the background region i First, use the following formula to determine p i Is it in a dense cluster of corner points?

[0018]

[0019] min j ||p i ,p j ||2≤Dis th (1)

[0020] Where ||p i ,p j ||2 represents feature point p i With p j European distance, Dis th To determine the distance threshold for corner density, D th This is the proportional threshold;

[0021] If p i If it is not in a dense cluster of corner points, it is directly included in the candidate feature point set Cd. iGiven a cluster of dense corner points, with N0 being the number of corner points in the cluster, calculate the number N corner points that should be retained:

[0022]

[0023] That is, a minimum of 1 corner point and a maximum of N-1 corner points must be retained in a dense cluster of corner points;

[0024] After determining the number of corner points N, iterate through all combinations of corner points with a quantity of N in the corner point cluster, select the combination with the largest spacing as the corner points to be retained, and include it in the candidate feature point set Cd to complete the coarse screening of feature points.

[0025] S203. Find the FAST4 neighborhood descriptor for each feature point in the candidate feature point set Cd, and expand the descriptor to 8 neighborhoods according to the Manhattan distance equality condition.

[0026] S204. Calculate the 8-neighborhood descriptor similarity features:

[0027]

[0028] Where D pq Let P be the normalized distance between two descriptors. pq To describe the normalized pixel difference:

[0029]

[0030]

[0031] Where x, y are the coordinates of the descriptor image, L, W are the image length and width, and P... range The range of image pixel values;

[0032] S205, Select S sm The top K feature points are selected as matching feature points after fine screening and used in perspective transformation generation.

[0033] Furthermore, the specific steps of the dual-source image pixel-level fusion in S3 are as follows:

[0034] S301. Use the K-Means algorithm to cluster trajectories to extract the size information of targets within the region and obtain the size of segmented image blocks;

[0035] S302. Use E-Net to perform semantic segmentation on the source image, and design a target area ratio threshold to determine whether a certain image patch belongs to the target or the background:

[0036] N = max{l mean ,w mean}+α

[0037]

[0038] Among them l mean ,w mean The length and width of the center bounding box obtained by the K-means algorithm are both integers, and α is a constant integer that makes N satisfy the network structure requirements; patch n For image patch n, tar and bgd represent that the image is classified as target and background, respectively. th This is a threshold representing the area of ​​the target region. If the area exceeds this threshold, the image patch is considered the target; otherwise, it is considered the background. These represent the areas occupied by the target in the visible light and infrared image blocks, respectively.

[0039] S303. Substitute the target image patch and the background image patch into the target network group and the background network group respectively for weight training and image fusion.

[0040] S304. Set the background network group network structure;

[0041] S305, restore the image block to the source image position, and output a fused image with anti-halo capability;

[0042] Furthermore, the specific structure of the target network group in S303 is as follows:

[0043] S3031. A U-shaped network is used for pseudo-fused image generation. A 7-layer network is designed, using convolutional layers with a stride of 2 to reduce the feature map resolution. The convolutional kernel is set to 4*4, and the padding is 1 to ensure the scale of the feature map and the input / output images.

[0044] The dimensions are consistent, and the activation function is designed as Leaky-Relu;

[0045] S3032. The target network adopts the PatchGAN structure to design the discriminator network. It uses three convolutional layers with a stride of 2 for downsampling to obtain the dimensionality-reduced feature map and performs zero-value padding of size 2. Next, it uses a convolutional layer with a stride of 1 to accurately extract features and normalizes the feature values ​​through BatchNorm + LeakyReLU. Finally, after padding the feature map again, it uses a convolutional layer with 1 kernel to merge the dimensions of the feature map and directly outputs an n*n score matrix through the tanh activation function.

[0046] S3033, The loss function of the target network generator is calculated as follows:

[0047]

[0048] Where L SSIM L is the structural loss calculated using the structural similarity index SSIM. L1The edge loss is calculated using the L1 norm, and δ1 and δ2 are weight parameters that measure the importance of structural loss and edge loss in the target network.

[0049] L SSIM The calculation method is as follows:

[0050]

[0051] in, The SSIM loss for pixel p can be expressed as:

[0052]

[0053] in Indicates the expected image variance. This represents the covariance between the expected image and the pseudo-fused image;

[0054] L L1 The calculation method is as follows:

[0055]

[0056]

[0057] Where the weight parameter w(I) k The weighting function is the same as that in the SSIM loss;

[0058] S3034. The loss function of the target network discriminator is calculated as follows:

[0059]

[0060]

[0061] Where p1 and p2 are weighting coefficients reflecting the amount of information in the source image, p1 represents the visible light image and p2 represents the infrared image, which are calculated by the bilateral filtering weighting function.

[0062] Furthermore, the specific network structure of the S304 background network group is as follows:

[0063] S3041. The background network generator uses a tensor addition network structure to generate fused images.

[0064] S3042. The background network group discriminator network structure is the same as the target network group.

[0065] S3043, The loss function of the background network group generator is calculated as follows:

[0066]

[0067] The generator adversarial loss is given by the discriminator in the adversarial network, L. con For content loss, λ is the weighting coefficient that adjusts the importance of the two types of loss;

[0068] Adversarial Loss: In this paper, the adversarial loss is taken as the cross-entropy of the discriminator's discrimination results on the pseudo-fused image, that is:

[0069]

[0070] Among them, D v (G(v,i)) represents the discriminator D. v The probability of distinguishing a pseudo-fused image G(v,i) from a true image, D i (G(v,i)) represents the discriminator D. i The probability of identifying a pseudo-fused image G(v,i) as true;

[0071] Content loss is defined as follows:

[0072]

[0073]

[0074] S3044. The loss function of the background network group discriminator is the same as that of the target network group.

[0075] Furthermore, the specific steps of the shape and position discrimination multi-target tracking algorithm in S4 are as follows:

[0076] S401. Perform background compensation on the image sequence, fix the pixel spatial information of the image sequence, implement coordinate transformation, and convert the target detection box data from the image coordinate system to the top-view Frenet coordinate system.

[0077] S402. Initialize the target tracking algorithm parameters, design a short-time sector displacement domain to estimate the target movement range, and initially screen candidate tracking boxes;

[0078] S403. Design a weighted scoring system for shape and position discrimination, update the predicted tracking box position according to the target type, calculate the kinematic score of the candidate tracking box, obtain the target similarity according to the cosine distance of the feature points of the tracking box, calculate the shape score of the tracking box, and estimate the inter-frame association result based on the comprehensive shape and position discrimination score.

[0079] S404. Design a multi-target trajectory verification method for traffic scenarios, extract complete trajectories within the scenario, and eliminate invalid trajectories.

[0080] Collect discontinuous trajectories and input them into the car-following model based on the spatiotemporal information of both ends to complete the missing trajectories. Combine traffic parameters and time-varying characteristics of trajectories to search for erroneous tracking trajectories and correct the tracking results.

[0081] Furthermore, S402 consists of the following specific steps:

[0082] Calculate the half-angle θ of the sector displacement domain search:

[0083]

[0084] Where θ0 represents the physical limit deflection angle of the real target, and V0 represents the physical limit displacement of the real target. h0 represents the average intersection-union ratio between the detection bounding box and the actual target position, and h0 represents the height of the target detection bounding box.

[0085] Calculate the search radius V of the sector displacement domain:

[0086]

[0087] Where w0 is the width of the target detection box, and DT is the maximum displacement coefficient, which is determined based on the target type and physical limits.

[0088] Within a fan-shaped region with a search radius of V and a search half-angle of θ, candidate tracking boxes are filtered out, and the position and size information of the tracking boxes within the region are retained.

[0089] Furthermore, the specific steps of S403 are as follows:

[0090] Calculate the predicted tracking box position

[0091]

[0092] Where a1 to a4 are the fitting parameters, S sh These are the scaling transformation coefficients. The detection bounding box for an existing trajectory;

[0093] The predicted tracking box position is updated based on the type of the target being tracked. For pedestrian targets, the updated predicted tracking box... for:

[0094]

[0095] For vehicle and non-motorized vehicle targets, the updated predicted tracking bounding box is:

[0096]

[0097] GIOU is used to calculate the kinematic score.

[0098]

[0099] Where C is the minimum bounding box of A and B. Compared with Intersection over Union (IOU), GIOU can measure the distance between A and B without overlapping parts.

[0100] The FAST operator is used to extract the target edge within the detection box. A distance criterion is designed to merge corner points and extract the contour key points. If there are other corner points within a discrete circle with a radius of 3 for a certain corner point, these corner points are grouped into a corner point cluster PM. This process continues until no more corner points can be added to the corner point cluster. Let K be the number of circles required to connect these corner points, then the total number N of corner points in the merged cluster is:

[0101]

[0102] The rule for merging corner points within a cluster is to maximize the sum of the total Euclidean distances between the corner points, i.e.:

[0103]

[0104] Where PM′ represents the merged point cluster, PM q Let N be the number of corner points and the other point clusters. Let ||a,b||2 be the L2 norm of a and b, representing the Euclidean distance.

[0105] For the point cluster PM′ after the detection box is merged i The feature points need to be multiplied by the scaling factor S. sh Obtain the expected feature point cluster Specifically, for a point PM′ within a cluster with centroid Q... i,n :

[0106]

[0107] Shape score calculated using cosine similarity. The scaling coefficient S is updated by the distance between the feature point clusters and their corresponding centroids. sh Let the detection frame cluster PM′ be... i With tracking box point cluster PM′ j The corresponding centers of gravity are Q i With Q j ,but:

[0108]

[0109] When a target is associated for the first time, there is no S obtained from the previous association. sh If S is involved in the calculation, then let S be an integer. sh =1, meaning the default scale remains unchanged.

[0110] Furthermore, the specific steps of S404 are as follows: Select missing trajectories that can be stitched together based on the effective frame interval:

[0111] 0 <f<f u

[0112] 0<Δx <V max *F

[0113] Δy <w h (twenty four)

[0115] Wherein, (Δx,Δy) is the distance between the two points at the break point of the two selected break trajectories, and F represents the number of frames missing due to breakage; the two break trajectories that meet this requirement are used as candidate break trajectories for splicing.

[0116] Substituting the stimulus-response car-following model to fit the vehicle trajectory along the lane line:

[0117]

[0118] Where x n (t) represents the coordinates of the vehicle's forward direction, n represents the sequence number of the trajectory point to be completed, t represents time, T represents the total duration of the trajectory to be completed, and λ is determined by the driver characteristics of the target vehicle and does not change with time. It can be obtained by calibrating the motion information of the beginning and end of the trajectory to be matched in time and space with the position and speed information of the corresponding preceding vehicle.

[0119] The target's trajectory perpendicular to the lane line and the tracking frame size are stitched together according to the following rules:

[0120] y f =f3(x f )

[0121]

[0122]

[0123] Where f represents the number of frames being stitched (f <F),(x f ,y f (x) represents the coordinates of the completed point; e ,y e ,l e ,w e (x) represents the coordinates and dimensions of the last point of the preceding trajectory. s ,y s ,l s ,w s ) indicates the coordinates of the first point and the length and width of the subsequent trajectory.

[0124] The beneficial effects of this invention include:

[0125] 1. This paper proposes an image fusion algorithm based on a dual-discriminator adversarial network to enhance the target information in visible light and infrared fused images and improve the applicability of the fusion algorithm to traffic scenes. The original network structure is improved in three aspects: First, the target network and background network are divided, and the corresponding image blocks of the source images are segmented to enhance the target fusion effect. Second, considering the image fusion under the condition of imbalance between the two source images, bilateral filtering weights are added to the network structure and loss function design to represent the information content of the source images, retain the source image features with larger information content, and improve the overall network's adaptability to poor lighting scenes. Finally, the generator loss function is improved by combining SSIM and L1 norm to improve the fusion effect of target edges in the result.

[0126] 2. A lightweight multi-target tracking algorithm based on visible light and infrared fusion images is proposed to improve the computation speed of the tracking algorithm. The algorithm takes into account the characteristics of specific target types, oblique acquisition viewpoints, and obvious target contours in traffic fusion image scenarios. It comprehensively designs shape and position scores from both kinematic and contour information aspects to achieve target tracking. In the kinematic part, the movement range of different targets is divided by combining the movement laws of people, motor vehicles, and non-motor vehicles, with displacement distance and direction as variables, to improve the tracking accuracy. In the contour part, contour feature points based on the FAST operator are used for similarity comparison to simplify the contour feature recognition steps and improve the algorithm efficiency.

[0127] 3. A multi-source target trajectory extraction framework based on visible light and infrared fusion images is constructed to improve the quality of target trajectory extraction results. Combining the image fusion implementation process with the single-source target trajectory extraction process, a microscopic traffic flow model is applied for trajectory completion and correction, realizing automated end-to-end target trajectory extraction and traffic parameter generation of visible light and infrared fusion image sequences, thus improving the efficiency of completing image fusion trajectory extraction tasks. This framework is applicable to traffic target trajectory extraction under normal lighting conditions and poor lighting conditions, expanding the application scenarios of trajectory extraction tasks. Compared with the single-source trajectory extraction framework, the trajectory accuracy and precision proposed in this paper are higher, thus improving the quality of trajectory extraction results. Attached Figure Description

[0128] Figure 1 This is a flowchart illustrating the heterogeneous image fusion anti-halo technology of the present invention.

[0129] Figure 2 This is a schematic diagram of the network structure of the target network group generator of the present invention;

[0130] Figure 3 This is a schematic diagram of the network structure of the target network group discriminator of the present invention;

[0131] Figure 4 This is a schematic diagram of the background network group generator structure of the present invention;

[0132] Figure 5 This is an illustration of an image sequence from an embodiment of the present invention;

[0133] Figure 6 This is a spatiotemporal trajectory diagram of an embodiment of the present invention. Detailed Implementation

[0134] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0135] like Figures 1-6 The image shows a method for extracting the trajectory of multiple targets vehicles based on pixel-level image fusion. The specific steps are as follows:

[0136] Step 1: Acquire heterogeneous image sequence data and road environment information. This example uses self-shot visible light and infrared video for demonstration. The shooting location is the Xiaoxingqiao bus stop overpass in Jiangning District, Nanjing. The image sequence and fusion effect are shown below. Figure 5 As shown;

[0137] Step 2: Perform image preprocessing operations, including cropping and scaling the medium-to-high resolution image (generally a visible light image); using the Shi-Tomasi operator for corner detection, employing the KNN algorithm for feature point association, and performing dual-source image registration. The specific steps are as follows:

[0138] 2.1 Perform centrally symmetrical scaling and cropping on the visible light image to unify the pixel information of the visible light and infrared images without changing the field of view of the visible light image;

[0139] 2.2, Use the Shi-Tomasi operator to find feature corner points in the background region;

[0140] 2.3, for the feature corner point p in the background region i First, use the following formula to determine p i Is it in a dense cluster of corner points?

[0141]

[0142]

[0143] Where ||p i ,p j ||2 represents feature point p i With p j European distance, Dis th To determine the distance threshold for corner density, D th This is the proportional threshold;

[0144] If p i If it is not in a dense cluster of corner points, it is directly included in the candidate feature point set Cd. i Given a cluster of dense corner points, with N0 being the number of corner points in the cluster, calculate the number N corner points that should be retained:

[0145]

[0146] That is, in a dense cluster of corner points, at least 1 corner point must be retained and at most N-1 corner points must be retained;

[0147] After determining the number of corner points N, iterate through all combinations of corner points with a quantity of N in the corner point cluster, select the combination with the largest spacing as the corner points to be retained, and include them in the candidate feature point set Cd;

[0148] 2.4 For the candidate feature point set Cd, the KNN algorithm is used to associate feature point pairs in the visible light and infrared source images, and the connection between matching point pairs is represented as f(x n )=kMx n +b, where M is an orthogonal 2x2 matrix with a determinant of 1, t is a 2x1 transformation vector, and k is a scaling factor. Minimize the following formula to find the slope of the line connecting the correct matching point pairs:

[0149]

[0150] Where λ>0 is the coordination ratio coefficient, σ is the standard deviation of the candidate feature point set, (x n ,y n () represents the image coordinates of feature point n;

[0151] The optimal connection equation f(x) n )=kMx n +b is denoted as f(x) n )=Kx n +B, where K and B are the slope and constant of the line equation, respectively. Point pairs whose line equation constants fall within the following ranges are selected as the points for registration:

[0152] K-3σ k ≤K i ≤K+3σ k

[0153] B-3σ b ≤B i ≤B+3σ b

[0154] 2.5. Obtain the set of point pairs to be registered, establish the perspective transformation matrix to generate the registration scheme, and generate the registered visible light and infrared source images;

[0155] Step 3: Design a dual-discriminator generative adversarial network (FT-GAN) to perform pixel-level fusion of dual-source images, enhancing the target details and anti-halo capability of the fused image. The specific steps are as follows:

[0156] 3.1 The K-Means algorithm is used to cluster trajectories to extract the size information of targets within the region and obtain the size of segmented image patches;

[0157] 3.2. E-Net is used for semantic segmentation of the source image, and a target area ratio threshold is designed to determine whether a certain image patch belongs to the target or the background:

[0158] N = max{l mean ,w mean}+α#(2-11)

[0159]

[0160] Among them l mean ,w mean The length and width of the center bounding box obtained by the K-means algorithm are both integers, and α is a constant integer that makes N satisfy the network structure requirements; patch n For image patch n, tar and bgd represent that the image is classified as target and background, respectively. th This is a threshold representing the area of ​​the target region. If the area exceeds this threshold, the image patch is considered the target; otherwise, it is considered the background. These represent the areas occupied by the target in the visible light and infrared image blocks, respectively.

[0161] 3.3 The target image patch and background image patch are respectively substituted into the target network group and background network group for weight training and image fusion. The specific structure of the target network group is as follows:

[0162] 3.3.1, The target network generator network structure is as follows: Figure 2 As shown, a U-Net network is used for pseudo-fusion image generation. A 7-layer network is designed, and a convolutional layer with a stride of 2 is used to reduce the resolution of the feature map. The convolutional kernel is set to 4*4 and the padding is 1 to ensure that the feature map and the input and output image sizes are consistent. The activation function is designed as Leaky-ReLU, which alleviates the problem of some neurons being in a silent state in the later stage of training to some extent.

[0163] 3.3.2, The target network discriminator structure is as follows: Figure 3As shown, a PatchGAN structure is used to design the discriminator network. Three convolutional layers with a stride of 2 are used for downsampling to obtain a dimensionality-reduced feature map, which is then padded with zeros of size 2. Next, a convolutional layer with a stride of 1 is used to accurately extract features, and the feature values ​​are normalized by BatchNorm (BN) + LeakyReLU. Finally, after padding the feature map again, a convolutional layer with 1 kernel is used to merge the dimensions of the feature map, and an n*n score matrix is ​​directly output through the tanh activation function.

[0164] 3.3.3 The loss function of the target network generator is calculated as follows:

[0165]

[0166] Where L sSIM L is the structural loss calculated using the structural similarity index SSIM. L1 The edge loss is calculated using the L1 norm, and δ1 and δ2 are weight parameters that measure the importance of structural loss and edge loss in the target network.

[0167] L SSIM The calculation method is as follows:

[0168]

[0169] in, The SSIM loss for pixel p can be expressed as:

[0170]

[0171] in Indicates the expected image variance. This represents the covariance between the expected image and the pseudo-fused image;

[0172] L L1 The calculation method is as follows:

[0173]

[0174]

[0175] Where the weight parameter w(I) k The weighting function is the same as that in the SSIM loss;

[0176] 3.3.4 The loss function of the target network discriminator is calculated as follows:

[0177]

[0178]

[0179] Where p1 and p2 are weighting coefficients that reflect the amount of information in the source image, p1 represents the visible light image and p2 represents the infrared image, which are calculated by the bilateral filter weighting function;

[0180] 3.4 The specific network structure of the background network group is as follows:

[0181] 3.4.1, Background Network Group Generator Network Structure as follows Figure 4 As shown, a tensor addition network structure is used to generate fused images. Specifically, for the input visible light and infrared training samples, the network first extracts low-level features such as edges and corners and high-level features such as semantics from the samples through two convolutional layers (Conv1, Conv21 and Conv2, Conv22). Then, the proposed dual-source sample feature maps F21 and F22 are weighted and added to obtain the fused feature map F3, where the weights p1 and p2 are calculated through bilateral filtering weights. Finally, the fused image is reconstructed from F3 through three convolutional layers (Conv3, Conv4, Conv5).

[0182] 3.4.2 The background network group discriminator network structure is the same as the target network group;

[0183] 3.4.3 The loss function of the background network group generator is calculated as follows:

[0184]

[0185] The generator adversarial loss is given by the discriminator in the adversarial network, L. con For content loss, λ is the weighting coefficient that adjusts the importance of the two types of loss;

[0186] Adversarial Loss: In this paper, the adversarial loss is taken as the cross-entropy of the discriminator's discrimination results on the pseudo-fused image, that is:

[0187]

[0188] Among them, D v (G(v,i)) represents the discriminator D. v The probability of distinguishing a pseudo-fused image G(v,i) from a true image, D i (G(v,i)) represents the discriminator D. i The probability of identifying a pseudo-fused image G(v,i) as true;

[0189] Content loss is defined as follows:

[0190]

[0191]

[0192] 3.4.4 The loss function of the background network group discriminator is the same as that of the target network group;

[0193] 3.5, restore the image patch to the original image position, and output a fused image with anti-halo capability;

[0194] Step 4: Use the YOLOv5 algorithm for fused image target detection, and design a shape-position discrimination multi-target tracking algorithm to realize the association of detection boxes and trajectory generation from multiple angles of kinematics and target contours, thereby improving the running speed of the target detection and tracking part. The specific steps are as follows:

[0195] 4.2 Initialize target tracking algorithm parameters, design short-time sector displacement domain to estimate target movement range, and initially screen candidate tracking boxes. The specific steps are as follows:

[0196] 4.2.1 Calculate the half-angle θ of the sector displacement domain search:

[0197]

[0198] Where θ0 represents the physical limit deflection angle of the real target, and V0 represents the physical limit displacement of the real target. h0 represents the average intersection-union ratio between the detection bounding box and the actual target position, and h0 represents the height of the target detection bounding box.

[0199] 4.2.2 Calculate the search radius V of the sector displacement domain:

[0200]

[0201] Where w0 is the width of the target detection box, and DT is the maximum displacement coefficient, which is determined based on the target type and physical limits.

[0202] 4.2.3 Filter candidate tracking boxes within a sector area with search radius V and search half angle θ, and retain the position and size information of the tracking boxes within the area;

[0203] 4.3 Design a shape and position discrimination weighted scoring system. Update the predicted tracking box position according to the target type and calculate the kinematic score of the candidate tracking box; obtain the target similarity based on the cosine distance of the tracking box feature points, calculate the tracking box shape score, and estimate the inter-frame association result based on the comprehensive shape and position discrimination score. The specific steps are as follows:

[0204] 4.3.1 Calculate the predicted tracking box position

[0205]

[0206] Where a1 to a4 are the fitting parameters, S sh These are the scaling transformation coefficients. The detection bounding box for an existing trajectory;

[0207] 4.3.2 Update the predicted tracking box position according to the type of the target being tracked. For pedestrian targets, the updated predicted tracking box... for:

[0208]

[0209] For vehicle and non-motorized vehicle targets, the updated predicted tracking bounding box is:

[0210]

[0211] 4.3.3, Calculate the kinematic score using GIOU.

[0212]

[0213] Where C is the minimum bounding box of A and B. Compared with Intersection over Union (IOU), GIOU can measure the distance between A and B without overlapping parts, and has a wider range of applications.

[0214] 4.3.4 The FAST operator is used to extract the target edge within the detection box. A distance criterion is designed to merge corner points and extract the contour key points. If there are other corner points within a discrete circle with a radius of 3 for a certain corner point, these corner points are grouped into a corner point cluster PM until no more corner points can be added to the corner point cluster. Let K be the number of circles required to connect these corner points, then the total number N of corner points in the merged cluster is:

[0215]

[0216] The rule for merging corner points within a cluster is to maximize the sum of the total Euclidean distances between the corner points, i.e.:

[0217]

[0218] Where PM′ represents the merged point cluster, PM q Let N be the number of corner points and the other point clusters. Let ||a,b||2 be the L2 norm of a and b, representing the Euclidean distance.

[0219] 4.3.5, for the point cluster PM′ after the detection frame is merged i The feature points need to be multiplied by the scaling factor S. sh Obtain the expected feature point cluster To overcome the problem of scale inconsistency between the detection box and the tracking box, specifically for a point PM′ within a point cluster with centroid Q. i,n :

[0220]

[0221] 4.3.6 Calculate shape score using cosine similarity The scaling coefficient S is updated by the distance between the feature point clusters and their corresponding centroids. sh Let the detection frame cluster PM′ be... i With tracking box point cluster PM′ j The corresponding centers of gravity are Q i With Q j ,but:

[0222]

[0223] When a target is associated for the first time, there is no S obtained from the previous association. sh If S is involved in the calculation, then let S be an integer. sh =1, meaning the default scale remains unchanged;

[0224] Step 5: Perform trajectory post-processing operations. Design a trajectory classification process to retain complete trajectories, delete ghost trajectories, filter fragmented trajectories, and complete missing trajectory data based on the stimulus-response model to improve the rationality of vehicle trajectories. Convert vehicle trajectories from image coordinates to geodetic coordinates, and use the EEMD algorithm for trajectory denoising to improve the overall accuracy of the trajectories. Finally, output high-precision vehicle trajectory and traffic parameter data. The specific steps are as follows:

[0225] 5.1 Filtering missing trajectories that can be stitched together based on the effective frame interval:

[0226] 0 <f<f u

[0227] 0<Δx <V max *F

[0228] Δy <w h

[0229] Wherein, (Δx,Δy) is the distance between the two points at the break point of the two selected break trajectories, and F represents the number of frames missing due to breakage; the two break trajectories that meet this requirement are used as candidate break trajectories for splicing.

[0230] 5.2, Substitute the stimulus-response car-following model to fit the vehicle trajectory along the lane line direction:

[0231]

[0232] Where x n (t) represents the coordinates of the vehicle's forward direction, n represents the sequence number of the trajectory point to be completed, t represents time, T represents the total duration of the trajectory to be completed, and λ is determined by the characteristics of the target vehicle's driver and does not change with time. It can be obtained by calibrating the spatiotemporal motion information of the beginning and end of the trajectory to be matched with the position and speed information of the corresponding preceding vehicle.

[0233] 5.3, stitch the target's trajectory perpendicular to the lane line and the tracking frame size according to the following rules:

[0234] y f =f3(x f )

[0235]

[0236]

[0237] Where f represents the number of frames being stitched (f <F),(x f ,y f (x) represents the coordinates of the completed point; e ,y e ,l e ,w e (x) represents the coordinates and dimensions of the last point of the preceding trajectory. s ,y s ,l s ,w s () indicates the coordinates of the first point and the length and width of the subsequent trajectory;

[0238] 5.4. Perform projection transformation based on the perspective model, which is equivalent to the superposition of linear transformation and translation transformation on the specific coordinate position. The specific perspective model is as follows:

[0239] [x′,y′,w′]=[u,v,w]×M

[0240]

[0241]

[0242]

[0243] Where u, v represent the original image coordinates, x, y are the image coordinates after perspective transformation, w, w′ represent the scaling parameters, and M is the perspective transformation matrix;

[0244] 5.5. The EEMD algorithm is used for trajectory denoising. EEMD decomposes the trajectory as a signal input into a synthesis of multiple signals of different intensities. The number of decomposed signals is 1 / time of the trajectory length. These decomposed signals are called IMFs, which are regarded as the superposition of effective signals and noise signals.

[0245] X i (t)=x(t)+w i (t)#(4-19)

[0246] Among them, X i x(t) is the trajectory signal, x(t) is the effective signal, and w i (t) represents the noise signal;

[0247] An energy threshold is set to filter each decomposition signal; the energy of each decomposition signal is represented as:

[0248]

[0249] Where num represents the total number of scatter points contained in the signal, and c j (k) represents the point set of the decomposed signal;

[0250] If the energy of the decomposition signal satisfies

[0251] log2E j >0#(4-21)

[0252] The signal is then considered valid; the valid signals are superimposed to form x(t);

[0253] Output the denoised high-precision vehicle trajectory as the trajectory output result of the multi-source data target trajectory extraction framework;

[0254] The spatiotemporal trajectory diagram extracted in this example is as follows: Figure 6 As shown.

[0255] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for extracting the trajectory of multiple targets vehicles based on pixel-level image fusion, comprising the following steps: S1. Acquire heterogeneous image sequence data and road environment information; the image source is visible light and infrared images; S2. Image preprocessing; cropping and scaling transformation of the visible light image in the image pair; The Shi-Tomasi operator is used for corner detection, and the KNN algorithm is used for feature point association and dual-source image registration. S3. Dual-source image fusion: Design a dual-discriminator generative adversarial network FT-GAN to implement pixel-level fusion of dual-source images, thereby enhancing the target details and anti-halo capability of the fused image. S4. Fusion image target detection and tracking: The YOLOv5 algorithm is used for fusion image target detection. A shape and position discrimination multi-target tracking algorithm is designed to realize the association of detection boxes and trajectory generation from multiple angles of kinematics and target contour, thereby improving the running speed of the target detection and tracking part. The shape and position discrimination multi-target tracking algorithm designs a shape and position discrimination weighted scoring system, updates the predicted tracking box position according to the target type, and calculates the kinematic score of the candidate tracking box; Target similarity is obtained based on the cosine distance of feature points in the tracking box, the shape score of the tracking box is calculated, and the inter-frame association result is estimated based on the comprehensive shape and position discrimination score. S5. Perform trajectory post-processing operations, design a trajectory classification process to retain complete trajectories, delete ghost trajectories, filter fragmented trajectories, and complete missing trajectory data based on stimulus-response model to improve the rationality of vehicle trajectories; convert vehicle trajectories from image coordinates to geodetic coordinates, and use the EEMD algorithm to denoise the trajectories to improve the overall accuracy of the trajectories; finally, output high-precision vehicle trajectory and traffic parameter data. The specific steps of the dual-source image pixel-level fusion in S3 are as follows: S301. Use the K-Means algorithm to cluster trajectories to extract the size information of targets within the region and obtain the size of segmented image blocks; S302. Use E-Net to perform semantic segmentation on the source image, and design a target area ratio threshold to determine whether a certain image patch belongs to the target or the background: ; (5); among which The length and width of the center bounding box obtained by the K-means algorithm are both integers. To ensure that N is a constant integer that satisfies the network structure requirements; For image patch n, tar and bgd represent that the image is classified as target and background, respectively. This is a threshold representing the area of ​​the target region. If the area exceeds this threshold, the image patch is considered the target; otherwise, it is considered the background. These represent the areas occupied by the target in the visible light and infrared image blocks, respectively. S303. Substitute the target image patch and the background image patch into the target network group and the background network group respectively for weight training and image fusion. S304. Set the background network group network structure; S305. Restore the image block to the source image position and output a fused image with anti-halo capability.

2. The multi-target vehicle trajectory extraction method based on pixel-level image fusion according to claim 1, characterized in that, In S1, the image source is visible light and infrared images, which are the same scene image data collected at the same time and place.

3. The multi-target vehicle trajectory extraction method based on pixel-level image fusion according to claim 1, characterized in that, S2 employs an improved FAST neighborhood pixel feature point matching strategy, with the following specific steps: S201. Use the Shi-Tomasi operator to find the feature corner points in the background region; S202, For feature corner points in the background area First, use the following formula to determine. Is it in a dense cluster of corner points? ; (1); where For feature points and European distance, To determine the distance threshold for corner density, This is the proportional threshold; if If it is not in a dense cluster of corner points, it is directly included in the candidate feature point set Cd. Given a dense cluster of corner points, let the number of corner points in the cluster be... Calculate the number of corner points N that should be retained: （2） That is, a minimum of 1 corner point and a maximum of N-1 corner points must be retained in a dense cluster of corner points; After determining the number of corner points N, iterate through all combinations of corner points with a quantity of N in the corner point cluster, select the combination with the largest spacing as the corner points to be retained, and include it in the candidate feature point set Cd to complete the coarse screening of feature points. S203. Find the FAST4 neighborhood descriptor for each feature point in the candidate feature point set Cd, and expand the descriptor to 8 neighborhoods according to the Manhattan distance equality condition. S204. Calculate the 8-neighborhood descriptor similarity features: （3） in The normalized distance between the two descriptors. To describe the normalized pixel difference: ; (4); where x, y are the coordinates of the descriptor image, and L, W are the image length and width. For the image pixel value range; S205, select The top K feature points are selected as matching feature points after fine screening and used in perspective transformation generation.

4. The method for extracting multi-target vehicle trajectories based on pixel-level image fusion according to claim 3, characterized in that, The specific structure of the target network group in S303 is as follows: S3031. A U-shaped network is used for pseudo-fusion image generation. A 7-layer network is designed, and a convolutional layer with a stride of 2 is used to reduce the resolution of the feature map. The convolutional kernel is set to 4*4 and the padding is 1 to ensure that the feature map and the input and output image sizes are consistent. The activation function is designed as Leaky-ReLU. S3032. The target network adopts the PatchGAN structure to design the discriminator network. It uses three convolutional layers with a stride of 2 for downsampling to obtain the dimensionality-reduced feature map and performs zero-value padding of size 2. Next, it uses a convolutional layer with a stride of 1 to accurately extract features and normalizes the feature values ​​through BatchNorm + LeakyReLU. Finally, after padding the feature map again, it uses a convolutional layer with 1 kernel to merge the dimensions of the feature map and directly outputs an n*n score matrix through the tanh activation function. S3033, The loss function of the target network generator is calculated as follows: （6） in The structural loss is calculated using the structural similarity index SSIM. The edge loss is calculated using the L1 norm. , Weight parameters used to measure the importance of structural loss and edge loss in the target network; The calculation method is as follows: （7） in, The SSIM loss for pixel p can be expressed as: （8） in Indicates the expected image variance. This represents the covariance between the expected image and the pseudo-fused image; The calculation method is as follows: ； （9） Among them, weight parameters The weighting function is the same as that in SSIM loss; S3034. The loss function of the target network discriminator is calculated as follows: ； （10） Where p1 and p2 are weighting coefficients reflecting the amount of information in the source image, p1 represents the visible light image and p2 represents the infrared image, which are calculated by the bilateral filtering weighting function.

5. The method for extracting multi-target vehicle trajectories based on pixel-level image fusion according to claim 4, characterized in that, The specific network structure of the S304 background network group is as follows: S3041. The background network generator uses a tensor addition network structure to generate fused images. S3042. The background network group discriminator network structure is the same as the target network group. S3043, The loss function of the background network group generator is calculated as follows: （11） The generator adversarial loss is given by the discriminator in the adversarial network. For content loss, To adjust the weighting coefficients for the importance of the two types of losses; Adversarial Loss: In this paper, the adversarial loss is taken as the cross-entropy of the discriminator's discrimination results on the pseudo-fused image, that is: （12） in, Discriminator For pseudo-fused images The probability of being true. Discriminator For pseudo-fused images The probability of being true; Content loss is defined as follows: ； （13） S3044. The loss function of the background network group discriminator is the same as that of the target network group.

6. The method for extracting multi-target vehicle trajectories based on pixel-level image fusion according to claim 1, characterized in that, The specific steps of the shape and position discrimination multi-target tracking algorithm of S4 are as follows: S401. Perform background compensation on the image sequence, fix the pixel spatial information of the image sequence, implement coordinate transformation, and convert the target detection box data from the image coordinate system to the top-view Frenet coordinate system. S402. Initialize the target tracking algorithm parameters, design a short-time sector displacement domain to estimate the target movement range, and initially screen candidate tracking boxes; S403. Design a shape and position discrimination weighted scoring system, update the predicted tracking box position according to the target type, and calculate the kinematic score of the candidate tracking box. Target similarity is obtained based on the cosine distance of feature points in the tracking box, the shape score of the tracking box is calculated, and the inter-frame association result is estimated based on the comprehensive shape and position discrimination score. S404. Design a multi-target trajectory verification method for traffic scenarios, extract complete trajectories within the scenario, eliminate invalid trajectories, collect discontinuous trajectories and input them into the car-following model based on the spatiotemporal information of both ends, complete the missing trajectories, search for erroneous tracking trajectories by combining traffic parameters and time-varying trajectory characteristics, and correct the tracking results.

7. The method for extracting multi-target vehicle trajectories based on pixel-level image fusion according to claim 6, characterized in that, The specific steps in S402 are as follows: Calculate the half-angle of the sector displacement domain search : （14） in This represents the physical limit of the target's deflection angle. This represents the physical limit displacement of the actual target. This represents the average intersection-union ratio (IoU) between the detection bounding box and the actual target location. Indicates the height of the target detection bounding box; Calculate the search radius V of the sector displacement domain: （15） in, DT is the width of the target detection box, and DT is the maximum displacement coefficient, which is determined based on the target type and physical limits. Within the search radius V, the search half-angle Candidate tracking boxes are filtered within a sector-shaped area, and the position and size information of the tracking boxes within the area are retained.

Images