A method for extracting a complete trajectory of a moving target in a traffic scene

Abstract

The application provides a traffic scene moving target complete trajectory extraction method, comprising the following steps: accessing video data in a complex traffic scene, tracking a moving target in the video based on target detection, obtaining a target type Type, a target foreground image objPic, target center point coordinates Obj(x center ,y center ), a minimum circumscribed rectangle, and left upper corner coordinates (x L ,y L ) and right lower corner coordinates (x R ,y R ) of the minimum circumscribed rectangle, and sequentially connecting the target center point coordinates Obj(x center ,y center ) of the target in continuous frames of the video to form a target trajectory.

Description

Technical Field

[0001] This invention relates to the field of surveillance video trajectory extraction technology, and in particular to a method for extracting the complete trajectory of a moving target that takes into account the multi-level semantic features of traffic scenes. Background Technology

[0002] Intelligent extraction of the complete trajectory of a target within the field of view can provide spatiotemporal foundational information for intelligent traffic supervision, such as target behavior analysis based on surveillance videos. However, in complex traffic scenarios such as mutual occlusion of moving targets and blurred video surveillance images, the trajectory of a moving target in the video may be intermittent, discontinuous, or unclear. Furthermore, moving targets move in real geographic space, while the target trajectory in surveillance video depends on the existence of image space. However, current intelligent analysis and application of moving targets in traffic scenarios need to be conducted in real geographic space. Establishing a mapping relationship between the trajectory of moving targets in image space and geographic space is also a key technical problem faced by traffic supervision departments in intelligent analysis. Therefore, how to accurately detect moving targets and precisely obtain their complete movement trajectory within geographic space is not only the foundation of traffic behavior analysis and intelligent traffic monitoring and early warning, but also one of the important topics that urgently need in-depth research in the field of video GIS (Sharma 2022; Song 2022; Luo 2021).

[0003] Against this backdrop, in response to the aforementioned problem of how to completely extract target trajectories, considering that each fixed-position camera has a specific field of view, and the traffic scene presented in its monitoring video consists of several keyframes, each containing several moving targets, the corresponding targets in keyframe images at different spatiotemporal nodes, and different targets within the same keyframe image carry a large amount of scene structural semantic information at different levels, such as image pixel features, target geometric features, temporal relationships between targets, and spatial relationships between targets. Furthermore, the imaging of targets in image space exhibits a "nearer is larger, farther is smaller" characteristic, while the target trajectory sequence exhibits a "closer is closer, farther is more dispersed" characteristic in terms of distance from the camera. These factors can all serve as references for extracting the complete target trajectory. Therefore, this paper proposes a method for extracting the complete trajectory of moving targets that takes into account the multi-level semantic features of traffic scenes, aiming to balance the completeness and reliability of intelligent target trajectory extraction, approximate the authenticity of the target trajectory, better reconstruct the target's movement trajectory, and provide a feasible technical method reference for intelligent traffic behavior supervision. Summary of the Invention

[0004] To address the problems in the prior art, this invention provides a method for extracting the complete trajectory of a moving target in a traffic scene (TraEA).

[0005] A method for extracting the complete trajectory of a moving target in a traffic scene, comprising the following steps:

[0006] a) Access video data from complex traffic scenarios, perform target detection-based tracking of moving targets in the video, and obtain the target type (Type), target foreground image (objPic), and target center point coordinates (Obj(x)). center ,y center The smallest bounding rectangle and the coordinates of its top-left corner (x) L ,y L ) and the coordinates of the lower right corner (x R ,y R Based on the center point coordinates of the target in consecutive video frames, Obj(x) center ,y center ) Connect them sequentially to form the target trajectory Tra i ={(x1,y1),(x2,y2),…,(x n ,y n )};

[0007] Based on feature point tracking, multiple feature corner points of a moving target are obtained. A set of N feature point trajectories, Traclus, is constructed from the coordinates of these feature corner points across consecutive video frames. i ={Traclus1,Traclus2,...,Traclus n}; Based on the Traclus set of feature points of the moving target i The orientational relationship between the target trajectory and the minimum bounding rectangle, i.e., the determination of whether a point is inside the polygon, is used to obtain the type information of the target trajectory and feature point trajectories. The target type includes two main categories: vehicles and pedestrians. Pedestrians include cyclists and pedestrians. Suppose that the trajectories of five feature points A, B, C, D, and E fall into the target trajectory Tra. i If the feature points A, B, C, D, and E are within the smallest bounding rectangle of the target, then the trajectories of all five feature points belong to the target Tra. i Target Tra i The type information is the N feature point trajectories Traclus. i ={Traclus1,Traclus2,...,Traclus n Type information: pedestrians (cyclists and pedestrians) / vehicles;

[0008] b) Based on step a), the target trajectory Tra can be obtained. i ={(x1,y1),(x2,y2),...,(x n y n )} and N feature point trajectories Traclus i ={Traclus1,Traclus2,...,Traclus n Based on this, by comparing the trajectory of each feature point with the target detection trajectory Tra...i The vector product judge is used to obtain the trajectories of N feature points. i Compared to Tra i Positional relationship, let the target trajectory Tra i The starting point is Tra start (x start y start The endpoint is Tra end (x end y end Traclus, feature point trajectory i The center point coordinates are Traclus mid (x mid y mid According to the formula for calculating the vector product:

[0009] judge=(x end -x start )×(y mid -y start )-(y end -y start )×(x mid -x start )

[0010] Calculate N Traclus i Located in Tra i The positional relationship between the left and right sides, i.e., when judge < 0, Traclus i Located in Tra i To the right of Tra, one can obtain the N feature point trajectories located at Tra. i The set of trajectories on the right is:

[0011] Traclus ri ={Traclus r1 Traclus r2 Traclus rn};

[0012] When judge > 0, Traclus i Located in Tra i To the left of Tra, among the N feature point trajectories, those located in Tra... i The set of trajectories on the left is:

[0013] Traclus li ={Traclus l1 Traclus l2 Traclus ln};

[0014] Where i is the number of moving targets, n is the number of trajectories of a single target, rn is the number of feature point trajectories on the right side of a single target, and ln is the number of feature point trajectories on the left side of a single target;

[0015] c) For the left trajectory Traclus in step b), li and the right track Traclus ri Perform cluster analysis, let:

[0016] The set of coordinate points for each left-side trajectory is:

[0017] Traclus li ={((x) l1 y l1 ), (x l2 y l2 ), ..., (x ln y ln ))}

[0018] The set of coordinate points for each right-hand trajectory is:

[0019] Traclus ri ={((x) r1 y r1 ), (x r2 y r2 ), ..., (x rn y rn ))};

[0020] Cluster analysis mainly includes four steps: trajectory average vector calculation, coordinate rotation, horizontal scan, and rotation back to the original coordinates.

[0021] Average vector calculation:

[0022] Get:

[0023] left-side trajectory sequence Traclus li ={Traclus l1 Traclus l2 Traclus ln}、

[0024] Left trajectory sequence vector and the right-hand trajectory sequence Traclus ri ={Traclus r1 Traclus r2 Traclus rn}、

[0025] Right-side trajectory sequence vector

[0026] And respectively for the left trajectory sequence vector and the right trajectory sequence vector Find the average vector:

[0027]

[0028] Using formulas respectively

[0029]

[0030]

[0031] Find the average vector

[0032] Where |lV| and |rV| are vectors and The model, (v lxn v lyn ) is the left trajectory vector of the nth line. The coordinates, (v rxn v ryn ) is the right-hand trajectory vector of the nth line. The coordinates of (V) lx V ly () represents the trajectory vector on the left. Coordinates and (V) rx V ry () represents the trajectory vector on the left. coordinate;

[0033] The angle rθ between the left-hand trajectory sequence Traclus and the unit vector along the x-axis will determine the trajectory sequence. li According to the average vector Rotate the direction by an angle lθ, and then rotate the right-hand trajectory sequence Traclus... ri According to the average vector Rotate the direction by an angle rθ to obtain:

[0034] Any coordinate point (x) on the left trajectory ln y ln The rotated X-coordinate cx ln =x ln coslθ-y ln sinlθ

[0035] Any coordinate point (x) on the left trajectory ln y ln The rotated Y-coordinate cy ln =y ln coslθ+x ln sinlθ

[0036] Any coordinate point (x) on the right-hand trajectory rn y rn The rotated X-coordinate cx rn =x rn cosrθ+y rn sinrθ

[0037] Any coordinate point (x) on the right-hand trajectory rn y rn The rotated Y-coordinate cy rn =y rn cosrθ-x rn sinrθ

[0038] Rotate each coordinate point on the trajectory according to the above steps to obtain the set of coordinates on the left side of the trajectory:

[0039] xTraclus li ={((cx) l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln ))}

[0040] The set of coordinates of the trajectory on the right is obtained as follows:

[0041] xTraclus ri ={((cx) r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn ))};

[0042] Horizontal scan: for the rotated left-side trajectory coordinate set xTraclus li The right-side trajectory coordinate set xTraclus ri Calculate by scanning at equal intervals along the average vector direction:

[0043] The set of n intersection points between the scan line and the left trajectory:

[0044] lInter i ={(lsx1′, lsy1′), (lsx2′, lsy2′),..., (lsx n ′, lsy n ′)}

[0045] The set of n intersection points between the scan line and the right-side trajectory:

[0046] rInter i ={(rsx1′, rsy1′), (rsx2′, rsy2′),..., (rsx n ′,rsy n ′)},

[0047] Calculated using the above formula:

[0048] The X-value of any coordinate of the left trajectory after clustering:

[0049]

[0050] The Y-value of any coordinate on the left side of the clustered trajectory:

[0051]

[0052] The X-value of any coordinate of the right-hand trajectory after clustering:

[0053]

[0054] The Y-value of any coordinate of the right trajectory after clustering:

[0055]

[0056] The coordinate sets of the left and right trajectories after cluster analysis are obtained:

[0057] The coordinate set of the left trajectory lTra i ={(cx l1 ′, cy l1 ′), (cx l2 ′, cy l2 ′), ..., (cx ln ′, cy ln ′)}

[0058] The coordinate set rTra of the right-hand trajectory i ={(cx r1 ′, cy r1 ′), (cx r2 ′, cy r2 ′), ..., (cx rn ′, cy rn ′)};

[0059] Then rotate the clustered left and right trajectories back to their original coordinates:

[0060] The X-value of any coordinate on the left side of the rotated trajectory: The Y-value of any coordinate on the left side of the rotated trajectory: The X-value of any coordinate on the right-hand trajectory after rotation:

[0061]

[0062] The Y-value of any coordinate on the right-hand trajectory after rotation:

[0063]

[0064] Finally, the rotated left-side trajectory cTra is obtained. left and the right-side trajectory cTra right :

[0065] Left track cTra left ={(cx l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln )}

[0066] Right track cTra right ={(cx r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn )};

[0067] d) Combining the corresponding high-definition remote sensing online imagery, select the iconic points q{q1(x1, y1), q2(x2, y2), ..., q... in the video frame image space. n (x n y n The selection criteria for landmark points are as follows: the target point must be obvious, clear, and easy to read on the remote sensing image, and have a corresponding position in the video frame; secondly, based on the high-precision remote sensing image map, corresponding points with the same name Q{Q1(X1, Y1), Q2(X2, Y2), ..., Q} are obtained in the geographic space. n (X n Y n Solve for the homography matrix H of the camera, and according to... The inverse matrix H of the homography matrix is ​​calculated. -1 , where (X) n Y n (x) represents the coordinates of any point on the remote sensing image. n ,yn) is the video frame with (X) n Y n The corresponding coordinates;

[0068] e) To further optimize the left-side trajectory cTra leftand the right-side trajectory cTra right After obtaining the high-precision complete trajectory, calculate the homography matrix H and the inverse matrix H' according to step d). -1 Establish the coordinate transformation relationship between the target trajectory in geographic space and image space:

[0069] According to the formula

[0070]

[0071] Solve cTra left ={(cx l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln )}

[0072] and cTra right ={(cx r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn The geographical coordinates of )};

[0073] That is, the geographical left-side trajectory gTra left ={(cX l1 ,cY l1 ), (cX l2 ,cY l2 ), ..., (cX ln ,cY ln )}

[0074] and the geography right-side trajectory gTra right ={(cX r1 ,cY r1 ), (cX r2 ,cY r2 ), ..., (cX rn ,cY rn )}

[0075] and the target detection trajectory Tra in step a) i ={(x1,y1),(x2,y2),...,(x n y n The set of geographic coordinates of )}

[0076] gTra i ={(X1, Y1), (X2, Y2),..., (X n Y n )}

[0077] And according to the formula

[0078]

[0079]

[0080] as well as

[0081]

[0082] Calculate the geographic left-side trajectory gTra left Relative to the target detection trajectory gTra i offset distance d ln And calculate with offset distance d ln For the hypotenuse and gTra left The X and Y coordinates of any point on the triangle are the included angle between the two legs of the right triangle. Calculate the geographic right-side trajectory gTra right Relative to the target detection trajectory gTra i offset distance d rn And calculate with offset distance d rn For the hypotenuse and gTra right The X and Y coordinates of any point on the triangle are the included angle between the two legs of the right triangle. in Using gTra left Formula for calculating the slope of the tangent line Obtain Using gTra right Formula for calculating the slope of the tangent line Get the lowercase x and y flags, including (cx rn cy rn ) and (cx ln cy ln All are coordinates in image space; uppercase X and Y are used as markers, including (cX). ln ,cY ln ) and (cX rn ,cY rn All coordinates are transformed from image space to geographic space.

[0083] f) d calculated based on step e) rn d ln and included angle To obtain a more accurate trajectory, the geographic left-side trajectory gTra needs to be... left and the geography right-side trajectory gTra right According to the offset distance d lnand d rn Move;

[0084] Obtain the left trajectory gTra left Any coordinate point (cX) ln ,cY ln ), calculate with hypotenuse d ln The included angle is The right-angled side X of the triangle ln Y ln That is (cX) ln ,cY ln ) to target detection trajectory gTra i The distance traveled, where:

[0085] cX ln gTra track detection of target i Distance moved X ln The calculation formula is:

[0086]

[0087] cY ln gTra track detection of target i Distance Y traveled ln The calculation formula is:

[0088]

[0089] Obtain the right-side trajectory gTra right Any coordinate point (cX) rn ,cY rn ), calculate with hypotenuse d rn The included angle is The right-angled side X of the triangle rn Y rn That is (cX) rn ,cY rn ) to target detection trajectory gTra i The distance traveled, where:

[0090] cX rn gTra track detection of target i Distance moved X rn The calculation formula is:

[0091]

[0092] cY rn gTra track detection of target i Distance Y traveled ln The calculation formula is:

[0093]

[0094] We obtain the coordinates (X) of each point on the left side of the geographic trajectory after its movement. rn Y rn ) and the coordinates (X) of each point on the right side of the geographic trajectory after its movement. ln Y ln );

[0095] Then, obtain the trajectory coordinate sequence after trajectory optimization, i.e., the left trajectory optimization sequence.

[0096] yTra left ={(cX l1 +X l1 ,cY l1 +Y l1 ), (cX l2 +X l2 ,cY l2 +Y l2 ), ..., (cX ln +X ln ,cY ln +Y ln )}

[0097] And the right-side trajectory optimization sequence

[0098] yTra right ={(cX r1 -X r1 ,cY r1 -Y r1 ), (cX r2 -X r2 ,cY r2 -Y r2 ), ..., (cX rn -X rn ,cY rn -Y rn )}

[0099] Finally, the inverse matrix H of the homography matrix calculated according to step d) -1 Then for yTra left andyTra right The result of performing a geographic coordinate to image coordinate transformation is:

[0100] The transformed left-side trajectory sequence:

[0101] pTra left ={(px l1 py l1 ), (px l2 py l2 ), ..., (px ln py ln )}

[0102] The transformed right-hand trajectory sequence:

[0103] pTra right ={(px r1 py r1 ), (px r2 py r2 ), ..., (px rn py rn )}

[0104] For pTra left pTra right and Tra i Calculate the average vector to obtain the vectors of the three trajectories:

[0105] and And according to the formula

[0106]

[0107] Calculate the average vector of the three trajectories.

[0108] Then follow the formula Calculate the average vector The angle θ between pTra and the unit vector of the x-axis left pTra right and Tra i The sequence is based on the average vector Rotate the direction by an angle θ, according to the formula tx ln =px ln cosθ-py ln sinθ and ty ln =py ln cosθ+px ln sinθ yields the rotated left-side coordinate set tTra left ={(tx l1 ty l1 ), (tx l2 ty l2 ), ..., (tx ln ty ln According to the formula tx rn =px rn cosθ-py rn sinθ and ty rn =py rn cosθ+px rn sinθ yields the rotated left-side coordinate set tTra right ={(txr1 ty r1 ), (tx r2 ty r2 ), ..., (tx rn ty rn )};

[0109] Then, the rotated trajectory is scanned at equal intervals according to the average vector direction, and the set Inter of the three intersection points of the scan line and the three trajectories is obtained. i ={(sx1′,sy1′),(sx2′,sy2′),(sx3′,sy3′)}, and according to the formula:

[0110]

[0111] and

[0112]

[0113] The coordinate set tTra of the clustered trajectory is obtained. i ={(tx1′, ty1′), (tx2′, ty2′),..., (tx n ′,ty n ′)};

[0114] Finally, according to the formula tx n =tx′ n ×cos(π+θ)-ty n ′×sin(π+θ) and ty n =ty′ n ×sin(π+θ)+ty n The function '×cos(π+θ) rotates the clustered trajectories back to their original coordinates, resulting in the optimized trajectory coordinate set tTra. i ={(tx1, ty1), (tx2, ty2),..., (tx n ty n )}.

[0115] This invention provides a method for extracting the complete trajectory of a moving target in a traffic scene, addressing the problem of incomplete target trajectory extraction within the field of view caused by situations such as mutual occlusion of moving targets and blurred target images in complex traffic scenes. Based on the field of view of a single camera's surveillance video, this invention includes two parts: coarse extraction of the complete target trajectory and trajectory optimization. First, in the coarse extraction stage, a method for coarse extraction of the complete trajectory of a moving target based on multi-layer semantic features is designed, integrating low-level semantic features such as scene, keyframes, and field of view, as well as mid-level temporal and spatial semantic features. On this basis, a trajectory optimization method is proposed based on the orientation, topology, and metric relationships of the mid-level spatial semantics of the scene, achieving high-precision complete trajectory optimization of the moving target within the field of view. Experimental test results show that: (1) In terms of the completeness of target trajectory extraction, compared with the two existing methods, namely target trajectory extraction based on target detection (TraT) and target trajectory extraction based on image features (TraD), the target trajectory length (TL) extracted by this technical solution based on the TraEA method is improved by 69.9%-76% and 80.05%-82.19%, respectively, and the target trajectory continuity (TC) is improved by 59.71%-60.65% and 77%-80.35%, respectively, reflecting that T (1) The TraEA method extracts the target trajectory with good integrity; (2) In terms of the accuracy of target trajectory extraction, the TraEA method in this technical solution reduces the mean square error (xMSE, yMSE) and root mean square error (xRMSE, yRMSE) of the target trajectory extraction by 65.43%-81.62%, 53.85%-89.27%, 63.85%-85.53%, and 53.93%-87.6% respectively compared with the TraT and TraD methods, indicating that the TraEA method extracts the target trajectory with high accuracy. Attached Figure Description

[0116] Figure 1 The diagram shown is a schematic of multi-layer semantic features;

[0117] Figure 2 The overall flowchart of the technical solution of the present invention is shown below;

[0118] Figure 3 The diagram shown illustrates the acquisition of feature point trajectory type information.

[0119] Figure 4 Figure (a) in the figure is a schematic diagram of the trajectory clustering analysis process used in the traditional target trajectory extraction method in the prior art; Figure 4 Figure (b) in the figure is a schematic diagram of the trajectory clustering analysis process in the target trajectory extraction method based on semantic information in this technical solution;

[0120] Figure 5 This is a schematic diagram of trajectory clustering analysis;

[0121] Figure 6 This is a schematic diagram illustrating the generation of the rotation trajectories on the left and right sides. Figure 6 Figure (a) in the diagram is a schematic diagram of solving the average vector on the left. Figure 6 Figure (b) in the diagram is a schematic diagram of the rotation of the trajectory coordinates on the left. Figure 6 Figure (c) in the diagram is a schematic diagram of the solution process for the average vector on the right. Figure 6 Figure (d) in the diagram is a schematic diagram of the rotation of the trajectory coordinates on the right.

[0122] Figure 7 This is a schematic diagram of the target trajectory extraction results;

[0123] Figure 8 This is a schematic diagram of the trajectory clustering method. Figure 8 Figure (a) in the diagram is a schematic diagram of the trajectory clustering method on the left. Figure 8 Figure (b) in the diagram is a schematic diagram of the trajectory clustering method on the right.

[0124] Figure 9 Figure (a) shows the left and right trajectory clustering analysis diagrams in the image space. Figure 9 Figure (b) shows the clustering and mapping relationships on the left and right sides of the geospatial map.

[0125] Figure 10 This is a schematic diagram of trajectory optimization. Figure 10 Figure (a) in the diagram is a schematic diagram of trajectory offset distance calculation. Figure 10 Figure (b) in the diagram is a schematic diagram of the trajectory movement coordinate calculation;

[0126] Figure 11 This is a diagram of the experimental scene.

[0127] Figure 12 Figure (a) shows a comparison of the offset distances on the left and right sides of Video#1 in the image space. Figure 12 Figure (b) shows a comparison of the offset distances of the left and right sides of Video#2 in the image space; Figure 12 Figure (c) is a comparison of the offset distances of the left and right sides of Video#1 in geographic space. Figure 12 Figure (d) is a comparison of the offset distances of the left and right sides of Video#2 in geographic space;

[0128] Figure 13 This is a schematic diagram of the target trajectory extraction process;

[0129] Figure 14 This is a diagram comparing the lengths of the extracted target trajectories: Figure 14 Figure (a) is a schematic diagram comparing the lengths of the target trajectory extracted from Video #1 using the TraT, TraDL, and TraEA methods. Figure 14 Figure (b) is a schematic diagram comparing the difference between the target trajectory length (TL) extracted by the three methods TraT, TraDL and TraEA and the actual length of Video#1. Figure 14 Figure (c) is a comparative diagram of the extraction of the target trajectory length of Video #2 by the three methods TraT, TraDL and TraEA; Figure 14 Figure (d) is a schematic diagram of the difference between the target trajectory length (TL) of Video#2 and the actual length extracted by the three methods TraT, TraDL and TraEA.

[0130] Figure 15 This is a diagram illustrating the continuity of the target trajectory: Figure 15 Figure (a) in the figure is a comparative diagram of the three methods TraT, TraDL and TraEA for extracting the target trajectory continuity (TC) of Video#1; Figure 15 Figure (b) in the figure is a comparative diagram of the three methods TraT, TraDL and TraEA for extracting the target trajectory continuity (TC) of Video#2;

[0131] Figure 16 This is a diagram illustrating the comparison of target trajectory offsets. Figure 16 Figure (a) in the figure is a schematic diagram comparing the target trajectory X offset extracted by the three methods for Video#1; Figure 16 Figure (b) in the figure is a schematic diagram comparing the Y offset of the target trajectory extracted from Video#1 by the three methods; Figure 16 Figure (c) in the figure is a comparative diagram of the target trajectory X offset extracted by the three methods from Video#2; Figure 16 Figure (d) in the figure is a comparative diagram of the Y offset of the target trajectory extracted by the three methods in Video#2;

[0132] Figure 17 This is a schematic diagram comparing the target trajectory error box plots. Figure 17 Figure (a) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory X offset extracted from Video#1 and the true X coordinate. Figure 17 Figure (b) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory Y offset extracted from Video#1 and the true Y coordinate. Figure 17 Figure (c) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory X offset extracted from Video#2 and the true X coordinate. Figure 17 Figure (d) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory Y offset extracted from Video#2 and the true Y coordinate. Detailed Implementation

[0133] The present invention proposes a method for extracting the complete trajectory of a moving target in a traffic scene (TraEA), which is created to address the problem that current target trajectory extraction methods do not consider target occlusion and image blurring in complex scenes, resulting in incomplete trajectory extraction of the target within the camera's field of view.

[0134] Figure 1 The diagram shown illustrates multi-layer semantic features:

[0135] 1) Low-level semantic features: Features that reflect the low-level semantics of images and videos in traffic scenes, such as geometric features, texture features, and color features.

[0136] 2) Mid-level semantic features: Building upon the extraction of low-level semantic features, this layer acquires the object's semantics and its spatial semantics. Mid-level semantic features bridge the gap between the extraction of low-level semantic features and the mining of high-level semantic features.

[0137] 3) High-level semantic features: Temporal semantics are obtained by using the orientation, topology and metric relationships between mid-level spatial semantics, laying the foundation for the extraction of the complete trajectory of the target.

[0138] Integrating multi-level semantic features from low, middle, and high layers, this invention proposes a complete trajectory extraction method based on multi-level semantic features, the process of which is as follows: Figure 2 :

[0139] (1) Coarse extraction of complete trajectory based on multi-layer semantic features:

[0140] To address the issue that existing target trajectory extraction processes fail to consider blurred target images and mutual occlusion, resulting in incomplete extraction of moving target trajectories, this paper proposes a method that, based on low-level semantic features, combines target detection-based and feature point-based target trajectory extraction algorithms to obtain mid-level semantic features such as object semantics and spatial semantics. Based on the topological and orientational relationships between these mid-level semantic features, a coarse extraction result of the complete moving target trajectory is obtained.

[0141] (2) Target trajectory optimization based on mid-level spatial semantics:

[0142] To obtain more accurate target trajectories, the coarse extraction results of complete trajectories based on multi-layer semantic features are further optimized. A transformation relationship between the target trajectory and image space and geographic space is established based on spatial relationships to obtain the offset distance of trajectories in different orientations. Based on this, the trajectories in different orientations are optimized according to the offset distance, achieving the extraction and optimization of the complete trajectory of the moving target.

[0143] This invention extracts the complete trajectory of a moving target by using low-level semantic features such as traffic scenes, video keyframes, and geometry and texture in the field of view, as well as mid-level semantic features such as objects and spatial relationships.

[0144] Target trajectory extraction based on low-level semantic features:

[0145] This technical solution combines two methods: target detection-based and feature point-based target trajectory extraction. It integrates low-level semantic features such as traffic scenes, video keyframes, and geometric and textural features in the field of view, as well as mid-level semantic features such as objects and spatial relationships. Based on target detection, it obtains the target type (Type), the target foreground image (objPic), the minimum bounding rectangle, and the coordinates of its upper left corner (x, y, y). L y L ) and the coordinates of the lower right corner (x R y R Based on the center point coordinates of the target in consecutive video frames, Obj(x) center y center ) Connect them sequentially to form the target trajectory Tra i ={(x1,y1),(x2,y2),...,(x n y n Based on feature point tracking, multiple feature corner points of a moving target are obtained, and N feature point trajectories (Traclus) of the moving target are constructed according to the coordinates of these feature corner points in consecutive video frames. i ={Traclus1,Traclus2,...,Traclus n According to Tra in the middle-level semantics i and Traclus i The orientational relationship between trajectory sequences is used to obtain the type information of the target trajectory.

[0146] The process is as follows: Based on the positional relationship between the trajectory of the moving target feature points and the smallest bounding rectangle (i.e., determining if a point is inside the polygon), obtain the type information of the target trajectory and feature point trajectory. The target type includes two main categories: vehicles and pedestrians. Pedestrians include cyclists and pedestrians. For example... Figure 3 As shown, the trajectories of five feature points A, B, C, D, and E fall into the target Tra. i If the feature points A, B, C, D, and E are within the smallest bounding rectangle of the target, then the trajectories of all five feature points belong to the target Tra. i ;Target Tra i The type information is the N feature point trajectories Traclus. i ={Traclus1,Traclus2,...,Traclus n Type information: pedestrians (including cyclists and pedestrians on foot) / vehicles.

[0147] Trajectory orientation extraction based on mid-level topological relationships:

[0148] Due to target occlusion or incomplete display of some areas during turning, the feature point trajectories are unevenly distributed on the left and right sides of the target, resulting in significant differences in the offsets between the target feature point trajectories and the detected target coordinates (X and Y). Figure 4 Figure (a) shows a schematic diagram of the trajectory clustering analysis process in traditional target trajectory extraction methods in the prior art; in order to further improve the accuracy of target trajectory extraction, trajectory Tra is obtained based on target detection. i ={(x1,y1),(x2,y2),...,(x n y n Traclus, a method for obtaining the trajectory of N feature points of a target, based on feature points. i ={Traclus1,Traclus2,...,Traclus n These mid-level semantic features; based on this, based on Tra i and Traclus i The type information of the target trajectory is obtained by the positional relationship between them, based on Tra i and Traclus i The topological relationships between N feature point trajectories are used to determine whether a sequence of N feature points is a left-hand or right-hand trajectory, laying the foundation for target trajectory clustering. Figure 4 Figure (a) shows a schematic diagram of the trajectory clustering analysis process in the target trajectory extraction method based on semantic information in this technical solution.

[0149] The process for determining whether a trajectory is on the left or right is as follows: Let Tra be the starting point of the moving target trajectory obtained based on target detection. start (x start y start The endpoint is Tra end (x end y end ) and the trajectory sequence is Tra i ={(x1,y1),(x2,y2),...,(x n y n The coordinates of the center point of the moving target trajectory extracted based on feature points are Traclus. mid (x mid y mid According to the formula (1) for calculating the vector product:

[0150] judge=(x end -x start )×(y mid -y start )-(y end -y start )×(x mid -x start )

[0151] Calculate N Traclus i Located in Tra i Left or right positional relationship: When judge < 0, Traclus i Located in Tra i To the right of Tra, one can obtain the N feature point trajectories located at Tra. i The set of trajectories on the right is: Traclus ri ={Traclus r1 Traclus r2 Traclus rn}; When judge > 0, Traclus i Located in Tra i To the left of Tra, among the N feature point trajectories, those located in Tra... i The set of trajectories on the left is: Traclus li ={Traclus l1 Traclus l2 Traclus ln};

[0152] Where i is the number of moving targets, n is the number of trajectories of a single target, rn is the number of feature point trajectories on the right side of a single target, and ln is the number of feature point trajectories on the left side of a single target; see Figure 5 The diagram shows the trajectory determination for the left and right sides.

[0153] Trajectory clustering analysis based on mid-level orientation relationships:

[0154] The orientational relationship of the moving target's trajectory can be obtained based on the topological relationships between the middle-level semantics, i.e., the left-side trajectory Traclus. li and the right track Traclus ri In order to acquire Traclus li and Traclus ri A single trajectory sequence requires Traclus ri and Traclus liCluster analysis is performed. Traditional target trajectory clustering methods (Etemad 2021; Kontopoulos 2021) mostly focus on GPS trajectories and trajectories acquired by dashcams, treating multiple trajectories as a whole to form a complex trajectory set. These sets are then divided into multiple trajectory clusters based on similarity methods, failing to accurately focus on individual target clusters. This results in low clustering accuracy and inaccurate trajectory coordinate positions. A trajectory clustering analysis based on mid-level azimuth relationships is proposed. Based on the aforementioned azimuth and topological relationships between mid-level semantics, the trajectory type information of feature points is obtained, accurately identifying the multiple feature point trajectories contained in a single moving target. Furthermore, based on the azimuth relationships between trajectory sequences, multiple trajectories are clustered to obtain a single trajectory sequence of the target. Simultaneously, based on the aforementioned Tra... i and Traclus i The topological relationships between N feature point trajectory sequences are used to determine their orientational relationships, i.e., Traclus. i Compared to Tra i It is the left-side trajectory Traclus li ={Traclus n Traclus l2 Traclus ln} or the right-hand track Traclus ri ={Traclus r1 Traclus r2 Traclus rn This is to further improve the accuracy of the target trajectory and reduce the offset values ​​of the trajectory coordinate points X and Y. Wherein:

[0155] The set of coordinate points for each left-side trajectory is:

[0156] Traclus li ={((x) l1 y l1 ), (x l2 y l2 ), ..., (x ln y ln ))}

[0157] The set of coordinate points for each right-hand trajectory is:

[0158] Traclus ri ={((x) r1 y r1 ), (x r2 y r2 ), ..., (x rn y rn ))}

[0159] In order to obtain Traclus li and Traclus ri The trajectory clustering results are processed through four main steps: trajectory average vector calculation, coordinate rotation, horizontal scanning, and rotation back to the original coordinates.

[0160] First, for the left trajectory sequence Traclus... li ={Traclus l1 Traclus l2 Traclus ln}, Traclus right-hand track sequence ri ={Traclus r1 Traclus r2 Traclus rn} and the trajectory Tra obtained from object detection i ={(x1,y1),(x2,y2),...,(x n y n The average vector is calculated to obtain the left-side trajectory sequence vector. and the vector of the trajectory sequence on the right And calculate the average vector for the trajectory vectors on the left and right sides respectively. As in formulas (2) and (3); the process is as follows: Figure 6 The diagrams showing the trajectory vector calculations and coordinate rotations on the left and right sides are shown below. Figure 6 Figure (a) in the diagram is a schematic diagram of solving the average vector on the left. Figure 6 Figure (b) in the diagram is a schematic diagram of the rotation of the trajectory coordinates on the left.

[0161]

[0162]

[0163] Where (v lxn v lyn ) is the left trajectory vector of the nth line. The coordinates, (v rxn v ryn ) is the right-hand trajectory vector of the nth line. The coordinates of (V) lx V ly () represents the trajectory vector on the left. Coordinates and (V) rx V ry () represents the trajectory vector on the right. coordinate;

[0164] Rotation of trajectory coordinates:

[0165] The average vector is calculated according to formulas (5) and (6). The angles lθ and rθ between the left trajectory sequence and the unit vector of the x-axis are used to determine the left trajectory sequence according to the average vector. Rotate the trajectory sequence on the right by an angle lθ in the direction of the average vector. Rotate the direction by an angle rθ to obtain the coordinate sets of the left and right trajectories after rotation:

[0166] xTraclus li ={((cx) l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln ))}

[0167] xTraclus ri ={((cx) r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn As shown in formulas (7), (8), (9), and (10), the process is as follows: Figure 6 As shown in Figures (c) and (d):

[0168] Figure 6 Figure (c) in the diagram is a schematic diagram of the solution process for the average vector on the right. Figure 6 Figure (d) in the diagram is a schematic diagram of the rotation of the trajectory coordinates on the right.

[0169]

[0170] cx ln =x ln coslθ-y ln sinlθ (7)

[0171] cy ln =y ln coslθ+x ln sinlθ (8)

[0172] cx rn =x rn cosrθ-y rn sinrθ (9)

[0173] cy rn =y rn cosrθ+x rn sinrθ (10)

[0174] Trajectory clustering and rotation to original coordinates:

[0175] The rotated trajectory is scanned at equal intervals along the average vector direction, and the set of n intersection points between the scan line and the left trajectory is calculated: lInter i ={(lsx1′,lsy1′),...,(lsx n ′, lsy n Given the given information, find the set rInter, which represents the n intersection points of the trajectory on the right. i ={(rsx1′, rsy1′), (rsx2′, rsy2′),..., (rsx n ′,rsy n According to formulas (11), (12), (13) and (14), the coordinate sets of the left and right trajectories after clustering are obtained:

[0176] lTra i ={(cx l1 ′, cy l1 ′), (cx l2 ′, cy l2 ′), ..., (cx ln ′, cy ln ′)}

[0177] rTra i ={(cx r1 ′, cy r1 ′), (cx r2 ′, cy r2 ′), ..., (cx rn ′, cy rn ′)}

[0178] The clustered trajectories are rotated back to their original coordinates according to formulas (15), (16), (17), and (18), finally yielding: the left trajectory cTra left ={(cx l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln )}

[0179] Right track cTra right ={(cx r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn)};

[0180] The process is as follows Figure 8 As shown in Figures (a) and (b), Figure 8 This is a schematic diagram of the trajectory clustering method. Figure 8 Figure (a) in the diagram is a schematic diagram of the trajectory clustering method on the left. Figure 8 Figure (b) in the diagram is a schematic diagram of the trajectory clustering method on the right.

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189] Target trajectory optimization based on mid-level spatial semantics:

[0190] To obtain a high-precision complete trajectory of a moving target, it is necessary to further mine the temporal and spatial semantics of the moving target contained in the mid-level semantic features. This technical solution, based on the extraction of the complete trajectory of the moving target, establishes the coordinate transformation relationship between the moving target trajectory in image space and geographic space based on spatial mapping to mine mid-level spatial semantic features; and obtains the offset distance of trajectories in different orientations based on the metric relationship between spatial semantics, thereby further optimizing the coarse extraction results of the complete trajectory of the moving target.

[0191] 1. Trajectory coordinate transformation based on spatial mapping:

[0192] Based on the aforementioned, trajectory clustering results for each target from different orientations can be obtained. Since there is a certain distance deviation between the trajectories from different orientations and the detected target trajectory, it is necessary to move the left and right trajectories a certain distance according to the direction of the target trajectory. However, targets exhibit a "nearer, larger; farther, smaller" characteristic in image space, and target trajectories exhibit a "nearer, more clustered; farther, more dispersed" characteristic. Therefore, it is necessary to establish a transformation relationship between the target trajectory's coordinates in image space and geographic space, and optimize the target trajectory in geographic space. This technical solution uses the homography method to solve the homography matrix H and proposes a trajectory coordinate transformation method based on spatial mapping: First, combining the corresponding high-definition remote sensing online image, select the iconic points q{q1(x1, y1), q2(x2, y2), ..., q...} in the video frame image space. n (x n y n The selection criteria for landmark points are as follows: the target point must be obvious, clear, and easy to read on the remote sensing image, and have a corresponding position in the video frame; secondly, based on the high-precision remote sensing image map, corresponding points with the same name Q{Q1(X1, Y1), Q2(X2, Y2), ..., Q} are obtained in the geographic space. n (X n Y n Finally, the homography matrix H of the camera-to-geospatial mapping and the inverse matrix H of the transformation from geographic coordinates to image coordinates are solved. -1 This lays the foundation for optimizing the target trajectory, as shown in formulas (19) and (20).

[0193]

[0194]

[0195] Among them (X) n Y n (x) represents the coordinates of any point on the remote sensing image. n y n ) is the video frame with (X) n Y n The corresponding coordinates, H -1 It is the 3×3 inverse matrix obtained from the homography matrix H.

[0196] 2. Trajectory optimization based on mid-level metric relationships:

[0197] Existing methods only optimize trajectories in the image space, neglecting the geographic location information of the target trajectory. This leads to complex and low-accuracy target trajectory optimization methods. Therefore, this paper proposes a trajectory optimization method based on mid-level metric relationships. The method obtains the offset distance based on the metric relationships between mid-level spatial semantic features, and optimizes the coarse extraction result of the target trajectory based on the offset distance to obtain a high-precision complete trajectory of the moving target. The homography matrix H and inverse matrix H' are calculated as described above. -1 Establish the coordinate transformation relationship between the target trajectory in geographic space and image space, and solve the cTra in the middle-level semantics. left ={(cx l1 cy l1 ), (cx l2 cy l2 ), ..., (cx ln cy ln )} and cTra right ={(cx r1 cy r1 ), (cx r2 cy r2 ), ..., (cx rn cy rn The geographic coordinates of )} are the left-side trajectory gTra left ={(cX l1 ,cY l1 ), (cX l2 ,cY l2 ), ..., (cX ln ,cY ln )}、Right-side trajectory gTra right ={(cX r1 ,cY r1 ), (cX r2 ,cY r2 ), ..., (cX rn ,cY rn The set of geographic coordinates of the target detection trajectory, gTra i ={(X1, Y1), (X2, Y2),..., (X n Y n )},like Figure 9 As shown: Figure 9 (a) shows schematic diagrams of trajectory clustering analysis on the left and right sides in the image space; Figure 9 (b) shows the clustering and mapping relationships on the left and right sides of the geospatial map. The lowercase x and y symbols include (cx... rn cy rn ) and (cx ln cy lnAll are coordinates on the image space video frame; uppercase X and Y markers, including (cX) ln ,cY ln ) and (cX rn ,cY rn All coordinates are converted from image space to geographic space.

[0198] The left trajectory gTra is calculated according to formulas (21) and (22). left Relative to the target detection trajectory gTra i offset distance d rn Right-side trajectory gTra right Relative to the target detection trajectory gTra i offset distance d ln .

[0199]

[0200]

[0201] And according to d rn d ln And the included angle is calculated using formulas (23) and (24). Further, the coordinates (X) of each point on the left trajectory after movement are calculated using formulas (25), (26), (27), and (28). ln Y ln ) and the coordinates (X) of each point on the right trajectory after it has moved. rn Y rn The purpose of the aforementioned movement is to optimize the trajectory results and obtain a more accurate complete trajectory of the moving target.

[0202]

[0203]

[0204]

[0205]

[0206]

[0207]

[0208] The final trajectory coordinate sequence after trajectory optimization is obtained:

[0209] Left trajectory sequence:

[0210] yTra left ={(cX l1 +X l1 ,cYl1 +Y l1 ), (cX l2 +X l2 ,cY l2 +Y l2 ), ..., (cX ln +X ln ,cY ln +Y ln )}

[0211] Right-hand trajectory sequence:

[0212] yTra right ={(cX r1 -X r1 ,cY r1 -Y r1 ), (cX r2 -X r2 ,cY r2 -Y r2 ), ..., (cX rn -X rn ,cY rn -Y rn )}

[0213] According to the inverse matrix H of the homography matrix -1 For yTra left andyTra right The transformed left-side trajectory sequence is obtained by converting geographic coordinates to image coordinates:

[0214] pTra left ={(px l1 py l1 ), (px l2 py l2 ), ..., (px ln py ln And the transformed right-hand trajectory sequence: pTra right ={(px r1 py r1 ), (px r2 py r2 ), ..., (px rn py rn )}.

[0215] pTra was clustered using the aforementioned methods. left pTra right and Tra i Clustering is performed to obtain optimized trajectory results, such as... Figure 10 As shown in Figures (a) and (b), Figure 10 The diagram shown is a schematic of trajectory optimization. Figure 10 Figure (a) in the diagram is a schematic diagram of trajectory offset distance calculation. Figure 10 Figure (b) in the diagram is a schematic diagram of the trajectory movement coordinate calculation;

[0216] The software environment used in the embodiments of this technical solution is PyTorch, VS C#, Emgu CV, Arcengine, and the hardware environment is GPU G00TX-1660Ti, CPU i7-10750H, and 16G of memory. Taking Video#1 and Video#2 of a certain highway in Beijing as examples, relevant experiments were carried out. The similarity between Video#1 and Video#2 is that both videos are complex traffic scenes. The differences are (1) the orientation of the camera field of view of Video#1 and Video#2 is different; (2) the driving direction of the target in Video#1 and Video#2 is different. The distribution of Video#1 and Video#2 on the map is as follows. Figure 12 The experimental scenario diagram is shown below; relevant information is shown in Table 1.

[0217] Table 1 Video Information

[0218]

[0219] To verify the accuracy and performance of the method, the true trajectory (TR) of the target was manually extracted from the original video. Specifically, when the target was close to the camera, the trajectory extracted by target detection was used as the true trajectory (TR). When the target was far from the camera, the target trajectory could not be obtained by target detection, so the position of the target in each frame was manually marked, and together with the target detection trajectory, they constituted the true trajectory (TR) of the target. To ensure the accuracy of the true trajectory (TR) extraction, the center point of the minimum bounding rectangle of each frame was used as the trajectory coordinate. The mean square error (MSE), root mean square error (RMSE), trajectory length (TL), and trajectory continuity (TC) were used as evaluation indicators. The smaller the MSE and RMSE, and the larger the TL and TC, the higher the target accuracy (Chen 2020). As shown in formulas (29), (30), (31), (32), (33), and (34).

[0220]

[0221]

[0222]

[0223]

[0224]

[0225]

[0226] Where xMSE and xRMSE are the mean square error and root mean square error of the target trajectory coordinate X, yMSE and yRMSE are the mean square error and root mean square error of the target trajectory coordinate Y, n is the number of trajectory points contained in the target, and T x and T y The X and Y coordinates extracted for the target trajectory, TR x and TR y The true trajectory (TR) coordinates are X and Y; (x start y start (x) is the starting point of the target trajectory. end y end ) is the endpoint of the target trajectory, and N is the number of interruptions to the target trajectory.

[0227] Target trajectory extraction:

[0228] Target trajectories were extracted from two videos, Video#1 and Video#2. Taking the most common vehicle trajectory extraction as an example, the target trajectory and trajectory semantic information were extracted every 100 frames. Following the aforementioned methods, trajectory average vector generation, coordinate rotation, trajectory clustering, and rotation to the original coordinates were performed. Taking frames 300, 400, and 500 of Video#1 and Video#2 as examples, the trajectory extraction and optimization results were obtained, as follows: Figure 8 The diagram shows the target trajectory extraction results.

[0229] To optimize the trajectory, the target optimization distance needs to be calculated as described above. Taking frames 300, 400, 500, 600, and 700 of Video#1 and Video#2 as examples, the optimized trajectory distances for the left and right sides of different targets are calculated. Experimental results show that under different experimental environments and scenes, the offset distance varies significantly at different positions in the image space. Figure 12 The left and right offset distance comparison charts shown are (a) and (b). Figure 12 Figure (a) shows a comparison of the offset distances on the left and right sides of Video#1 in the image space. Figure 12 Figure (b) is a comparison of the offset distances of the left and right sides of Video#2 in the image space.

[0230] Figure 12 Figure (c) is a comparison of the offset distances of the left and right sides of Video#1 in geographic space. Figure 12Figure (d) shows a comparison of the offset distances of the left and right sides of Video#2 in geographic space. The offset distances of the same target on the right and left sides are basically the same in geographic space, with differences varying within a certain range at different locations. Therefore, a specific offset distance can be used to optimize the target, reducing the complexity of the target trajectory extraction method. Simultaneously, based on the aforementioned target trajectory optimization method, the target trajectory is moved according to the optimized distance, and the target trajectory points are converted from image space to geographic space. Each trajectory point is optimized according to its corresponding offset distance, ultimately obtaining the optimized left and right trajectories of the target. Taking a frame from Video#1 as an example, the process is as follows... Figure 13 The target trajectory extraction process is shown.

[0231] Trajectory extraction completeness verification:

[0232] Select any target in a video frame, calculate the target trajectory length (TL) extracted by the image feature-based target trajectory extraction TraT (Cariou 2022; Wang 2022; Sugirtha 2022; Babu 2022), the target trajectory extraction based on target detection (TraDL) (Praveenkumar 2022; Wang 2022), and the TraEA method of this technical solution, and compare the length (TL) extracted by the three methods.

[0233] Figure 14 This is a diagram comparing the lengths of the extracted target trajectories: Figure 14 Figure (a) is a schematic diagram comparing the lengths of the target trajectory extracted from Video #1 using the TraT, TraDL, and TraEA methods. Figure 14 Figure (b) is a schematic diagram comparing the difference between the target trajectory length (TL) extracted by the three methods TraT, TraDL and TraEA and the actual length of Video#1. Figure 14 Figure (c) is a comparative diagram of the extraction of the target trajectory length of Video #2 by the three methods TraT, TraDL and TraEA; Figure 14 Figure (d) is a schematic diagram comparing the difference between the target trajectory length (TL) extracted by the three methods TraT, TraDL and TraEA and the actual length of Video#2.

[0234] See Figure 14Figures (a) and (c) are shown. A comparison of the trajectory lengths (TL) of 24 different targets extracted by the TraEA, TraDL, and TraT methods for Videos #1 and #2 shows that the target trajectory lengths (TL) extracted by the TraEA method are all greater than those extracted by the TraDL and TraT methods. Furthermore, the errors between the three methods and the actual trajectory lengths are calculated, where the actual target trajectories (TR) are obtained through a combination of manual annotation and target detection. See [link to relevant documentation]. Figure 15 Figure (b) shows that for Video#1, the TraEA method improves the target trajectory length (TL) by 76% compared to the TraDL method and by 82.19% compared to the TraT method. Figure 15 As shown in Figure (d), for Video#2, the TraEA method improves the target trajectory length (TL) extracted by 69.9% compared to the TraDL method and by 80.05% compared to the TraT method.

[0235] The integrity of the target trajectory needs to be evaluated using two metrics: trajectory length (TL) and trajectory continuity (TC). For Video#1 and Video#2, the trajectories of 24 different targets were extracted by image feature-based target trajectory extraction (TraT), target trajectory extraction based on target detection (TraDL), and the TraEA method proposed in this paper, and the target continuity (TC) was compared.

[0236] Figure 15 This is a diagram illustrating the continuity of the target trajectory: Figure 15 Figure (a) in the figure is a comparative diagram of the three methods TraT, TraDL and TraEA for extracting the target trajectory continuity (TC) of Video#1; Figure 15 Figure (b) in the figure is a comparative diagram of the three methods TraT, TraDL and TraEA for extracting the target trajectory continuity (TC) of Video#2.

[0237] like Figure 15 It can be seen that the TraEA method extracts a greater target trajectory continuity (TC) than both the TraDL and TraT methods. Specifically, for video #1, the TraEA method increases the trajectory continuity (TC) by 60.65% compared to TraDL, while the TraT method increases it by 77% compared to TraT. Figure 16 As shown in (a), for Video#2, the TraT method improved trajectory continuity (TC) by 59.71% compared to TraDL, while the TraT method improved trajectory continuity (TC) by 80.35% compared to TraT. Figure 15As shown in Figure (b).

[0238] Trajectory extraction accuracy verification:

[0239] To further verify the accuracy of the target trajectory, any target in the video was selected, and the coordinate offsets of the target trajectory extracted by the image feature-based target trajectory extraction (TraT), the target trajectory extraction based on target detection (TraDL), and the target trajectory extracted by the TraEA method in this paper were compared with the coordinate offsets of the real target trajectory (TR). The target trajectory points are connected sequentially to form a curve segment, and the coordinate offsets of each trajectory can be compared and analyzed.

[0240] Figure 16 The image shows a comparison of target trajectory offsets. Figure 16 (a) The figure shows a comparison of the target trajectory X offset extracted by three methods from Video#1; Figure 16 Figure (b) is a schematic diagram comparing the Y offset of the target trajectory extracted from Video#1 using the three methods; Figure 16 Figure (c) is a schematic diagram comparing the target trajectory X offset extracted by the three methods for Video#2; Figure 16 Figure (d) is a comparative diagram of the Y-offset of the target trajectory extracted from Video#2 using three methods. The "three methods" are TraT, TraDL, and TraEA.

[0241] Depend on Figure 16 It can be seen that the target trajectory extracted by the TraEA method is closest to the true target trajectory (TR), meaning that the TraEA method has the highest accuracy in extracting the target trajectory. Furthermore, the average offsets of the X and Y values ​​of the trajectory coordinates relative to the true coordinates for Video#1 and Video#2 using the TraT, TraDL, and TraEA methods were calculated. For Video#1, the average X and Y offsets for the TraT, TraDL, and TraEA methods were 2.493%, 4.037%, and 1.21%, and 1.995%, 0.38%, and 0.684%, respectively. The TraEA method improved the accuracy of the X and Y offsets by 2.112% and 3.352%, and 0.829% and 1.311%, respectively, compared to the TraT and TraDL methods.

[0242] See Figure 17 A comparative diagram of target trajectory error box plots; Figure 17 Figure (a) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory X offset extracted from Video#1 and the true X coordinate. Figure 17 Figure (b) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory Y offset extracted from Video#1 and the true Y coordinate. Figure 17 Figure (c) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory X offset extracted from Video#2 and the true X coordinate. Figure 17 Figure (d) is a comparative diagram of the error box plots obtained by the three methods for the difference between the target trajectory Y offset extracted from Video#2 and the true Y coordinate.

[0243] For Video#2, the average X and Y offsets of the TraT, TraDL, and TraEA methods were 2.61% and 4.21%, 1.641% and 2.558%, and 0.501% and 0.974%, respectively. The TraEA method improved the accuracy of X and Y offsets by 2.109% and 3.236%, and 1.139% and 1.583% compared to the TraT and TraDL methods, respectively.

[0244] Simultaneously, target trajectories of different targets (Object#1, Object#2, and Object#3) in Video#1 and Video#2 were extracted. Starting from 30 frames and ending at 300 frames, the target trajectory was recorded every 30 frames, and the mean square error (MSE) and root mean square error (RMSE) of the X and Y coordinate sequences of the trajectory were calculated and compared with the target trajectories extracted by TraT and TraDL methods. The results are shown in Tables 3 and 4. It can be seen that the xMSE, xRMSE, yMSE, and yRMSE of TraEA are all smaller than those of TraT and TraD, indicating that the target trajectory extracted by the TraEA method has higher accuracy. Specifically, for Video#1 and Video#2, TraEA reduced xMSE, yMSE, xRMSE, and yRMSE by 65.43%-81.62%, 53.85%-89.27%, 63.85%-85.53%, and 53.93%-87.6% respectively compared to TraT and TraD methods, demonstrating the high accuracy of the TraEA method.

[0245] Table 3 Comparison of MSE and RMSE of the target trajectory in Video #1

[0246]

[0247] Table 4 Comparison of MSE and RMSE of the target trajectory in Video #2

[0248]

[0249] In summary, this technical solution addresses the problem of incomplete target trajectory extraction within the field of view caused by factors such as mutual occlusion of targets and low spatial resolution of target images in traffic scenarios. Taking into account the spatiotemporal semantic relationship of geospatial data, a complete target trajectory extraction method (TraEA) for traffic scenarios is proposed. This method extracts multi-granular spatiotemporal semantic features from the video to achieve the extraction of complete target trajectories in traffic scenarios. Test results show that: (1) In terms of the completeness of target trajectory extraction, compared with the two existing methods, namely target trajectory extraction based on target detection (TraT) and target trajectory extraction based on image features (TraD), the target trajectory length (TL) extracted by the TraEA method of this invention is improved by 69.9%-76% and 80.05%-82.19%, respectively, and the target trajectory continuity (TC) is improved by 59.71%-60.65% and 77%-80.35%, respectively, reflecting the completeness of target trajectory extraction by the TraEA method. (2) In terms of target trajectory extraction accuracy, TraEA reduced xMSE, yMSE, xRMSE and yRMSE by 65.43%-81.62%, 53.85%-89.27%, 63.85%-85.53% and 53.93%-87.6% respectively compared with TraT and TraD methods, indicating the high accuracy of TraEA method.

[0250] This invention, in the target complete trajectory extraction stage, integrates low-level semantic features such as scene, keyframes, and view area, as well as mid-level temporal and spatial semantics, to design a method for extracting the complete trajectory of moving targets based on multi-layer semantic features. Building upon this, a trajectory optimization method is proposed based on the orientation, topology, and metric relationships of mid-level spatial semantics within the scene, achieving high-precision complete trajectory optimization for moving targets within the view area. This invention aims to provide a new perspective for target trajectory extraction, thereby improving the accuracy and efficiency of target tracking, and providing technical support for the integration of video and GIS.

[0251] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting the complete trajectory of a moving target in a traffic scene, characterized in that: Includes the following steps: a) Access video data from complex traffic scenarios, perform target detection-based tracking of moving targets in the video, and obtain the target type (Type), target foreground image (objPic), and target center point coordinates (Obj(x)). center ,y center The smallest bounding rectangle and the coordinates of its top-left corner (x) L ,y L ) and the coordinates of the lower right corner (x R ,y R Based on the center coordinates of the target in consecutive video frames, Obj(x) center ,y center ) Connect them sequentially to form the target trajectory Tra i ={(x1,y1),(x2,y2),…,(x n ,y n )}; Based on feature point tracking, multiple feature corner points of a moving target are obtained. A set of N feature point trajectories, Traclus, is constructed from the coordinates of these feature corner points across consecutive video frames. i ={Traclus1,Traclus2,…,Traclus n }; Based on the Traclus set of feature points of the moving target i The orientational relationship between the target trajectory and the minimum bounding rectangle, i.e., the determination that a point is inside the polygon, is used to obtain the type information of each trajectory in the target trajectory and feature point trajectory set. The target type includes two main categories: vehicles and pedestrians. Pedestrians include cyclists and pedestrians. If the five feature point trajectories A, B, C, D, and E fall within the minimum bounding rectangle of the moving target, then all five feature point trajectories A, B, C, D, and E belong to the moving target. The type information of the moving target is the set of N feature point trajectories Traclus. i ={Traclus1,Traclus2,…,Traclus n Type information: pedestrians (cyclists and pedestrians) / vehicles; b) Based on step a), the target trajectory Trai = {(x1,y1),(x2,y2),…,(x...} can be obtained. n ,y n )} and the set of N feature point trajectories Traclus i ={Traclus1,Traclus2,…,Traclus n Based on this, through the feature point trajectory set Traclus i Each trajectory in the target trajectory Tra i The vector product `judge` is used to obtain the relative vector product of each trajectory in the feature point trajectory set with respect to the target trajectory `Tra`. i Positional relationship, let the target trajectory Tra i The starting point is Tra start (x start ,y start The endpoint is Tra end (x end ,y end Traclus, a set of feature point trajectories i The coordinates of the center point of any trajectory in Traclus are... mid (x mid ,y mid According to the formula for calculating the vector product: judge=(x end -x start )×(y mid -y start )-(y end -y start )×(x mid -x start ) Calculate N Traclus i Located in Tra i The positional relationship between the left and right sides, i.e., when judge < 0, Traclus i Located in Tra i On the right side, one can obtain the N feature point trajectories located in Tra. i The set of trajectories on the right is: Traclus ri ={Traclus r1 ,Traclus r2 ,…,Traclus rn }; When judge > 0, Traclus i Located in Tra i To the left of Tra, the N feature point trajectories are located in Tra. i The set of trajectories on the left is: Traclus li ={Traclus l1 ,Traclus l2 ,…,Traclus ln }; Where i is the number of moving targets, n is the number of trajectories of a single target, rn is the number of feature point trajectory sets to the right of a single target, and ln is the number of feature point trajectory sets to the left of a single target; c) For the left-hand trajectory set Traclus in step b) li and the right-hand track set Traclus ri Perform cluster analysis, let: The set of coordinate points for any trajectory in the left-hand trajectory set is: Traclus ln ={((x l1 ,and l1 ),(x l2 ,and l2 ),…,(x ln ,and ln ))} The set of coordinate points for any trajectory in the right-hand trajectory set is: Traclus ln ={((x r1 ,and r1 ),(x r2 ,and r2 ),…,(x rn ,and rn ))}; Cluster analysis mainly includes four steps: trajectory average vector calculation, coordinate rotation, horizontal scan, and rotation back to the original coordinates. Average vector calculation: Get: Traclus, the set of trajectories on the left li ={Traclus l1 Traclus l2 ,…,Traclus ln }, Left-side trajectory set vector and the right-hand track set Traclus ri ={Traclus r1 Traclus r2 ,…,Traclus rn }, Right-side trajectory set vector And respectively for the left trajectory set vector and the right-hand trajectory set vector Find the average vector: Using formulas respectively Where |lV| and |rV| are vectors and The model, (v lxn ,v lyn ) is the left trajectory vector of the nth line. The coordinates, (v rxn ,v ryn ) is the right-hand trajectory vector of the nth line. The coordinates of (V) lx V ly ) is the vector of the left-hand trajectory set. Coordinates and (V) rx V ry ) is the vector of the left-hand trajectory set. coordinate; Rotation of trajectory coordinates: according to the formula and Calculate the average vector The angle lθ between the vector and the x-axis unit vector and the average vector The angle rθ between the left-hand trajectory set Traclus and the unit vector of the x-axis li According to the average vector Rotate the direction by an angle lθ, and then rotate the right-hand track set Traclus. ri According to the average vector Rotate the direction by an angle rθ to obtain: Any coordinate point (x) on the left trajectory ln ,y ln The rotated X-coordinate cx ln =x ln coslθ-y ln sinlθ Any coordinate point (x) on the left trajectory ln ,y ln The rotated Y-coordinate cy ln =y ln coslθ+x ln sinlθ Any coordinate point (x) on the right-hand trajectory rn ,y rn The rotated X-coordinate cx rn =x rn cosrθ+y rn sinrθ Any coordinate point (x) on the right-hand trajectory rn ,y rn The rotated Y-coordinate cy rn =y rn cosrθ-x rn sinrθ Rotate each coordinate point on the trajectory according to the above steps to obtain the set of coordinates on the left side of the trajectory: xTraclus li ={((cx l1 ,cy l1 ),(cx l2 ,cy l2 ),…,(cx ln ,xy ln ))} The set of coordinates of the trajectory on the right is obtained as follows: xTraclus ri ={((cx r1 ,cy r1 ),(cx r2 ,cy r2 ),…,(cx rn ,cy rn ))}; Horizontal scan: for the rotated left-side trajectory coordinate set xTraclus li The right-side trajectory coordinate set xTraclus ri Calculate by scanning at equal intervals along the average vector direction: The set of n intersection points between the scan line and the left trajectory: lInter i ={(lsx1',lsy1'),(lsx2',lsy2'),…,(lsx n ',lsy n ')} The set of n intersection points between the scan line and the right-side trajectory: rInter i ={(rsx1',rsy1'),(rsx2',rsy2'),…,(rsx n ',rsy n ')}, Calculated using the above formula: The X-value of any coordinate of the left trajectory after clustering: The Y-value of any coordinate on the left side of the clustered trajectory: The X-value of any coordinate of the right-hand trajectory after clustering: The Y-value of any coordinate of the right-hand trajectory after clustering: The coordinate sets of the left and right trajectories after cluster analysis are obtained: The coordinate set of the left trajectory lTra i ={(cx l1 ',cy l1 '),(cx l2 ',cy l2 '),…,(cx ln ',cy ln ')} The coordinate set rTra of the right-hand trajectory i ={(cx r1 ',cy r1 '),(cx r2 ',cy r2 '),…,(cx rn ',cy rn ')}; Then rotate the clustered left and right trajectories back to their original coordinates: The X-value of any coordinate on the left side of the rotated trajectory: The Y-value of any coordinate on the left side of the rotated trajectory: The X-value of any coordinate on the right-hand trajectory after rotation: The Y-value of any coordinate on the right-hand trajectory after rotation: Finally, the rotated left-side trajectory cTra is obtained. left and the right-side trajectory cTra right : Left track cTra left ={(cx l1 ,cy l1 ),(cx l2 ,cy l2 ),…,(cx ln ,cy ln } Right track cTra right ={(cx r1 ,cy r1 ),(cx r2 ,cy r2 ),…,(cx rn ,cy rn )}; d) Combining the corresponding high-definition remote sensing online imagery, select the iconic points q{q1(x1,y1),q2(x2,y2),…,q in the video frame image space. n (x n ,y n The selection criteria for landmark points are as follows: the target point must be obvious, clear, and easy to read on the remote sensing image, and have a corresponding position in the video frame; secondly, based on the high-precision remote sensing image map, corresponding points with the same name at the geographic location are obtained: Q{Q1(X1,Y1),Q2(X2,Y2),…,Q n (X n ,Y n Solve for the homography matrix H of the camera, and according to... The inverse matrix H of the homography matrix is ​​calculated. -1 , where (X) n , Y (n) represents the coordinates of any point on the remote sensing image, (x) n ,y n ) is the video frame with (X) n ,Y n The corresponding coordinates; e) To further optimize the left-side trajectory cTra left and the right-side trajectory cTra right After obtaining the high-precision complete trajectory, calculate the homography matrix H and the inverse matrix H' according to step d). -1 Establish the coordinate transformation relationship between the target trajectory in geographic space and image space: According to the formula Solve cTra left ={(cx l1 ,cy l1 ),(cx l2 ,cy l2 ),…,(cx ln ,cy ln )} and cTra right ={(cx r1 ,cy r1 ),(cx r2 ,cy r2 ),…,(cx rn ,cy rn The geographical coordinates of )}; That is, the geographical left-side trajectory gTra left ={(cX l1 ,cY l1 ),(cX l2 ,cY l2 ),…,(cX ln ,cY rn )} and the geographical right-side trajectory gTra right ={(cX r1 ,cY r1 ),(cX r2 ,cY r2 ),…,(cX rn ,cY rn )} and the target detection trajectory Tra in step a) i ={(x1,y1),(x2,y2),…,(x n ,y n The set of geographic coordinates gTra i ={(X1,Y1),(X2,Y2),…,(X n ,Y n )}; And according to the formula as well as Calculate the geographic left-side trajectory gTra left relative to the geographic target trajectory gTra i offset distance d ln And calculate with offset distance d ln For the hypotenuse and gTra left The X and Y coordinates of any point on the triangle are the included angle between the two legs of the right triangle. Calculate the geographic right-side trajectory gTra right relative to the geographic target trajectory gTra i offset distance d rn And calculate with offset distance d rn For the hypotenuse and gTra right The X and Y coordinates of any point on the triangle are the included angle between the two legs of the right triangle. in Using gTra left Formula for calculating the slope of the tangent line Obtain Using gTrg right Formula for calculating the slope of the tangent line Get the lowercase x and y flags, including (cx rn ,cy rn ) and (cx ln ,cy ln All are coordinates in image space; uppercase X and Y are used as markers, including (cX). ln ,cY ln ) and (cX rn ,cY rn All coordinates are transformed from image space to geographic space. f) d calculated based on step e) rn d ln and included angle To obtain a more accurate trajectory, the geographic left-side trajectory gTra needs to be... left and the geography right-side trajectory gTra right According to the offset distance d ln and d rn Move; Get the left-side geographic trajectory gTra left Any coordinate point (cX) ln ,cY ln ), calculate with hypotenuse d ln The included angle is The right-angled side X of the triangle ln Y ln That is (cX) ln ,cY ln ) trajectory to geographic target gTra i The distance traveled, where: cX ln gTra trajectory to geographic target i Distance moved X ln The calculation formula is: cY ln gTra trajectory to geographic target i Distance Y traveled ln The calculation formula is: Get the right-side geographic trajectory gTra right Any coordinate point (cX) rn ,cY rn ), calculate with hypotenuse d rn The included angle is The right-angled side X of the triangle rn Y rn That is (cX) rn ,cY rn ) trajectory to geographic target gTra i The distance traveled, where: cX rn gTra trajectory to geographic target i Distance moved X rn The calculation formula is: cY rn gTra trajectory to geographic target i Distance Y traveled ln The calculation formula is: We obtain the coordinates (X) of each point on the left side of the geographic trajectory after its movement. rn Y rn ) and the coordinates (X) of each point on the right side of the geographic trajectory after its movement. ln Y ln ); Then, obtain the trajectory coordinate sequence after trajectory optimization, i.e., the left trajectory optimization sequence. yTra left ={(cX l1 +X l1 ,cY l1 +Y l1 ),(cX l2 +X l2 ,cY l2 +Y l2 ),…,(cX ln +X ln ,cY ln +Y ln )} And the right-side trajectory optimization sequence yTra right ={(cX r1 -X r1 ,cY r1 -Y r1 ),(cX r2 -X r2 ,cY r2 -Y r2 ),…,(cX rn -X rn ,cY rn -Y rn )} Finally, the inverse matrix H of the homography matrix calculated according to step d) -1 Then for yTra left andyTra right The result of performing a geographic coordinate to image coordinate transformation is: The transformed left-side trajectory sequence: pTra left ={(px l1 ,py l1 ),(px l2 ,py l2 ),…,(px ln ,py ln )} The transformed right-hand trajectory sequence: pTra right ={(px r1 ,py r1 ),(px r2 ,py r2 ),…,(px rn ,py rn )} For pTra left pTra right and Tra i Calculate the average vector to obtain the vectors of the three trajectories: and And according to the formula Calculate the average vector of the three trajectories. Then follow the formula Calculate the average vector The angle θ between pTra and the unit vector of the x-axis left pTra right and Tra i Sequence according to average vector Rotate the direction by an angle θ, according to the formula tx ln =px ln cosθ-py ln sinθ and ty ln =py ln cosθ+px ln sinθ yields the rotated left-side coordinate set tTra left ={(tx l1 ,ty l1 ),(tx l2 ,ty l2 ),…,(tx ln ,ty ln According to the formula tx rn =px rn cosθ-py rn sinθ and ty rn =py rn cosθ+px rn sinθ yields the rotated left-side coordinate set tTra right ={(tx r1 ,ty r1 ),(tx r2 ,ty r2 ),…,(tx rn ,ty rn )}; Then, the rotated trajectory is scanned at equal intervals according to the average vector direction, and the set Inter of the three intersection points of the scan line and the three trajectories is obtained. i ={(sx1',sy1'0,(sx2',sy2'),(sx3',sy3')}, and according to the formula: The coordinate set tTra of the clustered trajectory is obtained. i ={(tx1',ty1'),(tx2',ty2'),…,(tx n ',ty n ')}; Finally, according to the formula tx n =tx' n ×cos(π+θ)-ty n '×sin(π+θ) and ty n =ty' n ×sin(π+θ)+ty n The function '×cos(π+θ)' rotates the clustered trajectories back to their original coordinates, resulting in the optimized trajectory coordinate set tTra. i ={(tx1,ty1),(tx2,ty2),…,(tx n ,ty n )}.

Images