A Visual Intelligence-Based Multi-Target Tracking Method for Intelligent Driving
By using semantic reasoning of road scenes and an improved DINO-DETR visual detection model, the problems of trajectory interruption and identity confusion in multi-target tracking under complex road scenes are solved, and the stability and continuity of traffic target detection are achieved.
Patent Information
- Application Number
- CN202611268535.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-target tracking technologies are prone to trajectory interruption and identity confusion in complex road scenarios due to occlusion, missed detection, and target intersection. They also lack semantic constraints for road scenarios, resulting in unstable traffic target detection.
A visual intelligence-based multi-target tracking method is adopted. The lane extension direction, drivable space range and traffic target association boundary are determined by semantic reasoning of road scene. The improved DINO-DETR visual detection model is combined to perform sparse query localization, construct occlusion phase signature and perform target inertial potential field inversion to generate continuous trajectory of traffic target.
It improves the scene adaptability and trajectory stability of traffic target detection, reduces trajectory breakpoints caused by occlusion and missed detection, reduces the probability of number jumps and identity confusion, and achieves better multi-target tracking results.
Smart Images

Figure CN123118692A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual perception for intelligent driving, and more particularly to a multi-target tracking method for intelligent driving based on visual intelligence. Background Technology
[0002] With the increasing application of intelligent driving vehicles in urban roads, expressways, intersections, and mixed traffic scenarios, vehicles need to continuously acquire continuous road image frames using onboard cameras and perform real-time perception of surrounding vehicles, pedestrians, non-motorized vehicles, and road obstacles. Existing multi-object tracking technologies typically first preprocess the continuous road image frames to obtain road image data suitable for the input of the detection model. Then, a visual detection model is used to detect traffic objects in each frame of the road image, obtaining the traffic object bounding box, traffic object category, and traffic object confidence score. Finally, based on the positional distance, appearance similarity, or scale variation relationship of the traffic object bounding boxes between adjacent frames, the traffic objects are assigned identities to form continuous trajectories. This type of technology can complete basic tracking tasks when traffic targets are unobstructed, road structures are clear, and traffic target movements are stable. However, in real-world intelligent driving scenarios, continuous road image frames are often affected by vehicle occlusion, pedestrian crossings, parallel non-motorized vehicles, changes in road boundaries, changes in lane extension directions, and rapid changes in target scale. Simply relying on target bounding boxes or appearance features for adjacent frame matching can easily lead to traffic target number jumps, trajectory interruptions after short-term missed detections, and identity confusion after occlusion is removed.
[0003] Current technologies for road scene understanding typically treat road images as ordinary image processing objects, directly performing traffic target detection without fully utilizing lane extension direction, drivable space range, and traffic target association boundaries to constrain the detection process. From the forward-looking perspective of an autonomous vehicle, traffic targets usually exhibit scale variations from far to near along the lane extension direction, and their locating areas have a strong correspondence with the drivable space range, road boundaries, and lane orientation. Especially at intersections, curved road sections, and densely trafficked areas, changes in road structure complicate the spatial relationships between traffic target bounding boxes, making detection results lacking a semantic constraint set for the road scene insufficient to reliably support subsequent tracking. Summary of the Invention
[0004] One objective of this invention is to propose a multi-target tracking method for intelligent driving based on visual intelligence. This invention fully utilizes computer vision, road scene semantic reasoning, an improved DINO-DETR visual detection model, occlusion phase signature, and target inertial potential field inversion. This invention can improve the continuity of traffic target detection and tracking in complex road scenes, reduce trajectory breakpoints caused by occlusion, missed detection, and target intersection, and improve identity consistency, road scene adaptability, and trajectory stability in the multi-target tracking process.
[0005] The multi-target tracking method for intelligent driving based on visual intelligence according to embodiments of the present invention includes the following steps: Collect and preprocess continuous road image frames during the driving process of intelligent driving vehicles to generate standardized image sequences of road scenes; Based on standardized image sequences of road scenes, road skeleton semantic reasoning guided by vanishing points is performed to determine lane extension direction, drivable space range and traffic target association boundary, and generate a set of semantic constraints for road scenes. The semantic constraint set of the road scene is input into the improved DINO-DETR visual detection model to perform sparse target query and localization, and a set of traffic target detection results is generated. Based on the traffic target detection result set, an occlusion phase signature of the traffic target is constructed and fused with the target appearance and target scale of the traffic target to generate a traffic target tracking feature set; Using the traffic target tracking feature set as the motion observation source, the motion attitude parameters of the traffic target in consecutive frames are determined, and the target inertial potential field inversion is performed on the traffic target to generate a set of traffic target motion states; Based on the set of traffic target motion states, a cross-frame association cost matrix of traffic targets between adjacent frames is constructed, and identity consistency matching of traffic targets is performed to generate an initial trajectory set of traffic targets. Based on the initial set of traffic target trajectories, perform residual topology rollback on the traffic target trajectory breakpoints caused by occlusion or missed detection to generate a continuous set of traffic target trajectories. Based on the continuous trajectory set of traffic targets, the tracking status information of traffic targets around intelligent driving vehicles is updated to generate multi-target tracking results for intelligent driving.
[0006] Optionally, the intelligent driving vehicle acquires continuous road image frames in chronological order of vehicle travel using an onboard front-view camera or surround-view camera, reads the acquisition timestamp corresponding to each continuous road image frame, performs lens distortion correction on the continuous road image frames to correct edge stretching and image curvature generated during the imaging process of the onboard camera, performs inter-frame temporal alignment on the continuous road image frames according to the acquisition timestamp, performs brightness equalization, contrast enhancement and motion blur suppression on the continuous road image frames, and standardizes and regularizes the continuous road image frames by unifying image size and pixel scale to generate a standardized image sequence of road scene.
[0007] Optionally, the generation of the road scene semantic constraint set specifically includes: Extract road direction lines from a standardized image sequence of a road scene to generate a set of road direction line elements; Based on the set of road direction line elements, perspective intersection analysis is performed to determine the road vanishing point. Then, using the road vanishing point as the perspective guide point, direction normalization is performed to generate a set of vanishing point guide direction lines. Perform road skeleton semantic reasoning on the set of vanishing point guide direction lines to generate a road skeleton semantic map; Based on the road skeleton semantic map, the lane extension direction, the drivable space range and the traffic target association boundary are determined, and a set of semantic constraints for the road scene is generated through semantic constraint aggregation.
[0008] Optionally, the generation of the traffic target detection result set specifically includes: The improved DINO-DETR visual detection model inputs a set of semantic constraints from the road scene. This improved model includes a road candidate localization module, a lane extension phase embedding module, a boundary adjacency query transfer module, and a joint target detection decoding module. The improvements to the traditional DINO-DETR model are as follows: the traditional model uses candidate boxes from image features, denoising queries, and iterative decoder updates as the basis for detection, predicting target categories and bounding boxes. The improved DINO-DETR model incorporates the drivable space range, traffic target association boundaries, and lane extension directions from the road scene semantic constraints into the target query generation process. It determines traffic target candidate regions using road cone projection, embeds lane direction phases using projective phase frames, introduces a boundary manifold folding mechanism to form boundary adjacency coordinates, and generates a set of traffic target detection results through query symbol layer parsing and decoding. In the road domain candidate localization module, based on the drivable space range and traffic target association boundary in the road scene semantic constraint set, the drivable space range is projected with a road domain cone strip to determine the traffic target candidate area, and sparse target query vectors are deployed in the traffic target candidate area to generate a road domain candidate query vector set. In the lane extension phase embedding module, the projective phase frame embedding is performed on the road domain candidate query vector set. Combined with the lane extension direction in the road scene semantic constraint set, the road domain candidate query vector set is mapped to the projective phase domain with the lane extension direction as the main axis to generate the lane phase query vector set. In the boundary adjacency query migration module, a boundary manifold folding mechanism is introduced to divide the traffic target associated boundary into boundary manifold fragments. Taking the phase position of the lane phase query vector set as the folding starting point, the boundary manifold fragment folding mapping is performed on the lane phase query vector set to generate the boundary adjacency query vector set. In the joint decoding module for target detection, query symbol parsing and decoding are performed on the set of boundary adjacency query vectors. The set of boundary adjacency query vectors is decomposed into category symbols, confidence symbols, and frame symbols. The traffic target detection result set is generated according to the index correspondence between the category symbols, confidence symbols, and frame symbols.
[0009] Optionally, the generation of the traffic target tracking feature set specifically includes: Based on the traffic target detection result set, phase division is performed on the visible area, occlusion area and boundary change area of traffic targets in continuous road image frames to determine the occlusion phase sequence corresponding to the traffic targets. Construct an occlusion phase signature based on the occlusion phase sequence; The occlusion phase signature is fused with the target appearance and scale of the traffic target, and gating capsule routing is performed according to the traffic target number and the frame number to generate a traffic target tracking feature set.
[0010] Optionally, the generation of the set of motion states of traffic targets specifically includes: Arrange the traffic target tracking feature set in a time sequence to generate a traffic target time sequence observation set; Based on the time-series observation set of traffic targets, the position change, target shape change and occlusion phase change of the same traffic target number in adjacent consecutive road image frames are determined, and a set of traffic target motion attitude parameters is generated. Perform target inertial potential field inversion on the set of traffic target motion attitude parameters to determine the motion vector state, target morphology state and occlusion continuation state of the same traffic target number in consecutive road image frames; The motion vector state, target shape state, and occlusion continuation state are associated and bound according to the traffic target number and the frame number to generate a set of traffic target motion states.
[0011] Optionally, the generation of the initial trajectory set of the traffic target specifically includes: Based on the set of motion states of traffic targets, establish a set of cross-frame candidate association pairs for traffic targets; Based on the set of cross-frame candidate association pairs of traffic targets, calculate the motion vector difference cost, target shape difference cost and occlusion continuation difference cost corresponding to each cross-frame candidate association pair of traffic targets, and generate a set of cross-frame difference costs of traffic targets. Construct a cross-frame correlation cost matrix based on the cross-frame difference cost set of traffic targets; Based on the minimum cost mutual exclusion allocation relationship between the matrix row index and matrix column index in the cross-frame association cost matrix, identity consistency matching is performed on traffic targets in adjacent consecutive road image frames to generate identity matching results. The traffic target numbers with identity consistency matching relationships in the identity matching results are connected consecutively according to their frame sequence numbers to generate the initial trajectory set of traffic targets.
[0012] Optionally, the generation of the continuous trajectory set of traffic targets specifically includes: The positions between adjacent frame numbers in the initial trajectory set of traffic targets are detected to determine the set of trajectory breakpoints of traffic targets. Perform a remnant topology rollback on the set of traffic target trajectory breakpoints to generate trajectory breakpoint connection relationships; Based on the trajectory breakpoint connection relationship, the preceding and subsequent trajectory segments with continuous connection relationship are chained and closed according to their frame numbers to generate a continuous trajectory set of traffic targets.
[0013] Optionally, based on the trajectory number, traffic target number, frame number, continuous trajectory chain, and missing frame position in the continuous trajectory set of traffic targets, and using the trajectory number as the tracking status update index and the frame number as the tracking status update timing field, the traffic target number in the same continuous trajectory chain is read frame by frame. When the continuous trajectory chain contains a traffic target number in the current frame number, the tracking status corresponding to the traffic target number is marked as continuous tracking status. When the continuous trajectory chain has a missing frame position in the current frame number, the tracking status corresponding to the missing frame position is marked as breakpoint closure status. When the current frame number is the frame number of the end of the continuous trajectory chain, the tracking status of the corresponding trajectory number is marked as trajectory termination status. The trajectory number, traffic target number, frame number, continuous tracking status, breakpoint closure status, and trajectory termination status are written into the tracking status information of traffic targets around the intelligent driving vehicle to generate multi-target tracking results for intelligent driving.
[0014] The beneficial effects of this invention are: This invention acquires continuous road image frames during the driving process of intelligent driving vehicles and generates a standardized image sequence of road scenes. It further utilizes vanishing point-guided road skeleton semantic reasoning to determine lane extension directions, drivable space ranges, and traffic target association boundaries. This allows traffic target detection to move beyond relying solely on pixel features in a single frame, instead combining a set of road scene semantic constraints for sparse target query and localization. By inputting the set of road scene semantic constraints into the improved DINO-DETR visual detection model, the traffic target detection result set maintains a correspondence with lane extension directions, drivable space ranges, and traffic target association boundaries. This reduces interference from complex road backgrounds, road edge areas, and non-driving areas, improving the scene adaptability of the traffic target detection result set in intersections, curved road sections, densely populated target areas, and scenarios with significant changes in near and far scales.
[0015] This invention constructs an occlusion phase signature for traffic targets and fuses this signature with target appearance and scale to generate a traffic target tracking feature set. This allows for the continuous representation of the state changes of traffic targets during occlusion entry, occlusion maintenance, and occlusion detachment. Furthermore, based on the traffic target tracking feature set, motion attitude parameters are determined, and a traffic target motion state set is generated through target inertial potential field inversion. This is combined with cross-frame association cost matrix, identity consistency matching, and residual topology rewind to generate a continuous traffic target trajectory set. This reduces traffic target trajectory breakpoints caused by occlusion, missed detection, and target intersection, lowers the probability of traffic target number jumps and identity confusion, and enables multi-target tracking results for intelligent driving to have better trajectory continuity, identity consistency, and tracking stability. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of the multi-target tracking method for intelligent driving based on visual intelligence proposed in this invention; Figure 2 This is a schematic diagram of the structure of the improved DINO-DETR visual detection model, which is based on visual intelligence and is intended for intelligent driving. Detailed Implementation
[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0018] refer to Figures 1-2 A multi-target tracking method based on visual intelligence for intelligent driving includes the following steps: Collect and preprocess continuous road image frames during the driving process of intelligent driving vehicles to generate standardized image sequences of road scenes; Based on standardized image sequences of road scenes, road skeleton semantic reasoning guided by vanishing points is performed to determine lane extension direction, drivable space range and traffic target association boundary, and generate a set of semantic constraints for road scenes. The semantic constraint set of the road scene is input into the improved DINO-DETR visual detection model to perform sparse target query and localization, and a set of traffic target detection results is generated. Based on the traffic target detection result set, an occlusion phase signature of the traffic target is constructed and fused with the target appearance and target scale of the traffic target to generate a traffic target tracking feature set; Using the traffic target tracking feature set as the motion observation source, the motion attitude parameters of the traffic target in consecutive frames are determined, and the target inertial potential field inversion is performed on the traffic target to generate a set of traffic target motion states; Based on the set of traffic target motion states, a cross-frame association cost matrix of traffic targets between adjacent frames is constructed, and identity consistency matching of traffic targets is performed to generate an initial trajectory set of traffic targets. Based on the initial set of traffic target trajectories, perform residual topology rollback on the traffic target trajectory breakpoints caused by occlusion or missed detection to generate a continuous set of traffic target trajectories. Based on the continuous trajectory set of traffic targets, the tracking status information of traffic targets around intelligent driving vehicles is updated to generate multi-target tracking results for intelligent driving.
[0019] In this embodiment, the intelligent driving vehicle acquires continuous road image frames in chronological order of vehicle travel using an onboard front-view camera or surround-view camera. It reads the acquisition timestamp corresponding to each continuous road image frame, performs lens distortion correction on the continuous road image frames to correct edge stretching and image curvature generated during the onboard camera imaging process, and aligns the continuous road image frames according to the acquisition timestamps to maintain a consistent time order between adjacent frames. It also performs brightness equalization, contrast enhancement, and motion blur suppression on the continuous road image frames to reduce the impact of vehicle driving vibrations, sudden changes in illumination, and imaging noise on road scene analysis and traffic target detection. Finally, it standardizes and regularizes the continuous road image frames by unifying image size and pixel scale to generate a standardized road scene image sequence.
[0020] In this embodiment, the generation of the road scene semantic constraint set specifically includes: Extract road direction lines from a standardized image sequence of a road scene to generate a set of road direction line elements; The process of generating the road direction line element set is as follows: Road images are read frame by frame from a standardized image sequence of the road scene, with a uniform pixel size of 1920×1080. In each frame, a road surface analysis area from 40% to 95% of the image height is selected. Gray-level gradient calculations are performed on this area, and road edge pixels with a gray-level gradient amplitude greater than 45 gray levels / pixel are selected. Continuous connection and line segment fitting are performed on the road edge pixels to obtain road direction line elements with a length greater than 30 pixels and a direction angle between 15° and 85°. Inter-frame consistency checks are performed on road direction line elements in adjacent frames with a direction angle difference of less than 5° and a lateral distance between endpoints of less than 20 pixels. Road direction line elements that pass the inter-frame consistency check are then grouped according to their frame number, start coordinates, end coordinates, line segment length, and direction angle to generate the road direction line element set. Based on the set of road direction line elements, perspective intersection analysis is performed to determine the road vanishing point. Then, using the road vanishing point as the perspective guide point, direction normalization is performed to generate a set of vanishing point guide direction lines. The process of generating the vanishing point guide direction line set is as follows: Read the start coordinates, end coordinates, line segment length, and direction angle of each road direction line element in the road direction line element set. Remove road direction line elements with a line segment length less than 30 pixels and a direction angle difference exceeding 45° of the main direction angle within the frame. Perform extension line intersection calculations on the remaining road direction line elements, statistically analyze the spatial concentration of each intersection position within 5 consecutive frames, and determine the intersection positions with a horizontal offset of less than 60 pixels, a vertical offset of less than 45 pixels, and a cumulative intersection count greater than 8 as the road vanishing point. Using the road vanishing point as the perspective guide point, calculate the angle deviation between the extension direction of each road direction line element and the road vanishing point. Correct the direction of road direction line elements with an angle deviation greater than 12°, and retain the original direction of road direction line elements with an angle deviation less than 12°. Then, aggregate the corrected road direction line elements and the road direction line elements with the retained original direction according to their frame number, start coordinates, end coordinates, line segment length, direction angle, and angle deviation to generate the vanishing point guide direction line set. Perform road skeleton semantic reasoning on the set of vanishing point guide direction lines to generate a road skeleton semantic map; The specific process of generating a road skeleton semantic graph is as follows: The vanishing point guide direction line set is read according to frame order. Road skeleton semantic reasoning is based on the spatial position, direction angle, and included angle deviation of road direction line elements to determine the extension and adjacency relationships of each road direction line element in the road scene. In the same frame of the road image, road direction line elements with a direction angle difference of less than 8° and an endpoint distance of less than 25 pixels are grouped into the same direction line group. Direction line groups located on the left and right sides of the road image and pointing towards the road vanishing point are identified as road boundary direction line groups. Direction line groups located in the middle of the road image and extending towards the road vanishing point are identified as lane extension direction line groups. Based on the positional adjacency relationship between the road boundary direction line groups and the lane extension direction line groups, road skeleton nodes and road skeleton edges are determined. Road skeleton nodes represent the endpoints, intersections, and extension connection points of road direction line elements, while road skeleton edges represent the road extension relationships between adjacent road skeleton nodes. The road skeleton nodes, road skeleton edges, and the corresponding road boundary direction line groups and lane extension direction line groups are semantically labeled to form the road skeleton semantic graph. Based on the road skeleton semantic map, the lane extension direction, the drivable space range and the traffic target association boundary are determined, and a set of semantic constraints for the road scene is generated through semantic constraint aggregation. The process of generating the semantic constraint set for the road scene is as follows: Read the road skeleton nodes and edges in the road skeleton semantic map. Use the road skeleton edges extending from the bottom of the road image to the road vanishing point as candidate lane extension edges. Perform direction consistency processing on candidate lane extension edges with a direction angle difference of less than 6° and an angle of less than 10° with the line connecting to the road vanishing point to determine the lane extension direction. Using the lane extension direction as the vertical reference, perform a lateral boundary search along both sides of the road skeleton edges in the road image. Define the area where the lateral distance between adjacent road skeleton edges is greater than 80 pixels and the continuous length of the boundary is greater than 120 pixels as the drivable space range. Traffic target association boundaries are determined by the left and right edges of the drivable space, the forward extension area, and the dense area of road skeleton nodes. Semantic constraint collection involves dividing the road image into 32×18 semantic grid units of 60×60 pixels, writing the lane extension direction into the direction constraint field of the corresponding semantic grid unit, writing the drivable space into the region constraint field of the corresponding semantic grid unit, and writing the traffic target association boundaries into the boundary constraint field of the corresponding semantic grid unit. The data is then structured and stored according to the frame number, semantic grid unit position, direction constraint field, region constraint field, and boundary constraint field to generate a set of semantic constraints for the road scene.
[0021] In this embodiment, the generation of the traffic target detection result set specifically includes: The improved DINO-DETR visual detection model inputs a set of semantic constraints from the road scene. This improved model includes a road candidate localization module, a lane extension phase embedding module, a boundary adjacency query transfer module, and a joint target detection decoding module. The improvements to the traditional DINO-DETR model are as follows: the traditional model uses candidate boxes from image features, denoising queries, and iterative decoder updates as the basis for detection, predicting target categories and bounding boxes. The improved DINO-DETR model incorporates the drivable space range, traffic target association boundaries, and lane extension directions from the road scene semantic constraints into the target query generation process. It determines traffic target candidate regions using road cone projection, embeds lane direction phases using projective phase frames, introduces a boundary manifold folding mechanism to form boundary adjacency coordinates, and generates a set of traffic target detection results through query symbol layer parsing and decoding. The improved DINO-DETR visual detection model comprises a road candidate localization module, a lane extension phase embedding module, a boundary adjacency query migration module, and a joint target detection decoding module, connected sequentially. The road candidate localization module receives a set of semantic constraints for the road scene and a visual feature grid corresponding to the road image. The visual feature grid has a data dimension of 120×68×256. The set of semantic constraints for the road scene is input according to the field format of frame number, semantic grid cell position, lane extension direction, drivable space range, and traffic target associated boundary. The output is a set of road candidate query vectors with a data format of N×256. N is less than 128; the lane extension phase embedding module converts the road domain candidate query vector set into a lane phase query vector set with longitudinal projective phase, lateral offset phase, and cone layer phase; the boundary adjacency query migration module converts the lane phase query vector set into a boundary adjacency query vector set with boundary manifold fragment number, boundary adjacency coordinates, fragment tangential direction, and fragment normal direction; the target detection joint decoding module decodes the boundary adjacency query vector set according to the query index number and outputs a traffic target detection result set consisting of traffic target category number, traffic target confidence value, and traffic target bounding box; The training data for the improved DINO-DETR visual detection model comes from continuous road image frames captured by the forward-facing camera of an autonomous driving vehicle. The image size is uniformly 1920×1080. The training samples include urban main roads, intersections, elevated roads, tunnel entrances, rainy roads, nighttime roads, and congested road scenes. A total of 8000 frames were selected as training data, 1200 frames as validation data, and 800 frames as test data. Each road image frame is labeled with traffic target category, traffic target bounding box, traffic target occlusion level, drivable space range, lane extension direction, and traffic target associated boundary. Boundary manifold fragment numbers are added to the traffic target associated boundary, so that the training samples simultaneously contain target detection annotations and road semantic constraint annotations. A joint loss function is used during training, and the loss terms include category symbol loss, confidence symbol loss, bounding box symbol loss, projective phase consistency loss, boundary manifold fragment adjacency loss, and query symbol layering loss. The loss parameters were set as follows: category symbol loss weight 1.0, confidence symbol loss weight 2.0, bounding box symbol loss weight 5.0, projective phase consistency loss weight 0.8, boundary manifold fragment adjacency loss weight 0.6, and query symbol layer parsing decoding loss weight 0.5. The training parameters were: input batch size 8, query vector dimension 256, number of query vectors per frame 128, number of decoding layers 6, initial learning rate 0.001, backbone feature layer learning rate 0.003, weight decay coefficient 0.023, preheating iterations 1500, training 60 rounds, with the learning rate adjusted to 0.1 times the original learning rate in rounds 40 and 52 respectively. The improved DINO-DETR visual detection model was considered to have converged when the total loss variation in the validation data was less than 0.02 for 5 consecutive rounds, and the category symbol loss, bounding box symbol loss, and boundary manifold fragment adjacency loss did not show a continuous increase. In the road domain candidate localization module, based on the drivable space range and traffic target association boundary in the road scene semantic constraint set, the drivable space range is projected with a road domain cone strip to determine the traffic target candidate area, and sparse target query vectors are deployed in the traffic target candidate area to generate a road domain candidate query vector set. The process of generating the road domain candidate query vector set is as follows: The drivable space range and traffic target association boundaries in the road scene semantic constraint set are read; a 1920×1080 road image is mapped to a 120×68 visual feature grid at a 16-fold downsampling ratio, and the drivable space range is simultaneously mapped to the visual feature grid; using the lower edge of the drivable space range as the near-end reference and the position pointing in the lane extension direction as the far-end reference, a road domain cone projection is performed on the drivable space range, forming six cone projection layers arranged along the lane extension direction; within each cone projection layer, traffic is determined based on the traffic target association boundaries. The target candidate region is composed of grid cells with a continuous area greater than 9 visual feature grids and a lateral distance of less than 4 visual feature grids from the boundary associated with the traffic target. Within each traffic target candidate region, a placement position is set according to a 3×3 visual feature grid, and a sparse target query vector with a dimension of 256 is configured at each placement position. The number of sparse target query vectors configured in a single frame of road image is less than 128. The sparse target query vectors are collected according to the frame number, cone projection layer number, placement position coordinates and corresponding traffic target candidate region number to generate a road domain candidate query vector set. In the lane extension phase embedding module, the projective phase frame embedding is performed on the road domain candidate query vector set. Combined with the lane extension direction in the road scene semantic constraint set, the road domain candidate query vector set is mapped to the projective phase domain with the lane extension direction as the main axis to generate the lane phase query vector set. The process of generating the lane phase query vector set is as follows: Read the frame number, cone projection layer number, projection location coordinates, and traffic target candidate area number corresponding to each road domain candidate query vector in the road domain candidate query vector set, and combine this with the lane extension direction in the road scene semantic constraint set; construct a projective phase frame using the lane extension direction as the projective axis and the near-end center point of the drivable space in the current frame as the frame origin. The projective axis direction is divided into 64 longitudinal phase scales, and the lateral direction perpendicular to the projective axis is divided into 32 lateral phase scales; project the projection location coordinates of each road domain candidate query vector into the projective phase frame to determine the longitudinal projective phase, lateral offset phase, and cone layer phase corresponding to the road domain candidate query vector, and write the longitudinal projective phase, lateral offset phase, and cone layer phase into the corresponding road domain candidate query vector; aggregate the road domain candidate query vectors that have completed phase writing according to their frame number, longitudinal projective phase, lateral offset phase, cone layer phase, and traffic target candidate area number to generate the lane phase query vector set; In the boundary adjacency query migration module, a boundary manifold folding mechanism is introduced to divide the traffic target associated boundary into boundary manifold fragments. Taking the phase position of the lane phase query vector set as the folding starting point, the boundary manifold fragment folding mapping is performed on the lane phase query vector set to generate the boundary adjacency query vector set. The process of generating the boundary adjacency query vector set is as follows: Read the frame number, longitudinal projective phase, lateral offset phase, cone layer phase, and traffic target candidate region number corresponding to each lane phase query vector in the lane phase query vector set, and obtain the traffic target associated boundary in the road scene semantic constraint set; the boundary manifold folding mechanism maps the traffic target associated boundary to a 120×68 visual feature grid, continuously connects boundary grids with adjacent distances less than 2 visual feature grids and directional angles less than 15° to form continuous boundary segments corresponding to the traffic target associated boundary; divides the continuous boundary segments into multiple boundary manifold fragments according to a segmentation length of 8 visual feature grids, each boundary manifold fragment recording the fragment number, fragment center coordinates, fragment tangential direction, fragment normal direction, and adjacent fragment numbers; Using the phase position of each lane phase query vector in the lane phase query vector set as the folding starting point, within a vertical window of ±4 visual feature grids and a horizontal window of ±6 visual feature grids, the target boundary manifold fragment corresponding to the phase position is determined. Based on the fragment center coordinates, fragment tangential direction, and fragment normal direction of the target boundary manifold fragment, the phase position of the corresponding lane phase query vector is folded and mapped to the boundary adjacency coordinates of the target boundary manifold fragment. The fragment number, boundary adjacency coordinates, fragment tangential direction, fragment normal direction, and adjacent fragment number of the target boundary manifold fragment are written into the corresponding lane phase query vector. The lane phase query vectors that have been written are summarized according to the frame number, cone layer phase, fragment number of the target boundary manifold fragment, and boundary adjacency coordinates to generate a boundary adjacency query vector set. In the joint decoding module for target detection, query symbol parsing and decoding are performed on the set of boundary adjacency query vectors. The set of boundary adjacency query vectors is decomposed into category symbols, confidence symbols, and frame symbols. The traffic target detection result set is generated according to the index correspondence between category symbols, confidence symbols, and frame symbols. The process of generating the traffic target detection result set is as follows: First, the frame number, query index number, boundary adjacency coordinates, and vector numerical content corresponding to each boundary adjacency query vector in the boundary adjacency query vector set are read. The dimension of the boundary adjacency query vector is 256. Second, in the joint decoding module for target detection, the boundary adjacency query vectors are arranged into a query decoding matrix according to the query index number. Third, query symbol layer decoding is performed on the query decoding matrix. Specifically, dimensions 1 to 96 of each boundary adjacency query vector are decoded as category symbols, dimensions 97 to 128 as confidence symbols, and dimensions 129 to 256 as bounding box symbols. Finally, category channel mapping is performed on the category symbols to determine the traffic target class. The traffic target category is numbered, including vehicle category, pedestrian category, non-motorized vehicle category, and road obstacle category. A single-value channel mapping is performed on the confidence symbol to determine the traffic target confidence value. A four-endpoint channel mapping is performed on the bounding box symbol to determine the top-left, top-left, bottom-right, and bottom-right x and y coordinates of the traffic target bounding box. Based on the same query index number, the traffic target category number, traffic target confidence value, traffic target bounding box, and corresponding frame number are bound together to form a single traffic target detection result. All traffic target detection results within the same frame are then aggregated according to the query index number to generate a traffic target detection result set.
[0022] In this embodiment, the generation of the traffic target tracking feature set specifically includes: Based on the traffic target detection result set, phase division is performed on the visible area, occlusion area and boundary change area of traffic targets in continuous road image frames to determine the occlusion phase sequence corresponding to the traffic targets. The process of determining the occlusion phase sequence is as follows: Read the traffic target bounding box, traffic target category, traffic target confidence value, and frame number from the traffic target detection result set; assign a traffic target number to the same traffic target according to the overlap relationship of the traffic target bounding boxes in adjacent consecutive road image frames; under the same traffic target number, compare the traffic target bounding boxes in adjacent consecutive road image frames sequentially; when the target outline within the traffic target bounding box is continuous, the traffic target confidence value is greater than 0.60, and the area change ratio of the traffic target bounding box is less than 25%, the corresponding area is determined as the visible area of the traffic target; when the traffic target... When the area change ratio of the target bounding box is greater than 25%, the confidence value of the traffic target is less than 0.60, or the overlap ratio of the traffic target bounding box with other traffic target bounding boxes is greater than 40%, the corresponding area is identified as an occluded area; when the positional offset of the left, right, top, or bottom boundary of the traffic target bounding box in adjacent consecutive road image frames is greater than 12 pixels, the corresponding area is identified as a boundary change area; based on the sequential relationship of the visible area, occluded area, and boundary change area of the traffic target in the frame sequence, the traffic target state under the same traffic target number is phase-divided to form an occlusion phase sequence; Construct an occlusion phase signature based on the occlusion phase sequence; The specific process of constructing the occlusion phase signature is as follows: Read the visible phase, occlusion entry phase, occlusion sustaining phase, and occlusion exit phase corresponding to the same traffic target number in the occlusion phase sequence, and record the start frame, end frame, and duration frame number of each occlusion phase in sequence according to the frame number; mark the visible phase as 0, the occlusion entry phase as 1, the occlusion sustaining phase as 2, and the occlusion exit phase as 3; under the same traffic target number, bind the phase mark, start frame, end frame, duration frame number, traffic target bounding box area change ratio, and traffic target confidence value corresponding to each occlusion phase in sequence to form the occlusion phase signature; The occlusion phase signature is fused with the target appearance and target scale of the traffic target, and gating capsule routing is performed according to the traffic target number and the frame number to generate a set of traffic target tracking features. The process of generating the traffic target tracking feature set is as follows: Read the occlusion phase signature, target appearance, and target scale under the same traffic target number and frame sequence number. Use the occlusion phase signature as the input to the phase capsule, the target appearance as the input to the appearance capsule, and the target scale as the input to the scale capsule. Set the gating routing weights based on the phase marker in the occlusion phase signature. When the phase marker is a visible phase, set the gating routing weights of the appearance capsule, scale capsule, and phase capsule to 0.60, 0.25, and 0.15, respectively. When the phase marker is an occlusion entry phase, set them to 0.40, 0.35, and 0.2, respectively. 5. When the phase is marked as occlusion-preserving phase, the values are set to 0.20, 0.45, and 0.35 respectively; when the phase is marked as occlusion-detached phase, the values are set to 0.45, 0.30, and 0.25 respectively. Three rounds of capsule routing iteration are performed on the appearance capsule, scale capsule, and phase capsule according to the gated routing weight. In each round of capsule routing iteration, the appearance capsule, scale capsule, and phase capsule are bound according to the traffic target number and the frame number. The outputs of the appearance capsule, scale capsule, and phase capsule after the three rounds of capsule routing iteration are concatenated with the same index to generate traffic target tracking features and form a traffic target tracking feature set.
[0023] In this embodiment, the generation of the traffic target motion state set specifically includes: Arrange the traffic target tracking feature set in a time sequence to generate a traffic target time sequence observation set; The process of generating a traffic target time-series observation set is as follows: Based on the traffic target tracking feature set; using the traffic target number as the identification field for the same traffic target and the frame number as the time-series arrangement field, the occlusion phase signature, target appearance, and target scale under the same traffic target number are arranged from front to back according to the frame number; when the same traffic target number exists in adjacent consecutive road image frames, the corresponding occlusion phase signature, target appearance, and target scale are written into the same time-series observation chain; when multiple traffic target numbers exist under the same frame number, time-series observation chains corresponding to the traffic target numbers are established respectively; all time-series observation chains are structured and stored according to the traffic target number and the frame number to generate a traffic target time-series observation set. Based on the time-series observation set of traffic targets, the position change, target shape change and occlusion phase change of the same traffic target number in adjacent consecutive road image frames are determined, and a set of traffic target motion attitude parameters is generated. The specific process for generating the traffic target motion attitude parameter set is as follows: Read the traffic target center coordinates, appearance, scale, and occlusion phase signature of adjacent consecutive road image frames under the same traffic target number in the traffic target time-series observation set, and compare adjacent frames sequentially according to their frame numbers; determine the position change based on the lateral and longitudinal differences of the traffic target center coordinates between the previous and subsequent frame numbers; determine the target morphology change based on the differences in width, height, and aspect ratio in the target scale between the previous and subsequent frame numbers; determine the occlusion phase change based on the phase marker changes in the occlusion phase signature between the previous and subsequent frame numbers; write the position change, target morphology change, and occlusion phase change into the same motion attitude record according to the traffic target number and the subsequent frame number, and form the traffic target motion attitude parameter set from all motion attitude records; Perform target inertial potential field inversion on the set of traffic target motion attitude parameters to determine the motion vector state, target morphology state and occlusion continuation state of the same traffic target number in consecutive road image frames; The process of determining the motion vector state, target morphology state, and occlusion continuation state is as follows: The position change, target morphology change, and occlusion phase change corresponding to the same traffic target number are read from the traffic target motion attitude parameter set. The motion attitude records corresponding to three consecutive frame numbers are used as the target inertial potential field inversion window. Within the target inertial potential field inversion window, the position change is used as the potential field direction component, the target morphology change as the potential field morphology component, and the occlusion phase change as the potential field occlusion component. Time weights of 0.2, 0.3, and 0.5 are assigned respectively according to the frame number from front to back. The potential field direction components are weighted and superimposed to determine the motion vector state of the same traffic target number in the current frame number. The potential field morphology components are weighted and superimposed to determine the target morphology state of the same traffic target number in the current frame number. The potential field occlusion components are weighted and superimposed to determine the occlusion continuation state of the same traffic target number in the current frame number. The motion vector state, target shape state, and occlusion continuation state are associated and bound according to the traffic target number and the frame number to generate a set of traffic target motion states.
[0024] In this embodiment, the generation of the initial trajectory set of traffic targets specifically includes: Based on the set of motion states of traffic targets, establish a set of cross-frame candidate association pairs for traffic targets; The process of establishing a cross-frame candidate association set for traffic targets is as follows: Based on the set of traffic target motion states, the preceding and following consecutive road image frames are determined according to their frame numbers; the traffic target number in the preceding consecutive road image frame is used as the preceding frame target number, and the traffic target number in the following consecutive road image frame is used as the following frame target number; the candidate search range in the following consecutive road image frame is determined according to the motion vector state corresponding to the preceding frame target number; when the position corresponding to the following frame target number falls within the candidate search range, and the target morphology state corresponding to the following frame target number matches the target morphology state corresponding to the preceding frame target number... When the morphological states belong to the same morphological interval, a cross-frame candidate association pair is established between the target number of the previous frame and the target number of the next frame. When the occlusion continuation state corresponding to the target number of the previous frame is the occlusion hold state, the candidate search range is extended along the direction pointed to by the motion vector state by one target scale length, and a cross-frame candidate association pair is established within the extended candidate search range. Each cross-frame candidate association pair is structured and written according to the target number of the previous frame, the target number of the next frame, the sequence number of the previous frame, the sequence number of the next frame, the motion vector state, the target morphological state, and the occlusion continuation state, forming a set of traffic target cross-frame candidate association pairs. Based on the set of cross-frame candidate association pairs of traffic targets, calculate the motion vector difference cost, target shape difference cost and occlusion continuation difference cost corresponding to each cross-frame candidate association pair of traffic targets, and generate a set of cross-frame difference costs of traffic targets. The process of generating the cross-frame difference cost set for traffic targets is as follows: Based on the cross-frame candidate association pair set for traffic targets, the motion vector state corresponding to the target number in the previous frame is compared with the motion vector state corresponding to the target number in the subsequent frame, and the magnitude difference is calculated to generate the motion vector difference cost; the target shape state corresponding to the target number in the previous frame is compared with the target shape state corresponding to the target number in the subsequent frame, and the width difference, height difference, and aspect ratio difference are calculated to generate the target shape difference cost; the occlusion continuation state corresponding to the target number in the previous frame is compared with the occlusion continuation state corresponding to the target number in the subsequent frame, and the occlusion continuation frame difference is calculated to generate the occlusion continuation difference cost; the motion vector difference cost, target shape difference cost, and occlusion continuation difference cost are written into the corresponding cross-frame candidate association pair for traffic targets to form a cross-frame difference cost record for traffic targets; and the cross-frame difference cost set for traffic targets is formed by all the cross-frame difference cost records for traffic targets. Construct a cross-frame correlation cost matrix based on the cross-frame difference cost set of traffic targets; The process of constructing the cross-frame association cost matrix is as follows: Read the target number of the previous frame, the target number of the next frame, the sequence number of the previous frame, the sequence number of the next frame, the motion vector difference cost, the target morphology difference cost, and the occlusion continuation difference cost corresponding to each traffic target cross-frame difference cost record in the traffic target cross-frame difference cost set; use the target number of the previous frame in the sequence number of the previous frame as the row index of the cross-frame association cost matrix, and use the target number of the next frame in the sequence number of the next frame as the column index of the cross-frame association cost matrix; weight and combine the motion vector difference cost, target morphology difference cost, and occlusion continuation difference cost in the same traffic target cross-frame difference cost record to obtain the total cross-frame association cost of the corresponding matrix element; write the total cross-frame association cost into the matrix element with the corresponding row and column indices; when there is no cross-frame candidate association pair for traffic targets, write the corresponding matrix element with an invalid association flag; build the matrix element frame by frame according to the sequence number of the previous and next frames to form the cross-frame association cost matrix. Based on the minimum cost mutual exclusion allocation relationship between the matrix row index and matrix column index in the cross-frame association cost matrix, identity consistency matching is performed on traffic targets in adjacent consecutive road image frames to generate identity matching results. The process of generating identity matching results is as follows: Read the matrix row index, matrix column index, matrix elements, total cross-frame association cost, and invalid association flag from the cross-frame association cost matrix; map the matrix row index to the target number of the previous frame in the preceding consecutive road image frame, and map the matrix column index to the target number of the subsequent frame in the following consecutive road image frame; mask the matrix elements with invalid association flags; establish a candidate allocation sequence for the remaining matrix elements according to the total cross-frame association cost in ascending order; in the candidate allocation sequence, sequentially select the matrix element with the smallest total cross-frame association cost as the identity matching result. Candidates are matched, and mutual exclusion marking is applied to the selected matrix row index and matrix column index, so that the same previous frame target number is connected to only one subsequent frame target number, and the same subsequent frame target number receives only one previous frame target number; when the total cross-frame association cost of identity matching candidates is less than the preset identity connection threshold, the corresponding previous frame target number, subsequent frame target number, previous frame sequence number, subsequent frame sequence number, and total cross-frame association cost are written into the identity matching record; when the total cross-frame association cost is greater than the preset identity connection threshold, the identity connection corresponding to the matrix element is stopped, and the identity matching result is formed. The traffic target numbers with identity consistency matching relationships in the identity matching results are connected consecutively according to their frame sequence numbers to generate an initial trajectory set of traffic targets; The process of generating the initial trajectory set for traffic targets is as follows: Based on the identity matching results, the target number of the previous frame, the target number of the next frame, the sequence number of the previous frame, the sequence number of the next frame, and the identity consistency matching relationship; the target number of the previous frame in the sequence number of the previous frame is used as the trajectory starting point, and the target number of the next frame in the sequence number of the next frame is used as the trajectory continuation point, connecting the traffic target numbers with identity consistency matching relationships in the order of their respective frame numbers from front to back; when multiple consecutive sequence numbers of their respective frame numbers have identity consistency matching relationships corresponding to the same traffic target number, the corresponding traffic target numbers are written into the same initial trajectory chain in sequence; when a traffic target number does not form an identity consistency matching relationship in the sequence number of its next frame, the continuation of the corresponding initial trajectory chain is terminated; when the target number of the next frame does not form an identity consistency matching relationship with the target number of the previous frame, a new initial trajectory chain is established with the target number of the next frame; all initial trajectory chains are stored in a structured manner according to the trajectory number, traffic target number, and sequence number of their respective frame to generate the initial trajectory set for traffic targets.
[0025] In this embodiment, the generation of the continuous trajectory set of traffic targets specifically includes: The positions between adjacent frame numbers in the initial trajectory set of traffic targets are detected to determine the set of trajectory breakpoints of traffic targets. The process of determining the traffic target trajectory breakpoint set is as follows: read the trajectory number, traffic target number, frame number, and initial trajectory chain from the initial trajectory set of traffic targets, and check the frame number corresponding to adjacent traffic target numbers in the same initial trajectory chain one by one according to the trajectory number; when there is a frame sequence interval between the frame number corresponding to the later traffic target number and the frame number corresponding to the earlier traffic target number, or when the same initial trajectory chain does not form an identity consistency matching relationship between adjacent frame numbers, the corresponding position is determined as the traffic target trajectory breakpoint, and the traffic target trajectory breakpoint set is generated. Perform a remnant topology rollback on the set of traffic target trajectory breakpoints to generate trajectory breakpoint connection relationships; The process of generating trajectory breakpoint connection relationships is as follows: Based on the set of traffic target trajectory breakpoints, the traffic target numbers that exist consecutively before the frame number where the breakpoint starts are determined as the preceding trajectory segments, and the traffic target numbers that exist consecutively after the frame number where the breakpoint ends are determined as the following trajectory segments. Residual topology rollback uses the traffic target trajectory breakpoint as the rollback center, extending the preceding trajectory segments forward along the frame number to the missing frame position, and tracing the following trajectory segments backward along the frame number to the missing frame position. Within the missing frame position, the trajectory numbers corresponding to the preceding and following traffic target numbers, the continuity of the frame numbers, and the correspondence of the breakpoint positions are compared. When the preceding and following trajectory segments on both sides of the same traffic target trajectory breakpoint satisfy the continuity of the frame numbers, and the preceding and following traffic target numbers belong to the same trajectory number, the preceding trajectory segment, the missing frame position, and the following trajectory segment are written into the same trajectory breakpoint connection record. The trajectory breakpoint connection relationship is formed by all trajectory breakpoint connection records. Based on the connection relationship of trajectory breakpoints, the preceding trajectory segments and the following trajectory segments with continuous connection relationships are chained and closed according to their frame numbers to generate a set of continuous trajectories of traffic targets. The process of generating a continuous trajectory set for traffic targets is as follows: The trajectory number, preceding trajectory segment, following trajectory segment, missing frame position, and continuous connection relationship in the trajectory breakpoint connection relationship are read. When a continuous connection relationship corresponds to the same trajectory number, the preceding trajectory segment, missing frame position, and following trajectory segment are written into the same continuous trajectory chain according to the order of their respective frame numbers from front to back. The chain closure process uses the traffic target number at the end of the preceding trajectory segment as the closure start point and the traffic target number at the beginning of the following trajectory segment as the closure end point, setting the missing frame position as the closure transition position to form a continuous trajectory chain. All continuous trajectory chains are structured and stored according to the trajectory number, traffic target number, and their respective frame number to generate a continuous trajectory set for traffic targets.
[0026] In this embodiment, based on the trajectory number, traffic target number, frame number, continuous trajectory chain, and missing frame position in the continuous trajectory set of traffic targets, and using the trajectory number as the tracking status update index and the frame number as the tracking status update timing field, the traffic target number in the same continuous trajectory chain is read frame by frame. When the continuous trajectory chain has a traffic target number in the current frame number, the tracking status corresponding to the traffic target number is marked as continuous tracking status. When the continuous trajectory chain has a missing frame position in the current frame number, the tracking status corresponding to the missing frame position is marked as breakpoint closure status. When the current frame number is the frame number of the end of the continuous trajectory chain, the tracking status of the corresponding trajectory number is marked as trajectory termination status. The trajectory number, traffic target number, frame number, continuous tracking status, breakpoint closure status, and trajectory termination status are written into the tracking status information of traffic targets around the intelligent driving vehicle to generate multi-target tracking results for intelligent driving.
[0027] Example 1: A test vehicle equipped with a forward-facing camera and an onboard computing unit was used in a mixed urban road scenario involving main roads and intersections. The test route was approximately 3.8 kilometers long and included a six-lane dual carriageway, a dedicated right-turn lane, a bus stop area, a non-motorized vehicle mixed traffic area, and a traffic light-equipped intersection. The test vehicle traveled at speeds ranging from approximately 25 km / h to 42 km / h. The forward-facing camera captured continuous road image frames at a resolution of 1920×1080, with a frame rate of 20 frames per second. This scenario included situations such as vehicles obstructing traffic, pedestrians crossing, non-motorized vehicles traveling side-by-side, vehicles changing lanes, and buses obstructing small vehicles at bus stops. The number of traffic targets in a single frame typically ranged from 6 to 24, with the highest number reaching 31 at congested intersections. Existing multi-target tracking methods are prone to traffic target number jumps in this scenario, especially when a vehicle in front briefly obscures a pedestrian, a non-motorized vehicle is obscured by a bus, or a small vehicle reappears from the side or rear of a large vehicle. These methods can easily identify the original traffic target as a new traffic target, leading to an increase in traffic target trajectory breakpoints.
[0028] In this scenario, continuous road image frames captured by the forward-looking camera undergo brightness correction, size unification, and noise suppression to generate a standardized road scene image sequence. Subsequently, based on this standardized image sequence, vanishing point-guided road skeleton semantic reasoning is performed to determine the lane extension direction, drivable space range, and traffic target association boundaries, generating a road scene semantic constraint set. This set restricts candidate regions for traffic target detection, enabling the improved DINO-DETR visual detection model to perform sparse target query and localization within the lane extension direction and drivable space range, generating a traffic target detection result set. For pedestrians partially obscured by vehicles ahead, vehicles intersecting with non-motorized vehicles, and traffic targets briefly missed at intersection edges, this embodiment further constructs an occlusion phase signature based on the traffic target detection result set. The occlusion phase signature, target appearance, and target scale are jointly written into the traffic target tracking feature set, ensuring continuous representation of changes in traffic targets during occlusion entry, occlusion maintenance, and occlusion detachment.
[0029] During continuous tracking, the traffic target tracking feature set is used as a motion observation source to determine the changes in position, shape, and occlusion phase of the traffic target in adjacent consecutive road image frames. Furthermore, a traffic target motion state set is generated through target inertial potential field inversion. Subsequently, a cross-frame association cost matrix is constructed based on the traffic target motion state set, and identity consistency matching is performed on traffic targets in adjacent consecutive road image frames to generate an initial trajectory set for the traffic targets. For trajectory breakpoints caused by occlusion or missed detection, residual topology rollback is performed to ensure that the preceding and following trajectory segments before and after the breakpoint complete a chain closure process under the same trajectory number, generating a continuous trajectory set for the traffic targets. Finally, the tracking state information of traffic targets around the intelligent driving vehicle is updated based on the continuous trajectory set, forming a multi-target tracking result for intelligent driving.
[0030] To verify the application effect of this method, this embodiment selected 7200 consecutive road image frames as test samples, including 21680 target annotation instances of vehicles, pedestrians, non-motorized vehicles, and road obstacles. Among them, 1426 image frames were occluded, accounting for 19.8% of the test samples; and 518 image frames had a short-term risk of missed detection, accounting for 7.2% of the test samples. In the test, conventional visual tracking methods that do not introduce road scene semantic constraint sets, occlusion phase signatures, and residual topology rewind were used as control methods, and the method of this invention was used as the implementation method. The comparison was carried out under the same road image data, the same computing platform, and the same target category range. The test results show that the method of this invention can reduce trajectory breakpoints and identity jumps in complex traffic target intersection and short-term occlusion scenarios, making the continuous trajectory set of traffic targets more stable. Especially in scenarios where pedestrians cross intersections and non-motorized vehicles run parallel, occlusion phase signatures can preserve information about the occlusion process of traffic targets, target inertial potential field inversion can maintain the temporal continuity of the set of traffic target motion states, and trace topology rollback can perform chain-like closure processing on the breakpoints of traffic target trajectories, thus making the multi-target tracking results for intelligent driving more suitable for the perception of dynamic targets around vehicles.
[0031] Table 1 Comparison of Multi-Target Tracking Performance in Complex Urban Road Scenarios
[0032] As shown in Table 1, under the same number of test image frames and target annotation instances, the traffic target detection recall and precision of the method of this invention are higher than those of conventional visual tracking methods. This indicates that the semantic constraint set of the road scene enables the improved DINO-DETR visual detection model to more effectively perform sparse target query and localization on traffic targets near the drivable space range and traffic target association boundaries, reducing interference from complex background areas. The number of identity jumps decreased from 173 to 96, and the number of trajectory breakpoints decreased from 221 to 128, indicating that the occlusion phase signature, traffic target motion state set, and cross-frame association cost matrix can improve the identity consistency of traffic targets during occlusion and intersection processes.
[0033] The method of this invention achieves a trajectory recovery success rate of 86.9% after occlusion, which is 10.5 percentage points higher than conventional visual tracking methods, indicating that residual topology rollback has a good practical effect on the chain-like closure processing of traffic target trajectory breakpoints. Although the average processing speed of this invention is reduced from 24.7 frames / second to 22.9 frames / second after introducing road scene semantic reasoning, target inertial potential field inversion, and residual topology rollback, this speed can still meet the real-time processing requirements of a 20 frames / second forward-looking camera. Therefore, this embodiment shows that the method of this invention can form a more continuous and stable set of continuous traffic target trajectories in urban road occlusion, missed detection, and multi-target intersection scenarios, and output multi-target tracking results that are more suitable for environmental perception of intelligent driving vehicles.
[0034] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A multi-target tracking method for intelligent driving based on visual intelligence, characterized in that, Includes the following steps: Collect and preprocess continuous road image frames during the driving process of intelligent driving vehicles to generate standardized image sequences of road scenes; Based on standardized image sequences of road scenes, road skeleton semantic reasoning guided by vanishing points is performed to determine lane extension direction, drivable space range and traffic target association boundary, and generate a set of semantic constraints for road scenes. The semantic constraint set of the road scene is input into the improved DINO-DETR visual detection model to perform sparse target query and localization, and a set of traffic target detection results is generated. Based on the traffic target detection result set, an occlusion phase signature of the traffic target is constructed and fused with the target appearance and target scale of the traffic target to generate a traffic target tracking feature set; Using the traffic target tracking feature set as the motion observation source, the motion attitude parameters of the traffic target in consecutive frames are determined, and the target inertial potential field inversion is performed on the traffic target to generate a set of traffic target motion states; Based on the set of traffic target motion states, a cross-frame association cost matrix of traffic targets between adjacent frames is constructed, and identity consistency matching of traffic targets is performed to generate an initial trajectory set of traffic targets. Based on the initial set of traffic target trajectories, perform residual topology rollback on the traffic target trajectory breakpoints caused by occlusion or missed detection to generate a continuous set of traffic target trajectories. Based on the continuous trajectory set of traffic targets, the tracking status information of traffic targets around intelligent driving vehicles is updated to generate multi-target tracking results for intelligent driving.
2. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, Intelligent driving vehicles acquire continuous road image frames in chronological order of vehicle travel time using onboard front-view cameras or surround-view cameras. They also read the acquisition timestamp corresponding to each continuous road image frame, perform lens distortion correction on the continuous road image frames to correct edge stretching and image curvature caused during the imaging process of the onboard camera, perform inter-frame temporal alignment according to the acquisition timestamps, perform brightness equalization, contrast enhancement, and motion blur suppression on the continuous road image frames, and standardize and regularize the continuous road image frames by unifying image size and pixel scale to generate a standardized image sequence of road scene.
3. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the road scene semantic constraint set specifically includes: Extract road direction lines from a standardized image sequence of a road scene to generate a set of road direction line elements; Based on the set of road direction line elements, perspective intersection analysis is performed to determine the road vanishing point. Then, using the road vanishing point as the perspective guide point, direction normalization is performed to generate a set of vanishing point guide direction lines. Perform road skeleton semantic reasoning on the set of vanishing point guide direction lines to generate a road skeleton semantic map; Based on the road skeleton semantic map, the lane extension direction, the drivable space range and the traffic target association boundary are determined, and a set of semantic constraints for the road scene is generated through semantic constraint aggregation.
4. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the traffic target detection result set specifically includes: The improved DINO-DETR visual detection model inputs a set of semantic constraints from the road scene. This improved model includes a road candidate localization module, a lane extension phase embedding module, a boundary adjacency query transfer module, and a joint target detection decoding module. The improvements to the traditional DINO-DETR model are as follows: the traditional model uses candidate boxes from image features, denoising queries, and iterative decoder updates as the basis for detection, predicting target categories and bounding boxes. The improved DINO-DETR model incorporates the drivable space range, traffic target association boundaries, and lane extension directions from the road scene semantic constraints into the target query generation process. It determines traffic target candidate regions using road cone projection, embeds lane direction phases using projective phase frames, introduces a boundary manifold folding mechanism to form boundary adjacency coordinates, and generates a set of traffic target detection results through query symbol layer parsing and decoding. In the road domain candidate localization module, based on the drivable space range and traffic target association boundary in the road scene semantic constraint set, the drivable space range is projected with a road domain cone strip to determine the traffic target candidate area, and sparse target query vectors are deployed in the traffic target candidate area to generate a road domain candidate query vector set. In the lane extension phase embedding module, the projective phase frame embedding is performed on the road domain candidate query vector set. Combined with the lane extension direction in the road scene semantic constraint set, the road domain candidate query vector set is mapped to the projective phase domain with the lane extension direction as the main axis to generate the lane phase query vector set. In the boundary adjacency query migration module, a boundary manifold folding mechanism is introduced to divide the traffic target associated boundary into boundary manifold fragments. Taking the phase position of the lane phase query vector set as the folding starting point, the boundary manifold fragment folding mapping is performed on the lane phase query vector set to generate the boundary adjacency query vector set. In the joint decoding module for target detection, query symbol parsing and decoding are performed on the set of boundary adjacency query vectors. The set of boundary adjacency query vectors is decomposed into category symbols, confidence symbols, and frame symbols. The traffic target detection result set is generated according to the index correspondence between the category symbols, confidence symbols, and frame symbols.
5. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the traffic target tracking feature set specifically includes: Based on the traffic target detection result set, phase division is performed on the visible area, occlusion area and boundary change area of traffic targets in continuous road image frames to determine the occlusion phase sequence corresponding to the traffic targets. Construct an occlusion phase signature based on the occlusion phase sequence; The occlusion phase signature is fused with the target appearance and scale of the traffic target, and gating capsule routing is performed according to the traffic target number and the frame number to generate a traffic target tracking feature set.
6. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the set of motion states of traffic targets specifically includes: Arrange the traffic target tracking feature set in a time sequence to generate a traffic target time sequence observation set; Based on the time-series observation set of traffic targets, the position change, target shape change and occlusion phase change of the same traffic target number in adjacent consecutive road image frames are determined, and a set of traffic target motion attitude parameters is generated. Perform target inertial potential field inversion on the set of traffic target motion attitude parameters to determine the motion vector state, target morphology state and occlusion continuation state of the same traffic target number in consecutive road image frames; The motion vector state, target shape state, and occlusion continuation state are associated and bound according to the traffic target number and the frame number to generate a set of traffic target motion states.
7. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the initial trajectory set of the traffic target specifically includes: Based on the set of motion states of traffic targets, establish a set of cross-frame candidate association pairs for traffic targets; Based on the set of cross-frame candidate association pairs of traffic targets, calculate the motion vector difference cost, target shape difference cost and occlusion continuation difference cost corresponding to each cross-frame candidate association pair of traffic targets, and generate a set of cross-frame difference costs of traffic targets. Construct a cross-frame correlation cost matrix based on the cross-frame difference cost set of traffic targets; Based on the minimum cost mutual exclusion allocation relationship between the matrix row index and matrix column index in the cross-frame association cost matrix, identity consistency matching is performed on traffic targets in adjacent consecutive road image frames to generate identity matching results. The traffic target numbers with identity consistency matching relationships in the identity matching results are connected consecutively according to their frame sequence numbers to generate the initial trajectory set of traffic targets.
8. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, The generation of the continuous trajectory set of traffic targets specifically includes: The positions between adjacent frame numbers in the initial trajectory set of traffic targets are detected to determine the set of trajectory breakpoints of traffic targets. Perform a remnant topology rollback on the set of traffic target trajectory breakpoints to generate trajectory breakpoint connection relationships; Based on the trajectory breakpoint connection relationship, the preceding and subsequent trajectory segments with continuous connection relationship are chained and closed according to their frame numbers to generate a continuous trajectory set of traffic targets.
9. The multi-target tracking method for intelligent driving based on visual intelligence according to claim 1, characterized in that, Based on the trajectory number, traffic target number, frame number, continuous trajectory chain, and missing frame position in the continuous trajectory set of traffic targets, and using the trajectory number as the tracking status update index and the frame number as the tracking status update timing field, the traffic target number in the same continuous trajectory chain is read frame by frame. When the continuous trajectory chain contains a traffic target number in the current frame number, the tracking status corresponding to the traffic target number is marked as continuous tracking status. When the continuous trajectory chain has a missing frame position corresponding to the current frame number, the tracking status corresponding to the missing frame position is marked as breakpoint closure status. When the current frame number is the frame number of the end of the continuous trajectory chain, the tracking status of the corresponding trajectory number is marked as trajectory termination status. The trajectory number, traffic target number, frame number, continuous tracking status, breakpoint closure status, and trajectory termination status are written into the tracking status information of traffic targets around the intelligent driving vehicle to generate multi-target tracking results for intelligent driving.
