Traffic scene multi-target detection method and system based on deep learning

Abstract

The invention provides a traffic scene multi-target detection method and system based on deep learning, and relates to the technical field of traffic, and the method comprises the steps: carrying out the semantic prior driven multi-scale feature extraction of a multi-frame image, and carrying out the point-by-point fusion; obtaining a motion field through optical flow estimation and feature similarity calculation, and executing motion compensation to obtain a moving target mask; obtaining a static target boundary by using boundary regression decoupling and geometric consistency constraint; and finally, moving and static target results are combined, and quadratic regression is executed based on consistency evaluation. The dynamic and static targets in the traffic scene can be effectively detected, the boundary regression precision is improved, and the false detection rate caused by shielding is reduced.

Description

Technical Field

[0001] This invention relates to the field of transportation technology, and in particular to a method and system for multi-target detection in traffic scenes based on deep learning. Background Technology

[0002] Multi-target detection technology in traffic scenarios is of great significance for applications such as intelligent transportation systems, autonomous driving, and traffic monitoring. With the development of deep learning technology, target detection methods based on deep neural networks have been widely used in traffic scene analysis. Traditional multi-target detection in traffic scenes mainly relies on single-frame image processing, extracting features through convolutional neural networks and performing target localization and classification. In recent years, researchers have begun to explore using temporal information and multimodal data fusion to improve detection accuracy, especially for the identification of moving and static targets in complex traffic scenarios.

[0003] Existing feature extraction methods typically employ single-scale or simple multi-scale feature fusion strategies, which struggle to simultaneously meet the detection needs of targets of different sizes. They perform poorly, especially in traffic scenarios where small targets at a distance and large targets at close range coexist, lacking an effective cross-scale feature fusion mechanism.

[0004] Existing technologies fail to adequately consider background interference caused by camera motion when processing dynamic targets. This makes it difficult to effectively distinguish between background motion and foreground target motion when the installation position of roadside equipment is not fixed or there is slight shaking, resulting in a large number of false detections.

[0005] Existing bounding box regression methods often treat target center localization and size estimation as a whole problem. In the case of partial occlusion, it is difficult to accurately estimate the complete boundary of the target, and there is a lack of effective mechanisms to solve the consistency problem of detection results for moving and static targets, resulting in a decline in detection performance in complex traffic scenarios. Summary of the Invention

[0006] This invention provides a method and system for multi-target detection in traffic scenes based on deep learning, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a multi-target detection method for traffic scenes based on deep learning, comprising: Semantic prior-driven multi-scale feature extraction is performed on continuous multi-frame images acquired by roadside sensing devices. A dual-path feature consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. For moving targets, the motion field is calculated based on the optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region under the influence of vehicle self-motion. The connected component aggregation is performed on the compensated residual motion response to obtain the moving target mask. For static targets, the center localization and width and height estimation are decoupled by boundary regression with context constraints on the spatiotemporal fusion features. The geometric consistency constraint is used to screen out false detections caused by instantaneous occlusion, and the boundary regression results of static targets are obtained. The regression results of the moving target mask and the static target boundary are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, a secondary regression is performed on the candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidates are then classified and their confidence is calibrated.

[0008] A dual-path feature structure consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point-by-point at each spatial location and temporal index, resulting in spatiotemporal fused features including: The resolution branch extracts boundary features from consecutive frames of images while maintaining spatial sampling density, while the context branch extracts global semantic representations while expanding the receptive range. By leveraging cross-scale guided attention weights, the resolution branch representation and the context branch representation are fused point-by-point at the same spatial coordinates and the same temporal index, forming a spatiotemporal fusion feature that includes boundary feature representation and global semantic representation.

[0009] The method further includes: The local representation of each pixel location in multiple consecutive frames of images is jointly encoded with the context representation in its adjacent range using weighted methods. The probability of the target presence and the target boundary offset corresponding to the pixel location are directly output through a continuously differentiable regression mapping. A candidate set is constructed based on the target existence probability and target boundary offset at all pixel locations. The candidate set is then aggregated across frames using similarity constraints and geometric correspondence constraints between features from multiple frames to obtain a consistent candidate sequence with temporal identifiers and spatial coordinates. Within a unified candidate sequence, the spatial overlap relationship between candidate sequences under the same time identifier is calculated, and the cross-frame correspondence relationship between candidate sequences is calculated between adjacent time identifiers. The spatial overlap relationship and the cross-frame correspondence relationship are combined to suppress redundant candidates.

[0010] For moving targets, the motion field is calculated based on optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region affected by vehicle self-motion. Connected component aggregation is performed on the compensated residual motion response to obtain the moving target mask, including: Using the correspondence between adjacent frame images in the same image coordinate system as a constraint, a pixel-level motion field is jointly established based on the correspondence between the photometric correspondence between adjacent frame images and the correspondence between the spatiotemporal fusion features, so that each spatial position and each temporal index can simultaneously obtain the displacement vector and the matching confidence. Based on the geometric correspondence formed by the static background, the homography mapping is calculated in the same image coordinate system. The homography mapping is then applied to the spatiotemporal fusion features of adjacent frame images to perform motion compensation for the global displacement caused by the vehicle's self-motion, resulting in the compensated residual motion response. On the compensated residual motion response, spatial connectivity analysis and continuity verification under temporal consistency constraints are performed according to the effective response set determined by the matching confidence. The continuous responses are aggregated into candidate connected domains and a correspondence is established between them and the displacement vectors and matching confidence of the pixel-level motion field. Boundary refinement and background consistency removal are performed on the candidate connected domains to obtain the moving target mask.

[0011] For static targets, the center localization and width / height estimation are decoupled through context-constrained boundary regression based on spatiotemporal fusion features. Geometric consistency constraints are used to filter out false detections caused by transient occlusion. The resulting static target boundary regression results include: Under the same spatial coordinates and the same time index, the spatiotemporal fusion features are decomposed into local representations and contextual representations. The local representations are obtained by mapping while maintaining spatial sampling density and edge sensitivity, and the contextual representations are obtained by aggregation mapping performed in the spatial and temporal neighborhoods. Alignment constraints are constructed based on the local and contextual representations without changing the geometric coordinates. In a coordinate system consistent with the alignment constraint, the boundary regression is divided into a center localization mapping and a width and height estimation mapping. The center localization mapping takes the local representation and alignment constraint as input to obtain the center localization result, while the width and height estimation mapping takes the context representation and alignment constraint as input to obtain the width and height estimation result. The center localization result and the width and height estimation result are combined into a static target boundary regression result under the same spatial coordinates and the same time index.

[0012] The regression results of moving target masks and static target boundaries are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation between the target bounding box and the pixel mask, a secondary regression is performed on candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidates are then classified and their confidence is calibrated, including: A closed-loop processing flow is executed for the same time index. The correspondence between the moving target mask and the boundary regression results of the static target is established in the same image coordinate system. Based on the spatial coverage relationship and the consistency of the time index, the candidates that overlap or are adjacent are logically merged to form a set of candidates to be checked. Under the same image coordinate system, a consistency evaluation based on the bounding box and pixel mask is performed on the candidate set to be checked. For candidates that are determined to be inconsistent between the boundary and the mask in the consistency evaluation, a second regression is triggered while keeping the same image coordinate system and the same time index unchanged. After the quadratic regression and the consistency evaluation converge, the final candidate is subjected to category determination and confidence calibration under the same image coordinate system and the same time index. The confidence calibration is jointly established based on the coverage consistency metric, contour deviation metric and temporal continuity stability between the final boundary and the pixel mask, so as to suppress the confidence bias caused by instantaneous occlusion and boundary drift.

[0013] A second aspect of this invention provides a multi-target detection system for traffic scenes based on deep learning, comprising: The first unit is used to perform semantic prior-driven multi-scale feature extraction on continuous multi-frame images acquired by roadside sensing devices, construct a dual-path feature consisting of resolution branch and context branch, and use cross-scale guided attention to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. The second unit is used to calculate the motion field based on the optical flow estimation and feature similarity of adjacent frames for moving targets, perform motion compensation based on homography transformation on the background region under the influence of vehicle self-motion, and perform connected component aggregation on the compensated residual motion response to obtain the moving target mask. The third unit is used to decouple the center localization and width and height estimation of static targets by using context-constrained boundary regression on the spatiotemporal fusion features, and to filter out false detections caused by instantaneous occlusion by using geometric consistency constraints, so as to obtain the static target boundary regression results. The fourth unit is used to logically merge and resolve conflicts between the dynamic target mask and the static target boundary regression results in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, it performs secondary regression on candidates with inconsistent boundaries and masks and updates their boundary positions and sizes. Finally, it performs category determination and confidence calibration on the final candidates.

[0014] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0016] The beneficial effects of this application are as follows: By leveraging semantic prior-driven multi-scale feature extraction and dual-path feature construction, combined with a cross-scale guided attention mechanism, spatiotemporal fusion features are generated. This effectively enhances the detection system's ability to perceive targets at different scales, improves the richness and robustness of feature representation, and provides more effective feature representations for subsequent detection tasks.

[0017] Differentiated processing strategies are adopted for moving and static targets. For moving targets, the motion region is accurately segmented through optical flow estimation and motion compensation techniques. For static targets, boundary regression with context constraints and geometric consistency constraints are used to eliminate false detections. This effectively solves the detection problem of complex backgrounds and variable motion states in traffic scenes and improves the detection accuracy.

[0018] By combining the logical merging and conflict resolution mechanism of dynamic and static target results with the quadratic regression technique based on the consistency of target bounding box and pixel mask, the system achieves accurate estimation of target boundary position and size. At the same time, by combining category determination and confidence calibration, the system's detection performance and reliability are significantly improved, meeting the needs of accurate detection of high-density, multi-category targets in traffic scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the multi-target detection method for traffic scenes based on deep learning, as described in an embodiment of the present invention. Figure 2 This is a flowchart of the adjacent image displacement matching process according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Figure 1 This is a flowchart illustrating the multi-target detection method for traffic scenes based on deep learning, as described in an embodiment of the present invention. Figure 1 As shown, the method includes: Semantic prior-driven multi-scale feature extraction is performed on continuous multi-frame images acquired by roadside sensing devices. A dual-path feature consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. For moving targets, the motion field is calculated based on the optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region under the influence of vehicle self-motion. The connected component aggregation is performed on the compensated residual motion response to obtain the moving target mask. For static targets, the center localization and width and height estimation are decoupled by boundary regression with context constraints on the spatiotemporal fusion features. The geometric consistency constraint is used to screen out false detections caused by instantaneous occlusion, and the boundary regression results of static targets are obtained. The regression results of the moving target mask and the static target boundary are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, a secondary regression is performed on the candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidates are then classified and their confidence is calibrated.

[0023] In one optional implementation, a dual-path feature consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point-by-point at each spatial location and temporal index, resulting in spatiotemporal fused features including: The resolution branch extracts boundary features from consecutive frames of images while maintaining spatial sampling density, while the context branch extracts global semantic representations while expanding the receptive range. By leveraging cross-scale guided attention weights, the resolution branch representation and the context branch representation are fused point-by-point at the same spatial coordinates and the same temporal index, forming a spatiotemporal fusion feature that includes boundary feature representation and global semantic representation.

[0024] In one embodiment, the input data is a sequence of consecutive multi-frame images, each frame being 1920×1080 pixels in size. For the resolution branch, the spatial resolution of the original images is preserved, and feature extraction is performed through convolutional network layers. Specifically, for each input frame, a convolutional layer with 3×3 kernels is first applied to extract low-level features, with a stride of 1 to maintain spatial sampling density. The output of the convolutional layer has 64 feature channels, followed by a non-linear transformation introduced through the ReLU activation function. Next, a residual connection structure is applied to enhance feature propagation; each residual block contains two 3×3 convolutional layers, each followed by batch normalization and ReLU activation.

[0025] To capture boundary information, an edge detection module is introduced, consisting of a Sobel operator and learnable parameters, focusing on extracting contour and texture details from the image. For temporal processing, a 3D convolution operation is employed with a kernel size of 3×3×3, where the first dimension represents the time dimension. This process handles five consecutive frames of image data to capture motion features over a short period. The final resolution branch output feature map size is maintained at the original spatial resolution of 1920×1080, while the number of feature channels is expanded to 128 to ensure accurate representation of boundary features.

[0026] For the context branch, the goal is to expand the receptive field and extract global semantic representations. The input image sequence is downsampled, and average pooling is used to reduce the spatial size to 480×270 pixels. Subsequently, a deep convolutional network is applied, containing 5 convolutional blocks, each consisting of two 3×3 convolutional layers. Max pooling layers are inserted between the convolutional layers to further reduce the feature map size.

[0027] Following the third convolutional block, dilated convolution (or attenuated convolution) is introduced, using convolutional kernels with different dilation rates (2, 4, 8, and 16) to process the feature maps in parallel, expanding the receptive field without sacrificing resolution. These feature maps obtained at different dilation rates are then fused through channel concatenation to form a multi-scale contextual representation. To further enhance global semantic understanding, an attention mechanism is applied to calculate the importance weights of each region in the feature maps, focusing on semantically salient regions. The final context branch output feature map has a size of 120×68 and 256 feature channels, containing rich global semantic information.

[0028] The cross-scale guided attention fusion module is responsible for effectively fusing features from the resolution branch and the context branch. This module first upsamples the context branch feature map to the same size (1920×1080) as the resolution branch feature map using bilinear interpolation. Then, it calculates the cross-scale guided attention weights: the upsampled context features are reduced to 64 channels using a 1×1 convolution, and then concatenated with the resolution branch features (128 channels) to obtain a 192-channel joint feature. Two convolutional layers (96 and 1 channels respectively) and a sigmoid activation function are applied to the joint feature to generate an attention weight map with values ​​ranging from 0 to 1 across the spatial range, measuring 1920×1080×1. This attention weight map guides the fusion ratio of the resolution branch features and the context branch features at each spatial location.

[0029] In the time dimension, for a sequence containing T frames (T=5 in this example), the resolution features and context features corresponding to each frame are extracted, and attention weights are independently calculated at each time index t (1≤t≤T). At each time index t and spatial location (x,y), the resolution branch feature Fr is applied according to the corresponding attention weight αt(x,y). t (x,y) and context branch feature Fc t (x,y) are fused as follows: t (x,y)=αt(x,y)·Fr t (x,y)+(1-αt(x,y))·Fc t (x,y). Among them, when the boundary features of a certain position are obvious, the attention weight αt(x,y) value is larger, and the fused feature is more inclined to retain the detailed information of the resolution branch; when it is necessary to understand the semantic context, the value of αt(x,y) is smaller, and the fused feature retains more of the global semantic information of the context branch.

[0030] The final spatiotemporal fusion feature dimension is T×1920×1080×C, where T=5 represents the time length, 1920×1080 represents the spatial resolution, and C=192 represents the number of feature channels after fusion. This fusion method can maintain high-precision spatial details in boundary regions, while making reasonable use of contextual information in flat regions, effectively balancing the needs of detail preservation and semantic understanding.

[0031] In practical applications, processing a high-definition video sequence containing 5 frames using the above method generates spatiotemporal fusion features that simultaneously preserve the fine contours of the target object and the scene context. Experiments show that when regions rich in boundary information (such as object contours) appear, the attention weight can reach above 0.85; while in background regions with simple textures, the attention weight drops to around 0.3, demonstrating that this method can adaptively adjust the fusion strategy according to the image content. The fusion features generated in this way improve the performance of object recognition and segmentation tasks by an average of 9.5% compared to single-branch methods, while reducing the boundary error rate by 27.3%.

[0032] In one optional implementation, the method further includes: The local representation of each pixel location in multiple consecutive frames of images is jointly encoded with the context representation in its adjacent range using weighted methods. The probability of the target presence and the target boundary offset corresponding to the pixel location are directly output through a continuously differentiable regression mapping. A candidate set is constructed based on the target existence probability and target boundary offset at all pixel locations. The candidate set is then aggregated across frames using similarity constraints and geometric correspondence constraints between features from multiple frames to obtain a consistent candidate sequence with temporal identifiers and spatial coordinates. Within a unified candidate sequence, the spatial overlap relationship between candidate sequences under the same time identifier is calculated, and the cross-frame correspondence relationship between candidate sequences is calculated between adjacent time identifiers. The spatial overlap relationship and the cross-frame correspondence relationship are combined to suppress redundant candidates.

[0033] Based on local and contextual representations, weighted joint encoding is performed. Specifically, for each pixel location (x, y) in the image, assuming its local representation is L(x, y) and its contextual representation is C(x, y), the system calculates a weighted sum expression by setting weight coefficients α and β: F(x, y) = α·L(x, y) + β·C(x, y), where α and β are learnable weight parameters, typically initialized to 0.5 and continuously optimized during training. This weighting method enables the system to adaptively adjust the importance of local and contextual information according to different scenarios.

[0034] The jointly encoded features are processed through a continuously differentiable regression mapping network, which consists of three fully connected layers, each containing 128, 64, and 2 neurons respectively, with ReLU as the activation function. The output of the regression mapping includes two parts: the probability of the target presence P(x,y) and the target boundary offset O(x,y). The probability of the target presence ranges from [0,1], representing the likelihood of a target being present at that pixel location; the target boundary offset is a four-dimensional vector (dx1,dy1,dx2,dy2), representing the relative offsets from the current pixel location to the top-left and bottom-right corners of the target bounding box, respectively.

[0035] For example, for a 640×480 resolution video frame, at position (320,240), if the system calculates P(320,240)=0.92 and O(320,240)=(-50,-30,45,40), it means that there is a 92% probability that a target exists at that position, and the upper left corner coordinates of the target bounding box are (270,210) and the lower right corner coordinates are (365,280).

[0036] Based on the target presence probability and boundary offset at each pixel location, a candidate set is constructed. Specific operations include: setting a probability threshold T. p (Usually 0.5), select all P(x,y)>T p The pixel positions are used to generate candidate boxes based on their corresponding boundary offsets. In practice, for a 640×480 image, thousands of candidate boxes will be generated, which constitute the initial candidate set S.

[0037] To establish associations across multiple consecutive frames, cross-frame aggregation is performed using similarity constraints and geometric correspondence constraints between features from different frames. Similarity constraints are achieved by calculating the cosine similarity of candidate box features in different frames; a cosine similarity greater than a threshold T is considered valid. s Candidate boxes with a similarity score of 0.75 (typically set to 0.75) are considered to have high similarity. Geometric correspondence constraints are evaluated by calculating the positional offset and size change of the candidate boxes in adjacent frames; if the offset distance is less than a preset threshold T... d (Typically set to 10% of the image width) and the rate of change in size is less than the threshold T. r (Usually set to 0.2), then these candidate boxes are considered to be geometrically corresponding.

[0038] For example, suppose there is a candidate box A in frame 1 located at (100,100,200,200) (representing the top left corner coordinates (100,100) and the bottom right corner coordinates (200,200)), and a candidate box B in frame 2 located at (110,105,215,210). If the feature similarity between these two candidate boxes is 0.85, the positional offset is 15 pixels, and the size change rate is 0.15, then the system will aggregate these two candidate boxes into different time instances of the same target.

[0039] Through cross-frame aggregation, the system obtains a uniform candidate sequence C={C1,C2,...,C...} with temporal identifiers and spatial coordinates. n}, each C i This indicates the same target being tracked across multiple consecutive frames. For example, C1 contains {(1,100,100,200,200),(2,110,105,215,210),(3,115,110,225,215),...}, where the first number of each element represents the frame index, and the last four numbers represent the bounding box coordinates.

[0040] After obtaining the consistent candidate sequences, further processing is performed to suppress redundant candidates. The spatial overlap between candidate sequences under the same time identifier is calculated, and the Intersection over Union (IoU) ratio is used as the metric. For any two bounding boxes, if their IoU is greater than the threshold T, then... iou If the threshold is 0.5, then they are considered to have spatial overlap. For example, in the first frame, if candidate sequences C1 and C2 contain bounding boxes (1,100,100,200,200) and (1,110,110,210,210) respectively, and the calculated IoU is 0.64, which is greater than the threshold of 0.5, then these two candidate sequences are considered to represent the same target in the first frame.

[0041] Calculate the cross-frame correspondence between adjacent time markers. Assume that between frame t and frame t+1, the candidate sequence C i and C j The strength of the correspondence is determined by their feature similarity and motion consistency. If the correspondence strength is greater than the threshold T... c (Usually set to 0.8), then C is considered to be i and C j There is a strong correspondence between these two frames.

[0042] Spatial overlap and cross-frame correspondence are combined to suppress redundant candidates. Specifically, the system constructs a relation graph G, where nodes represent candidate sequences and edges represent spatial overlap or cross-frame correspondence. Within this graph, the system iteratively selects the candidate sequence with the highest confidence and suppresses other strongly correlated candidate sequences to obtain the final detection result. The confidence level can be calculated using the average target presence probability of the candidate sequences.

[0043] Through the above steps, this method can effectively handle the problem of target detection in video, and is especially suitable for the task of detecting and tracking targets in multiple consecutive frames of images. It can significantly improve processing efficiency while ensuring detection accuracy.

[0044] In one optional implementation, for a moving target, a motion field is calculated based on optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region affected by vehicle self-motion. Connected component aggregation is performed on the compensated residual motion response to obtain a moving target mask, including: Using the correspondence between adjacent frame images in the same image coordinate system as a constraint, a pixel-level motion field is jointly established based on the correspondence between the photometric correspondence between adjacent frame images and the correspondence between the spatiotemporal fusion features, so that each spatial position and each temporal index can simultaneously obtain the displacement vector and the matching confidence. Based on the geometric correspondence formed by the static background, the homography mapping is calculated in the same image coordinate system. The homography mapping is then applied to the spatiotemporal fusion features of adjacent frame images to perform motion compensation for the global displacement caused by the vehicle's self-motion, resulting in the compensated residual motion response. On the compensated residual motion response, spatial connectivity analysis and continuity verification under temporal consistency constraints are performed according to the effective response set determined by the matching confidence. The continuous responses are aggregated into candidate connected domains and a correspondence is established between them and the displacement vectors and matching confidence of the pixel-level motion field. Boundary refinement and background consistency removal are performed on the candidate connected domains to obtain the moving target mask.

[0045] like Figure 2 As shown, the method includes: The establishment of correspondences between adjacent frames is achieved through two levels: feature point matching and dense correspondence estimation. Feature point matching uses a corner detection algorithm to extract key points in the current frame and adjacent frames. Corner detection is achieved by calculating the eigenvalues ​​of the covariance matrix of the image gradient; locations with larger eigenvalues ​​are identified as corners. For each detected corner, an image patch centered on that point is extracted as a feature descriptor, which is represented by a normalized grayscale vector. Feature point matching between adjacent frames is achieved by calculating the Euclidean distance between the descriptors; feature point pairs with the smallest distance less than a set threshold are identified as matching pairs.

[0046] The photometric correspondence is established through the assumption of constant brightness, which holds that the brightness value of the same physical point remains unchanged in adjacent frames. Based on this assumption, pixel-level photometric constraint equations are established. The calculation of the photometric constraint iterates through each pixel position in the image, calculating the brightness difference at that position in the current frame and adjacent frames. The calculation of the brightness difference considers the influence of the brightness gradient, and uses Taylor expansion to correlate brightness changes with the displacement vector. The quality of the photometric correspondence is evaluated by calculating the brightness variance within a local window; regions with larger variance have more reliable photometric constraints.

[0047] The spatiotemporal fusion feature correspondence is constructed using a feature correlation matching method. This method calculates the similarity of feature vectors at corresponding positions in adjacent frames in the feature space. The similarity calculation uses a normalized cross-correlation coefficient, obtained by calculating the inner product after normalizing the feature vectors. The establishment of feature correspondence also includes a multi-scale matching strategy, calculating feature similarity at different resolution levels and improving matching reliability through inter-scale consistency verification. The results of multi-scale matching are fused using a weighted average, with weights determined based on the feature reliability at each scale.

[0048] The pixel-level motion field is established by jointly optimizing photometric and feature constraints. This optimization process employs an iterative least squares method, solving for the displacement vector of each pixel by minimizing the weighted sum of the photometric and feature residuals. The photometric residual represents the brightness difference of the same pixel between adjacent frames, while the feature residual represents the distance of the corresponding feature vector. The weight coefficients are dynamically adjusted based on the local image gradient and feature confidence, with regions having larger gradients and higher feature confidence receiving greater weights.

[0049] The displacement vector calculation employs a hierarchical estimation strategy, which refines the displacement estimation layer by layer starting from a low resolution. The initial estimate for each layer comes from the interpolation result of the previous layer. The search range of the displacement vector is determined based on the expected target velocity. The search process uses a block matching algorithm, which determines the displacement by finding the optimal matching position within the search window. The similarity metric for block matching combines absolute difference and normalized cross-correlation coefficient to improve the robustness of the matching.

[0050] Match confidence is calculated based on a comprehensive evaluation of multiple reliability metrics, including three dimensions: uniqueness, consistency, and stability. Uniqueness is assessed by comparing the similarity difference between the best and second-best matches; a greater difference indicates a more reliable match. Consistency is evaluated by checking the degree of agreement between the displacement vector and its neighboring displacements; displacements significantly different from their neighboring motion patterns are marked as having low confidence. Stability is evaluated by analyzing displacement changes at the same location over multiple consecutive frames; smoothly changing displacements receive higher confidence scores.

[0051] The identification of geometric correspondences between static and background regions employs motion segmentation technology, which divides the image into moving foreground and static background. The identification of static background regions is based on statistical analysis of displacement vectors. The amplitude distribution of displacement vectors is obtained through histogram statistics, and pixels with amplitudes close to zero are initially labeled as static background candidates. Further verification of the static background is achieved by analyzing the spatial consistency of the displacement vectors; continuous regions with high consistency are confirmed as static background.

[0052] The homography calculation employs a random sampling consensus algorithm, which randomly selects four pairs of points from the feature point matching pairs in the static background region and calculates the corresponding homography matrix parameters. The homography matrix is ​​calculated by solving a system of linear equations based on the projection relationship of feature points between two frames. For each candidate homography matrix, its reprojection error across all matching points is calculated; matching points with a reprojection error less than a threshold are considered inliers. The homography matrix with the most inliers is selected as the final homography result.

[0053] Motion compensation is performed through the inverse operation of homography transformation. This operation geometrically corrects the images and features of adjacent frames based on the calculated homography matrix, eliminating global displacement caused by vehicle motion. Image geometric correction uses bilinear interpolation to ensure image quality. Feature geometric correction uses nearest-neighbor interpolation to preserve the discreteness of feature values. The corrected image and features are aligned with the current frame in the same coordinate system, providing a benchmark for subsequent residual calculations.

[0054] The residual motion response is calculated by comparing the differences in the motion field before and after compensation. This calculation process compares the original motion field and the compensated motion field pixel by pixel, calculating the difference in displacement vectors as the residual response. The amplitude of the residual response reflects the independence of the motion at that position; positions with larger amplitudes indicate the existence of target motion independent of the vehicle's own motion. The directional information of the residual response preserves the directional characteristics of the target motion, providing a basis for subsequent target analysis.

[0055] The determination of the effective response set is based on threshold segmentation with matching confidence. This segmentation process marks pixel locations with confidence scores higher than a preset threshold as effective responses. The selection of effective responses also considers the amplitude condition of the residual response. This dual-condition screening ensures that effective responses possess both reliable motion estimation and significant independent motion characteristics. The spatial distribution of effective responses is optimized through morphological operations to remove isolated noise points and connect adjacent response regions.

[0056] Spatial connectivity analysis employs a connected component labeling algorithm. This algorithm traverses each pixel in the valid response set, grouping spatially adjacent pixels with similar motion characteristics into the same connected component. Connectivity is determined based on the eight-neighborhood connection criterion, while also considering the similarity of the direction and magnitude of the displacement vectors. Motion feature similarity is measured by calculating the angular difference and magnitude ratio of the displacement vectors of adjacent pixels; pixels with both differences and ratios within tolerance ranges are considered similar.

[0057] Temporal consistency constraints are implemented by analyzing the evolution of connected components across consecutive frames. This constraint examines the positional and shape changes of the same connected component in adjacent time frames, and connected components whose changes conform to the motion continuity assumption are preserved. The evaluation of temporal consistency includes two aspects: position prediction and shape matching. Position prediction is based on linear extrapolation of historical motion trajectories, while shape matching is achieved by calculating the similarity of connected component contours.

[0058] Continuity verification is achieved by establishing a temporal correlation graph, which records the correspondence between connected components in different time frames. The correspondence is established based on positional overlap and motion consistency evaluation. Overlap is obtained by calculating the spatial intersection ratio of connected components in adjacent frames, and consistency is evaluated by comparing the differences in the average motion vectors of connected components. Connected components that meet the continuity condition are marked as candidate target regions.

[0059] The correspondence between candidate connected components and the motion field is established through region statistics. This statistical process calculates the mean displacement vector and mean confidence score of pixels within each connected component, serving as the motion feature description for that component. The motion features of a connected component also include the variance of the displacement vector, reflecting the degree of consistency of motion within the component. Connected components with smaller variances indicate more consistent internal motion, corresponding to the true moving target.

[0060] Boundary refinement is achieved using an active contour model, which optimizes the boundary positions of connected components through an energy minimization method. The energy function includes a boundary smoothness term and an image gradient term. The boundary smoothness term constrains the geometric regularity of the contour, preventing excessively jagged boundaries. The image gradient term drives the contour to converge towards the image edge, improving the accuracy of boundary localization. The evolution of the active contour employs a gradient descent algorithm, achieving energy minimization by iteratively updating the positions of contour points.

[0061] Background consistency removal is achieved by analyzing the similarity between connected components and the background. This analysis calculates the degree of difference between the feature distribution of pixels within a connected component and the feature distribution of the surrounding background region, with the degree of difference obtained using a statistical distance metric. Connected components with high similarity are identified as background noise and removed. Background consistency evaluation also includes texture analysis, which further validates the effectiveness of the target by comparing the texture feature differences between connected components and the background.

[0062] The moving target mask is generated by converting the thinned connected components into a binary mask. This conversion process labels pixels belonging to the moving target as foreground values ​​and the remaining pixels as background values. Post-processing of the mask includes hole filling and boundary smoothing operations. Hole filling is achieved through morphological closing operations, and boundary smoothing is achieved through Gaussian filtering. The final moving target mask maintains the same spatial resolution as the original image, providing accurate region labeling for subsequent target analysis and tracking.

[0063] In one optional implementation, for static targets, the center localization and width / height estimation are decoupled through context-constrained boundary regression based on spatiotemporal fusion features. Geometric consistency constraints are used to filter out false detections caused by transient occlusion, resulting in the following static target boundary regression results: Under the same spatial coordinates and the same time index, the spatiotemporal fusion features are decomposed into local representations and contextual representations. The local representations are obtained by mapping while maintaining spatial sampling density and edge sensitivity, and the contextual representations are obtained by aggregation mapping performed in the spatial and temporal neighborhoods. Alignment constraints are constructed based on the local and contextual representations without changing the geometric coordinates. In a coordinate system consistent with the alignment constraint, the boundary regression is divided into a center localization mapping and a width and height estimation mapping. The center localization mapping takes the local representation and alignment constraint as input to obtain the center localization result, while the width and height estimation mapping takes the context representation and alignment constraint as input to obtain the width and height estimation result. The center localization result and the width and height estimation result are combined into a static target boundary regression result under the same spatial coordinates and the same time index.

[0064] The decomposition process of spatiotemporal fusion features is implemented based on a dual-path feature extraction architecture. This architecture performs parallel decomposition operations on the input fusion feature tensor while maintaining the original spatial coordinate system and temporal index. The decomposition process is implemented through two independent convolutional branches, each using different convolutional kernel sizes and stride parameters to extract specific types of feature representations.

[0065] Local representation acquisition employs fine-grained feature extraction using small-sized convolutional kernels. This process utilizes 3x3 convolutional kernels to perform pixel-by-pixel scanning of the spatiotemporal fusion features, with the convolution stride set to one pixel unit to maintain spatial sampling density. The convolution operation padding strategy employs a boundary duplication mode to ensure that the output feature map has the same spatial size as the input feature map. Edge sensitivity is preserved by adding an edge enhancement filter after the convolutional layer. This filter, implemented using the discretization of the Laplacian operator, highlights edge information and detailed textures in the feature map.

[0066] The construction of contextual representations is achieved through spatial and temporal aggregation mapping. Spatial neighborhood aggregation employs deformable convolution, which introduces a spatial offset parameter on top of standard convolution, enabling the convolution kernel to adaptively adjust its sampling position based on local feature content. The offset parameter of deformable convolution is learned through independent convolutional branches. The input of these branches is the original spatiotemporal fusion features, and the output is a two-dimensional offset vector corresponding to the convolution kernel position. Temporal neighborhood aggregation is achieved through three-dimensional convolution, which integrates feature information from multiple consecutive frames along the time axis. The temporal dimension of the convolution kernel is set to five frames to capture the target's motion patterns and appearance changes over a short period of time.

[0067] The alignment constraint term is constructed based on feature correlation analysis between local and contextual representations. This process calculates the similarity of feature vectors of the two representations at each spatial location, using cosine distance as the similarity metric, obtained through the inner product of normalized feature vectors. The alignment constraint term is obtained by weighted fusion of local and contextual features, with the weight coefficients dynamically adjusted based on the similarity value, assigning larger fusion weights to locations with high similarity. The calculation of the constraint term maintains the geometric coordinates unchanged, and all operations are performed within the coordinate system of the original feature map.

[0068] Coordinate alignment for boundary regression is achieved through an affine transformation matrix, which ensures that center localization and width / height estimation are performed in a unified coordinate reference system. The parameters of the affine transformation matrix are obtained by analyzing the spatial distribution characteristics of the alignment constraint terms, and the transformation process includes a combination of translation, rotation, and scaling operations. The coordinate-aligned feature map maintains the same spatial resolution and channel dimensions as the original input.

[0069] The center localization mapping employs a regression network architecture. This network takes the concatenated features of local representations and alignment constraints as input and outputs the relative coordinate offset of the target center point through a multi-layer fully connected network. The network architecture consists of three hidden layers, each containing 512 neurons, with a modified linear unit function as the activation function. The output layer contains two neurons, corresponding to the coordinate offsets in the horizontal and vertical directions, respectively. The center localization result is obtained by adding the predicted offset to the initial coordinates of the candidate location.

[0070] The width and height estimation mapping employs a context-information-based size prediction network. The network's input is a feature fusion result of contextual representations and alignment constraints, and its structure uses an encoder-decoder architecture. The encoder progressively reduces the feature map size and increases the receptive field through successive convolution and pooling operations, while the decoder recovers the feature map size through deconvolution operations and outputs the size prediction result. The width and height estimation result includes the width and height values ​​of the target bounding box, which are normalized relative to preset anchor point sizes.

[0071] The combined process of center localization and width / height estimation is performed in a unified spatial coordinate system. This process uses the center localization result as the geometric center of the bounding box and the width / height estimation result as the size parameters of the bounding box, generating the complete bounding box coordinates through geometric transformation. The coordinates of the four vertices of the bounding box are obtained by adding and subtracting half the width and half the height from the center coordinates, ensuring that the geometric center of the bounding box is perfectly aligned with the predicted center position.

[0072] The geometric consistency constraint is established based on prior knowledge of the target shape. This constraint is achieved by analyzing geometric properties such as the target's aspect ratio, area range, and boundary smoothness. The aspect ratio constraint limits the aspect ratio of the predicted bounding box to a reasonable range, which is determined based on the statistical characteristics of different target categories. The area constraint ensures that the predicted target area is consistent with the expected area estimated based on contextual information. Area consistency is evaluated by comparing the relationship between the predicted area and the intensity of neighborhood features.

[0073] Transient occlusion detection is achieved by analyzing the integrity and continuity of the feature map. This detection process calculates the spatial gradient changes of local representations and locates occlusion boundaries by identifying regions of abrupt gradient changes. Occlusion detection also includes temporal consistency analysis, which determines the presence of occlusion by comparing the feature similarity of the same target region in the current frame with that in historical frames. Regions with significantly decreased similarity are marked as potential occlusion areas.

[0074] False detections are filtered using a multi-layered verification mechanism, comprising three levels: geometric rationality check, feature consistency check, and temporal stability check. The geometric rationality check evaluates whether the shape and size of the predicted bounding box conforms to the typical features of the target category; detections that do not meet the geometric constraints are marked as false detections. The feature consistency check compares the matching degree between the features inside the bounding box and the prototype features of the target category; detections with a matching degree below a threshold are filtered out. The temporal stability check analyzes the consistency of target detection across consecutive frames; detections exhibiting frequent jumps or sudden disappearances are identified as unstable detections.

[0075] The static target boundary regression results are generated through a comprehensive evaluation mechanism. This mechanism scores the quality of candidate targets selected through geometric consistency constraints, using criteria including boundary accuracy, feature matching degree, and temporal stability. Boundary accuracy is evaluated by comparing the overlap between the predicted boundary and the true target contour, with the overlap calculated using the intersection-union ratio (IU). Feature matching degree is obtained by calculating the similarity between the target region features and the category prototype features. Temporal stability is evaluated by statistically analyzing the continuity of target detection and the smoothness of positional changes in historical frames. Candidates with quality scores higher than a set threshold are confirmed as valid static target detection results, forming the final boundary regression output.

[0076] In one optional implementation, the regression results of the moving target mask and the static target boundary are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation between the target bounding box and the pixel mask, a secondary regression is performed on candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidate is then classified and its confidence is calibrated, including: A closed-loop processing flow is executed for the same time index. The correspondence between the moving target mask and the boundary regression results of the static target is established in the same image coordinate system. Based on the spatial coverage relationship and the consistency of the time index, the candidates that overlap or are adjacent are logically merged to form a set of candidates to be checked. Under the same image coordinate system, a consistency evaluation based on the bounding box and pixel mask is performed on the candidate set to be checked. For candidates that are determined to be inconsistent between the boundary and the mask in the consistency evaluation, a second regression is triggered while keeping the same image coordinate system and the same time index unchanged. After the quadratic regression and the consistency evaluation converge, the final candidate is subjected to category determination and confidence calibration under the same image coordinate system and the same time index. The confidence calibration is jointly established based on the coverage consistency metric, contour deviation metric and temporal continuity stability between the final boundary and the pixel mask, so as to suppress the confidence bias caused by instantaneous occlusion and boundary drift.

[0077] The logical merging process of moving target masks and static target boundary regression results establishes a spatial correspondence within the same image coordinate system. This process involves traversing all moving target mask regions and static target bounding boxes under the current time index, calculating the Euclidean distance between the centroid coordinates of each moving target mask and the center point of the static target bounding box. When the distance is less than a preset threshold, the proportion of the overlap area between the moving target mask and the static target bounding box to their respective total areas is further calculated. Candidate pairs with an overlap ratio exceeding a set threshold are marked as potential correspondences.

[0078] Spatial coverage relationship evaluation is achieved by calculating the inclusion relationship between pixels within the moving target mask and the bounding box of the static target. Specifically, the moving target mask is converted into a binary image, where foreground pixels have a value of 1 and background pixels have a value of 0. For each static target bounding box, the number of foreground pixels contained within it is counted, and the ratio of this number to the total number of pixels in the mask is calculated. When this ratio exceeds a coverage threshold, the moving target mask and the static target bounding box are considered to have a significant spatial relationship.

[0079] The time index consistency check ensures the temporal synchronization of moving and static targets. This check maintains a timestamp mapping table that records the generation time index of each moving target mask and static target bounding box. By comparing the differences in time indices, candidate pairs with time differences within a tolerable range are selected. The time consistency evaluation also includes verifying the trajectory continuity of candidate targets across consecutive frames, judging the reasonableness of the trajectory by calculating the amount of change in the same target position in adjacent frames.

[0080] The logical merging of overlapping candidates employs a priority-based fusion strategy, which prioritizes candidates based on their detection confidence, mask integrity, and boundary accuracy. For multiple spatially overlapping candidates, the candidate with the highest confidence is retained as the dominant candidate, and other overlapping candidates are merged into the dominant candidate. The merging process includes the union of mask regions and the calculation of bounding boxes, ensuring that the merged candidate completely covers the original multiple candidate regions.

[0081] The process of forming the candidate set to be verified is achieved by establishing a candidate index table, which records the spatial location, temporal index, category label, and confidence information of each candidate. The index table supports fast lookup based on spatial adjacency. The image space is divided into grid cells using spatial hashing technology, and each candidate is assigned to a corresponding grid cell according to its location. Candidates in adjacent grid cells are considered spatially adjacent and require further merging and evaluation.

[0082] The consistency evaluation between the bounding box and the pixel mask is achieved by calculating the geometric overlap. This evaluation process calculates the area of ​​the overlap region between the minimum bounding rectangle of the moving target mask and the static target bounding box, and compares it with the union area of ​​the two rectangles. The consistency metric reflects the degree of matching between the mask shape and the bounding box shape. When the consistency metric is below a threshold, it indicates a significant inconsistency between the bounding box and the mask.

[0083] The identification of candidates with inconsistencies between the boundary and the mask employs a shape analysis method. This method calculates the difference between the convex hull of the mask contour and the bounding box, quantifying the degree of inconsistency by analyzing the deviation between the convex hull vertices and the bounding box corners. The calculation of the degree of inconsistency includes two dimensions: positional deviation and dimensional deviation. Positional deviation is calculated through the difference in centroid coordinates, and dimensional deviation is calculated through the difference in aspect ratio.

[0084] The quadratic regression process re-optimizes the boundary parameters in the original feature space. This process uses the gradient descent algorithm to adjust the center coordinates and size parameters of the bounding box to achieve a better match with the corresponding pixel mask. The objective function of the quadratic regression combines the overlap between the bounding box and the mask, shape similarity, and positional stability constraints. The optimization process employs an adaptive learning rate strategy, dynamically adjusting the parameter update step size based on the current gradient magnitude.

[0085] The boundary position and size are updated using a constraint optimization method. This method ensures that the updated bounding box remains within the effective area of ​​the image, and that the size change remains within a reasonable range. Position updates are achieved by adding an offset to the original center coordinates, and the offset calculation takes into account the difference between the mask centroid and the bounding box center. Size updates are achieved by adjusting the length and width parameters of the bounding box, and the adjustment range is constrained by the size of the mask's bounding rectangle.

[0086] Consistency assessment and convergence determination of quadratic regression are achieved through iterative error analysis. This determination process monitors the changes in boundary parameters between consecutive iterations, and the algorithm is considered to have converged when the changes are less than a preset threshold. The convergence determination also includes a stability check of the consistency metric to ensure that the metric maintains a stable upward trend in consecutive iterations.

[0087] The category determination process makes classification decisions based on fused features. This process extracts visual features within the final bounding box and shape features of the mask region, and generates a category probability distribution through a feature fusion network. The category determination adopts the maximum probability criterion, selecting the category with the highest probability as the final category label for the target.

[0088] Confidence calibration comprehensively considers multiple evaluation dimensions to establish a calibration model. Coverage consistency metric is obtained by calculating the intersection-union ratio (IUU) of the bounding box and the mask, reflecting the accuracy of geometric matching. Contour deviation metric is obtained by analyzing the average distance between the mask boundary and the bounding box boundary, quantifying the accuracy of shape matching. Temporal continuity stability is obtained by calculating the variance of the target's position and size changes between consecutive frames, evaluating the temporal consistency of the detection results.

[0089] Transient occlusion detection is achieved by analyzing mask integrity. This detection method calculates the connectivity and compactness of the mask region, and determines the existence of occlusion by identifying voids and boundary irregularities in the mask. The occlusion detection results are used to adjust the confidence weights, reducing the confidence score of occluded targets.

[0090] Boundary drift suppression is achieved through temporal filtering, which maintains the target's boundary position sequence in historical frames and predicts the expected position in the current frame by calculating the trend of position changes. When the detected boundary position differs too much from the predicted position, a boundary drift correction mechanism is triggered, obtaining the corrected boundary position by weighted averaging of historical and current detected positions. Confidence bias suppression is achieved by establishing a dynamic threshold mechanism, adjusting the confidence evaluation criteria based on the target's historical detection stability.

[0091] A second aspect of this invention provides a multi-target detection system for traffic scenes based on deep learning, comprising: The first unit is used to perform semantic prior-driven multi-scale feature extraction on continuous multi-frame images acquired by roadside sensing devices, construct a dual-path feature consisting of resolution branch and context branch, and use cross-scale guided attention to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. The second unit is used to calculate the motion field based on the optical flow estimation and feature similarity of adjacent frames for moving targets, perform motion compensation based on homography transformation on the background region under the influence of vehicle self-motion, and perform connected component aggregation on the compensated residual motion response to obtain the moving target mask. The third unit is used to decouple the center localization and width and height estimation of static targets by using context-constrained boundary regression on the spatiotemporal fusion features, and to filter out false detections caused by instantaneous occlusion by using geometric consistency constraints, so as to obtain the static target boundary regression results. The fourth unit is used to logically merge and resolve conflicts between the dynamic target mask and the static target boundary regression results in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, it performs secondary regression on candidates with inconsistent boundaries and masks and updates their boundary positions and sizes. Finally, it performs category determination and confidence calibration on the final candidates.

[0092] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0093] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0094] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target detection method for traffic scenes based on deep learning, characterized in that, include: Semantic prior-driven multi-scale feature extraction is performed on continuous multi-frame images acquired by roadside sensing devices. A dual-path feature consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. For moving targets, the motion field is calculated based on the optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region under the influence of vehicle self-motion. The connected component aggregation is performed on the compensated residual motion response to obtain the moving target mask. For static targets, the center localization and width and height estimation are decoupled by boundary regression with context constraints on the spatiotemporal fusion features. The geometric consistency constraint is used to screen out false detections caused by instantaneous occlusion, and the boundary regression results of static targets are obtained. The regression results of the moving target mask and the static target boundary are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, a secondary regression is performed on the candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidates are then classified and their confidence is calibrated.

2. The method according to claim 1, characterized in that, A dual-path feature structure consisting of a resolution branch and a context branch is constructed. Cross-scale guided attention is used to fuse the two paths point-by-point at each spatial location and temporal index, resulting in spatiotemporal fused features including: The resolution branch extracts boundary features from consecutive frames of images while maintaining spatial sampling density, while the context branch extracts global semantic representations while expanding the receptive range. By leveraging cross-scale guided attention weights, the resolution branch representation and the context branch representation are fused point-by-point at the same spatial coordinates and the same temporal index, forming a spatiotemporal fusion feature that includes boundary feature representation and global semantic representation.

3. The method according to claim 1, characterized in that, The method further includes: The local representation of each pixel location in multiple consecutive frames of images is jointly encoded with the context representation in its adjacent range using weighted methods. The probability of the target presence and the target boundary offset corresponding to the pixel location are directly output through a continuously differentiable regression mapping. A candidate set is constructed based on the target existence probability and target boundary offset at all pixel locations. The candidate set is then aggregated across frames using similarity constraints and geometric correspondence constraints between features from multiple frames to obtain a consistent candidate sequence with temporal identifiers and spatial coordinates. Within a unified candidate sequence, the spatial overlap relationship between candidate sequences under the same time identifier is calculated, and the cross-frame correspondence relationship between candidate sequences is calculated between adjacent time identifiers. The spatial overlap relationship and the cross-frame correspondence relationship are combined to suppress redundant candidates.

4. The method according to claim 1, characterized in that, For moving targets, the motion field is calculated based on optical flow estimation and feature similarity of adjacent frames. Motion compensation based on homography transformation is performed on the background region affected by vehicle self-motion. Connected component aggregation is performed on the compensated residual motion response to obtain the moving target mask, including: Using the correspondence between adjacent frame images in the same image coordinate system as a constraint, a pixel-level motion field is jointly established based on the correspondence between the photometric correspondence between adjacent frame images and the correspondence between the spatiotemporal fusion features, so that each spatial position and each temporal index can simultaneously obtain the displacement vector and the matching confidence. Based on the geometric correspondence formed by the static background, the homography mapping is calculated in the same image coordinate system. The homography mapping is then applied to the spatiotemporal fusion features of adjacent frame images to perform motion compensation for the global displacement caused by the vehicle's self-motion, resulting in the compensated residual motion response. On the compensated residual motion response, spatial connectivity analysis and continuity verification under temporal consistency constraints are performed according to the effective response set determined by the matching confidence. The continuous responses are aggregated into candidate connected domains and a correspondence is established between them and the displacement vectors and matching confidence of the pixel-level motion field. Boundary refinement and background consistency removal are performed on the candidate connected domains to obtain the moving target mask.

5. The method according to claim 1, characterized in that, For static targets, the center localization and width / height estimation are decoupled through context-constrained boundary regression based on spatiotemporal fusion features. Geometric consistency constraints are used to filter out false detections caused by transient occlusion. The resulting static target boundary regression results include: Under the same spatial coordinates and the same time index, the spatiotemporal fusion features are decomposed into local representations and contextual representations. The local representations are obtained by mapping while maintaining spatial sampling density and edge sensitivity, and the contextual representations are obtained by aggregation mapping performed in the spatial and temporal neighborhoods. Alignment constraints are constructed based on the local and contextual representations without changing the geometric coordinates. In a coordinate system consistent with the alignment constraint, the boundary regression is divided into a center localization mapping and a width and height estimation mapping. The center localization mapping takes the local representation and alignment constraint as input to obtain the center localization result, while the width and height estimation mapping takes the context representation and alignment constraint as input to obtain the width and height estimation result. The center localization result and the width and height estimation result are combined into a static target boundary regression result under the same spatial coordinates and the same time index.

6. The method according to claim 1, characterized in that, The regression results of the moving target mask and the static target boundary are logically merged and conflict resolved in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, a secondary regression is performed on the candidates with inconsistent boundaries and masks, and their boundary positions and sizes are updated. The final candidate category determination and confidence level calibration include: A closed-loop processing flow is executed for the same time index. The correspondence between the moving target mask and the boundary regression results of the static target is established in the same image coordinate system. Based on the spatial coverage relationship and the consistency of the time index, the candidates that overlap or are adjacent are logically merged to form a set of candidates to be checked. Under the same image coordinate system, a consistency evaluation based on the bounding box and pixel mask is performed on the candidate set to be checked. For candidates that are determined to be inconsistent between the boundary and the mask in the consistency evaluation, a second regression is triggered while keeping the same image coordinate system and the same time index unchanged. After the quadratic regression and the consistency evaluation converge, the final candidate is subjected to category determination and confidence calibration under the same image coordinate system and the same time index. The confidence calibration is jointly established based on the coverage consistency metric, contour deviation metric and temporal continuity stability between the final boundary and the pixel mask, so as to suppress the confidence bias caused by instantaneous occlusion and boundary drift.

7. A multi-target detection system for traffic scenes based on deep learning, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to perform semantic prior-driven multi-scale feature extraction on continuous multi-frame images acquired by roadside sensing devices, construct a dual-path feature consisting of resolution branch and context branch, and use cross-scale guided attention to fuse the two paths point by point at each spatial location and time index to obtain spatiotemporal fusion features. The second unit is used to calculate the motion field based on the optical flow estimation and feature similarity of adjacent frames for moving targets, perform motion compensation based on homography transformation on the background region under the influence of vehicle self-motion, and perform connected component aggregation on the compensated residual motion response to obtain the moving target mask. The third unit is used to decouple the center localization and width and height estimation of static targets by using context-constrained boundary regression on the spatiotemporal fusion features, and to filter out false detections caused by instantaneous occlusion by using geometric consistency constraints, so as to obtain the static target boundary regression results. The fourth unit is used to logically merge and resolve conflicts between the dynamic target mask and the static target boundary regression results in the same coordinate system. Based on the consistency evaluation of the target bounding box and the pixel mask, it performs secondary regression on candidates with inconsistent boundaries and masks and updates their boundary positions and sizes. Finally, it performs category determination and confidence calibration on the final candidates.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images