Transform-based scale adaptive pedestrian detection method and device

Abstract

The invention discloses a scale adaptive pedestrian detection method and device based on Transform, and solves the problems of small-scale target leak detection, insufficient occlusion target feature representation and low label distribution discrimination in a dense pedestrian scene in an existing Transform detection method. The method comprises a training and reasoning stage, wherein in the training stage, data preprocessing, multi-scale feature extraction, self-adaptive screening, fractional-scale Transform coding, feature enhancement, decoding prediction and label distribution and loss calculation based on shape consistency measurement are executed; and the reasoning stage simplifies the forward calculation process and outputs a detection result. The core improvement lies in designing a multi-scale feature adaptive screening network, a scale-division coding mechanism and a feature enhancement network, and introducing shape consistency measurement in Hungary matching. The method improves the detection precision of multi-scale and shielded pedestrians, accelerates model convergence, adapts to scenes of intelligent monitoring, automatic driving and the like, and is superior to an existing mainstream method in detection performance.

Description

Technical Field

[0001] This invention belongs to the field of pedestrian detection and processing, specifically relating to a scale-adaptive pedestrian detection method and apparatus based on Transformer. Background Technology

[0002] Pedestrian detection, a fundamental object detection task in the field of computer vision, aims to automatically identify the locations of all pedestrians in an image or video sequence and accurately label them with bounding boxes. Its technological achievements are widely applied in key areas such as intelligent surveillance, public safety, and autonomous driving. However, with the development of the social economy and the increasing richness of public life, highly crowded scenes such as train stations, shopping malls, and large event venues are becoming more common, posing a serious challenge to pedestrian detection technology.

[0003] In such complex and dense scenes, pedestrian targets exhibit significant multi-scale variations, severe mutual occlusion, and high degree of obfuscation with the background. Multi-scale variations manifest as a huge size difference between near and far pedestrians in the same image; severe occlusion leads to incomplete target feature information; and background obfuscation further increases the difficulty of target differentiation. These factors together make traditional detection algorithms prone to false positives and false negatives, severely restricting the reliability and practicality of related applications. Therefore, achieving accurate and rapid pedestrian target detection in complex and dense environments has become an urgent technical challenge to be solved.

[0004] From a technological evolution perspective, traditional pedestrian detection methods primarily rely on detection frameworks based on convolutional neural networks (CNNs), such as Faster R-CNN and the YOLO series. While these methods have achieved some success in general object detection, they have significant limitations in dense pedestrian scenes. Their detection performance is highly dependent on manually pre-designed anchor boxes, making it difficult to adapt to extreme scale variations and overlapping target distributions in dense scenes. Furthermore, non-maximum suppression (NMS) in the post-processing step can easily erroneously suppress correct detection results when targets are highly overlapping, further affecting detection accuracy.

[0005] To address the shortcomings of traditional methods, Facebook AI Research (FAIR) proposed the DETR (Detection Transformer) framework based on the Transformer architecture in 2020, pioneering a new paradigm for end-to-end object detection. DETR treats object detection as an ensemble prediction problem through a Transformer encoder-decoder structure, directly outputting the prediction results. It abandons the traditional manually designed anchor boxes and non-maximum suppression post-processing steps, and its global attention mechanism and ensemble prediction approach are better suited for handling occlusion problems. However, the standard DETR and its early variants still have significant shortcomings in dense pedestrian scenes: the attention mechanism has weak perception of small-scale target features, leading to missed detections of small-scale pedestrians; the model training convergence speed is slow, affecting the efficiency of practical deployment; and its global matching strategy based on the Hungarian algorithm relies only on location and category information, resulting in insufficient discrimination for densely overlapping targets and limited label assignment accuracy, making it difficult to meet the detection needs in complex scenes. Summary of the Invention

[0006] To address this, the present invention provides a scale-adaptive pedestrian detection method and apparatus based on Transformer, which solves the problems of existing Transformer-based detection methods in dense pedestrian scenes, such as missed detection of small-scale targets, insufficient feature representation of occluded targets, and low label assignment discrimination.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a scale-adaptive pedestrian detection method based on Transformer, comprising the following steps: Data preprocessing: Perform augmentation operations on the image data in the training set; Multi-scale feature extraction: A backbone network is used to extract features from the input image after enhancement. The output feature maps of the backbone network at a specified stage are selected to form a multi-scale feature pyramid with a set spatial resolution and semantic level. Multi-scale feature adaptive filtering: The feature map of the multi-scale feature pyramid is processed by an adaptive filtering network, the feature map of different scales is divided into image blocks and flattened into feature sequences. After capturing the context dependency, the filtering weight of the image blocks is generated. Important image blocks are filtered and retained according to the weight, forming a set of filtered multi-scale feature blocks. Multi-scale Transformer encoding: The filtered multi-scale feature block set is flattened and its position is encoded. The features are grouped according to their original scale, and each group of features is input into an independent Transformer encoder for feature encoding. The encoded features of the corresponding scale are output. Feature enhancement network processing: The encoded features are reconstructed into a feature map, which is then input into the feature enhancement network. Through multi-resolution feature capture, fusion, and feature repair operations, the enhanced feature map is output. Transformer Decoding and Prediction: The enhanced feature map is input into the Transformer decoder. The decoder interacts with the encoded features through the object query vector and outputs the pedestrian prediction category and bounding box.

[0008] As a preferred embodiment of the Transformer-based scale-adaptive pedestrian detection method, the enhancement operations include random horizontal flipping, random cropping, and scaling.

[0009] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the backbone network is a ResNet network, and the output feature maps of the last three stages of the ResNet network are selected to construct a multi-scale feature pyramid.

[0010] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the adaptive filtering network includes a two-layer Long Short-Term Memory (LSTM) network and a fully connected layer, and the context dependency capture process satisfies the formula:

[0011] The formula for the selection weight of the generated image patches is:

[0012] In the formula, represents the time step of the first LSTM layer. The hidden state, This indicates that the second LSTM layer is at time step The hidden state, This represents the hidden state of the concatenation of two LSTM layers. ; The scalar value is the output of the fully connected layer. This is the weight matrix of the fully connected layer. Bias for fully connected layers; For the first The selection weight of each image patch; For the corresponding number k The scalar values ​​obtained after mapping image patches through a fully connected layer k It is the time step index of the feature sequence.

[0013] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the location encoding is the same as the location encoding method in the DETR framework, and the fusion of the flattened vector and the location encoding satisfies the formula:

[0014] In the formula, This is the set of multi-scale feature blocks after filtering. The vector obtained by flattening the filtered multi-scale feature block set; for height, for width, Number of channels; This is the position encoding matrix; The encoder input vector; This is the encoded feature for the corresponding scale in the final output.

[0015] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the feature enhancement network processes the following steps: First, the dimensions of the encoded features are reconstructed to obtain the reconstructed feature map; then, a convolution operation is performed on the reconstructed feature map to generate the initial features. The spatial resolution of the feature map is gradually reduced through multi-layer downsampling operations; residual bottleneck calculation is performed on the feature map after the last layer of downsampling; the spatial resolution of the feature map is gradually restored through multi-layer upsampling operations, and the top-level upsampling result is directly fused with the downsampling feature of the corresponding layer; the enhanced feature map is obtained through the fused feature map.

[0016] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the output process of the Transformer decoder is as follows: The scaling dot product attention mechanism is used for calculation, and the similarity score between the object query vector and the enhanced feature map is obtained by dot product. Scaling is achieved by dividing the similarity score by the square root of the dimensions of the query vector and the key vector; The scores are converted into attention weights in the form of a probability distribution through normalization. Finally, the attention weights are weighted and summed on the enhanced feature map to obtain the attention calculation result. The attention calculation results are input into the category prediction branch and the bounding box prediction branch, respectively. The category prediction branch outputs the pedestrian prediction category probability through an activation function, and the bounding box prediction branch outputs the predicted bounding box coordinates through a feedforward network.

[0017] As a preferred option for scale-adaptive pedestrian detection methods based on Transformer, the following also include: Label assignment and loss calculation based on shape consistency metric: A shape consistency metric is introduced into the Hungarian matching algorithm. A matching cost matrix is ​​constructed by combining the classification cost and the bounding box cost. The optimal matching between the predicted box and the ground truth box is achieved through the Hungarian algorithm. The total loss is calculated and the model parameters are iteratively optimized through backpropagation.

[0018] As a preferred scheme for the scale-adaptive pedestrian detection method based on Transformer, the calculation of the matching cost matrix satisfies the following formula:

[0019] In the formula, For the first i The prediction box and the first j The classification cost of a single ground truth bounding box. For bounding box cost, As a measure of shape consistency, M This represents the number of real pedestrian targets in the image. j>M The time indicates that the predicted bounding box matches an empty target.

[0020] The present invention also provides a scale-adaptive pedestrian detection device based on Transformer, employing the above-mentioned scale-adaptive pedestrian detection method based on Transformer, including: The data preprocessing module is used to perform augmentation operations on the image data in the training set; The multi-scale feature extraction module is used to extract features from the input image after enhancement using a backbone network, and select the output feature map of the backbone network at a specified stage to form a multi-scale feature pyramid with a set spatial resolution and semantic level. The multi-scale feature adaptive filtering module is used to process the feature map of the multi-scale feature pyramid through an adaptive filtering network, divide the feature map of different scales into image blocks and flatten them into feature sequences. After capturing the context dependency, the filtering weights of the image blocks are generated, and important image blocks are filtered and retained according to the weights to form a set of filtered multi-scale feature blocks. The scale-based Transformer encoding module is used to flatten and encode the selected multi-scale feature block set, group the features according to their original scale, input each group of features into an independent Transformer encoder for feature encoding, and output the encoded features at the corresponding scale. The feature enhancement network processing module is used to reconstruct the encoded features into a feature map, input it into the feature enhancement network, and output the enhanced feature map through multi-resolution feature capture, fusion and feature repair operations. The Transformer decoding and prediction module is used to input the enhanced feature map into the Transformer decoder. The decoder interacts with the encoded features through the object query vector and outputs the pedestrian prediction category and bounding box.

[0021] The present invention has the following advantages: First, this invention dynamically focuses on effective target features through multi-scale feature adaptive screening and multi-scale encoding mechanism, avoids interference from target features of different scales, significantly reduces the false detection rate of pedestrians at extreme scales, and adapts to the needs of pedestrians with large size differences in dense scenes.

[0022] Second, the feature enhancement network of the present invention reconstructs the complete feature representation of occluded pedestrians through multi-resolution feature fusion and detail restoration, significantly improving the detection recall rate in severely occluded scenes and improving the recognition problem caused by blurred target features.

[0023] Third, this invention reduces computational load by filtering redundant information through an adaptive filtering network, and accelerates model training convergence by combining targeted modeling with scaled encoding, while ensuring detection speed during the inference stage, thus balancing accuracy and practicality.

[0024] Fourth, this invention is designed for complex scenarios with dense pedestrian traffic, severe occlusion, and varying scales. It has been verified that its average accuracy and false negative rate are superior to existing mainstream methods. It can be stably applied to key areas such as intelligent monitoring and public safety, and has broad practical value. Attached Figure Description

[0025] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0026] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0027] Figure 1 This is a flowchart of an existing Deformable DETR provided in an embodiment of the present invention; Figure 2This is a flowchart of the training phase of the scale-adaptive pedestrian detection method based on Transformer provided in this embodiment of the invention. Figure 3 This is a diagram of the adaptive filtering network architecture of the scale-adaptive pedestrian detection method based on Transformer provided in this embodiment of the invention. Figure 4 This is a diagram of the scale-adaptive pedestrian detection method based on Transformer provided in this embodiment of the invention; Figure 5 This is a feature enhancement network architecture diagram of the scale-adaptive pedestrian detection method based on Transformer provided in this embodiment of the invention; Figure 6 This is a schematic diagram of the scale-adaptive pedestrian detection device architecture based on Transformer provided in an embodiment of the present invention. Detailed Implementation

[0028] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] See Figure 1 The closest existing technical solution to this invention is the "Deformable DETR Detection Model." This solution is an optimization of the DETR framework for dense scene detection. Its core lies in replacing the computationally intensive global attention with a deformable attention module to more efficiently handle multi-scale and occluded targets. Its method steps are as follows: Step 1: Multi-scale feature map extraction and fusion: First, a convolutional neural network (CNN) backbone is used to process the input image, extracting feature maps from the outputs of the last three stages to form a multi-scale feature pyramid. These feature maps are then processed through a 1x1 convolutional layer to unify the number of channels, added to their corresponding positional codes, and fed together into the Transformer encoder. This step provides the model with rich multi-scale features covering everything from shallow details to deep semantics.

[0030] Step 2: Encoding and Decoding Based on Deformable Attention: This step is the core innovation of Deformable DETR and is integrated into the operations of both the encoder and decoder.

[0031] In the encoder: each feature point is treated as a query, and its attention computation no longer covers all spatial locations. For each query, the model predicts a set (e.g., 4) of reference point offsets adapted to its content via a light quantum network. Subsequently, the attention operation samples and weights these offsets only on a few key feature points. This achieves sparse and efficient modeling of image spatial information.

[0032] In the decoder: a fixed set of learnable object queries interacts with the multi-scale features output by the encoder. Each object query also predicts a set of reference point offsets, and features are aggregated only from these dynamically determined key regions that may contain the target. This mechanism allows the decoder to focus directly and efficiently on potential object regions.

[0033] Step 3: Two-stage refined forecasting: Deformable DETR typically employs a two-stage prediction architecture to improve positioning accuracy: The first stage, region proposal generation: The decoder's output, which is the interactive object query, first generates a set of coarse bounding box proposals and corresponding class scores through a prediction head. These proposals serve as preliminary localizations of potential targets in the image.

[0034] The second stage, feature refinement and final prediction, involves mapping each bounding box proposal region generated in the first stage back to its corresponding multi-scale encoder features. A deformable attention module then extracts refined region features. These refined features are fed into another prediction head, ultimately outputting accurate bounding box coordinates and class labels.

[0035] Step 4: Bipartite graph matching and loss calculation based on the Hungarian algorithm: The model's final predicted output (an unordered set) is optimally matched one-to-one with the ground truth image annotations (another unordered set) using the Hungarian algorithm. The matching strategy is typically defined by the bounding box coordinate errors (specifically, L1 loss and GIoU loss) and the class prediction error. The overall loss is calculated based on this matching result, and this loss is used to supervise the end-to-end training of the entire model.

[0036] Starting from the Deformable DETR technology solution itself, its inherent technical feature selection leads to the following defects: First, because the core module of Deformable DETR—the deformable attention mechanism—is designed to achieve efficient sparse modeling by predicting a small number of reference points, its receptive field and feature aggregation range are highly dependent on the initial query content and the learned offset prediction model. Furthermore, when faced with extreme and dynamic multi-scale targets in dense pedestrian scenes (such as large-scale pedestrians in the foreground and small-scale pedestrians in the background), the mechanism lacks a global, adaptive scale filtering capability. Its reference point sampling may not be able to allocate appropriate attention regions for targets with vastly different scales, easily leading to insufficient feature coverage or defocusing of small-scale or ultra-large-scale targets. Therefore, the model's feature extraction capability for multi-scale pedestrians is uneven, and its robustness in simultaneously and accurately detecting extremely large and extremely small pedestrians within a single image is limited, resulting in missed detections or inaccurate localization at these extreme scales.

[0037] Secondly, because the attention mechanism of Deformable DETR focuses on local keypoints, the expressive power of the visible local features on which the model relies is insufficient when pedestrian targets are severely occluded. Consequently, in occluded areas, the overall semantic and contour information of the target becomes blurred, making it difficult to reconstruct a reasonable representation of the occluded portion relying solely on local keypoint features. Therefore, the model has a weak ability to represent the features of severely occluded pedestrians, making it difficult to recover complete and robust target features from occlusion interference, thus leading to a decrease in the recall rate for detecting occluded pedestrians, i.e., an increase in the false negative rate.

[0038] Finally, Deformable DETR inherits and adopts the bipartite graph matching strategy based on the Hungarian algorithm from the standard DETR framework. Its matching cost function typically only integrates bounding box coordinate errors (such as class loss and IoU loss) and class prediction confidence, ignoring the target's own morphological features (such as shape and contour structure). Consequently, in scenarios with dense pedestrian traffic and highly overlapping or even nearly overlapping bounding boxes, it is difficult to make accurate and unambiguous one-to-one matches between multiple similar candidate boxes or overlapping targets based solely on position and class scores. Incorrect label assignment can mislead the model's optimization direction. Therefore, the accuracy of label assignment during training is a bottleneck, especially in high-density overlapping regions. This limits the upper limit of model learning and may introduce more mismatches (misassigning a predicted box to another real target).

[0039] In view of this, to address the problems of insufficient multi-scale perception and adaptation capabilities, weakened feature representation under severe occlusion, and inaccurate assignment due to ignoring shape in matching strategies, this invention designs a multi-scale feature adaptive filtering network. This network enables the model to dynamically perceive and coordinate target information at different scales in an image, thereby improving the model's robustness to scale changes and reducing related false negatives and negatives. A feature enhancement network is constructed, focusing on strengthening the features encoded by the Transformer, effectively improving the feature extraction quality of the model in heavily occluded scenarios and increasing the recall rate for detecting occluded pedestrians. Existing label matching strategies are improved by introducing a metric that reflects the consistency of target shape, which is then used in conjunction with location and category information to optimize the matching process, thereby achieving more accurate and stable label assignment during the training phase. The following are the specific details of the embodiments of this invention. Example 1

[0040] See Figure 2 Embodiment 1 of this invention provides a scale-adaptive pedestrian detection method based on Transformer, which mainly includes two stages: model training and inference. During training, the input data is first preprocessed, and then a multi-scale feature extraction network is used to obtain feature representations at different levels. Based on this, an adaptive filtering network is introduced to effectively perceive changes at different scales and filter redundant information. A multi-scale Transformer encoder is used to specifically model features at different scales. A feature enhancement network further integrates and strengthens key information, and then a Transformer decoder generates the detection results. The training process also includes a shape consistency measurement label allocation strategy to improve the localization accuracy of the detection boxes. This method can adaptively handle multi-scale pedestrian targets and achieve robust and accurate detection performance in complex scenes. The training process specifically includes the following steps: Step S1: Data Preprocessing. Enhancement operations are performed on the image data in the training set, including random horizontal flipping, random cropping, and scaling, to improve the model's generalization ability.

[0041] Step S2: Multi-scale feature extraction. Using a ResNet network as the backbone, features are extracted from the input image. The output feature maps from the last three stages are selected to construct a multi-scale feature pyramid with different spatial resolutions and semantic levels, which serves as the input for subsequent modules.

[0042] Step S3: Adaptive Multi-Scale Feature Selection. To address the problem of heavy computational load and long training time caused by directly using all multi-scale features, an adaptive selection network consisting of a two-layer Long Short-Term Memory (LSTM) network and fully connected layers is designed, such as... Figure 3 As shown. The specific calculation process is as follows: First, feature maps of different scales are divided into regularly arranged image patches, and then flattened into a sequence. ,in For sequence length, It is the first Image patches The sequence is then fed into a two-layer LSTM to capture its cross-scale contextual dependencies. It first passes through the first LSTM layer (see formula (1), then through the second LSTM layer (see formula (2), and finally the vectors are concatenated (see formula (3)). (1) (2) (3) In the formula, This indicates that the first LSTM layer is at time step The hidden state, This indicates that the second LSTM layer is at time step The hidden state, This indicates the hidden state of the concatenation of two LSTM layers. .

[0043] The final hidden state of LSTM The calculation is shown in formula (4) via mapping through the fully connected layer: (4) In the formula, The scalar value is the output of the fully connected layer. This is the weight matrix of the fully connected layer. This is used for biasing the fully connected layer.

[0044] Then, a set of screening weights is generated using the Softmax function, and the calculation is shown in formula (5): (5) Finally, the importance of image patches at each scale is evaluated based on their weights, and the patches with the highest weights are retained. n Each image patch is used to form a filtered multi-scale feature block set. .

[0045] Step S4: Scaled Transformer encoding. For example... Figure 4 As shown, the first step is to process the filtered multi-scale feature block set. Flattening yields the vector The calculation is shown in formula (6): (6) In the formula, is for flattening calculation. for height, for width, This represents the number of channels.

[0046] (7) The position code is calculated based on the height and width, as shown in formula (7), where the position code is the same as the position code in DETR, resulting in the position code matrix. Then, the encoder input vector is obtained by summing the flattened vector with the vector according to formula (8).

[0047] (8) Will The features are grouped according to their original scale. Each scale is input in parallel into an independent Transformer encoder for feature encoding. Each encoder uses a standard structure and is calculated according to formula (9), ultimately outputting the encoded features of the corresponding scale. .

[0048] (9) In the formula, It is a standard Transformer encoder. This design allows features at different scales to be context-modeled at their appropriate levels of abstraction, enhancing scale-specific representation.

[0049] Step S5: Feature enhancement network processing. During the feature enhancement network processing: First, the dimensions of the encoded features are reconstructed to obtain the reconstructed feature map; then, a convolution operation is performed on the reconstructed feature map to generate the initial features. The spatial resolution of the feature map is gradually reduced through multi-layer downsampling operations; residual bottleneck calculation is performed on the feature map after the last layer of downsampling; the spatial resolution of the feature map is gradually restored through multi-layer upsampling operations, and the top-level upsampling result is directly fused with the downsampling feature of the corresponding layer; the enhanced feature map is obtained through the fused feature map.

[0050] Specifically, to alleviate the target feature degradation problem caused by severe occlusion, a feature enhancement network is designed. Its computation process is as follows: First, the encoder output is... Reconstruction Feature Map The calculation formula is shown in (10): (10) In the formula, It involves rearranging the element structure without changing the data content, so that... The dimensions and shape are adapted to the input requirements of subsequent processing steps.

[0051] Then input into the feature enhancement network, such as... Figure 5 As shown, this network captures and fuses contextual information at different resolutions through a repeated downsampling-upsampling structure. Let the network depth be... The feature map is downsampled; the specific calculation process is shown in formula (11-13). (11) (12) (13) In the formula, , It is feature convolution. It is downsampling. Bottleneck processing is applied, see formula (14): (14) In the formula, The process involves calculating the residual bottleneck and then upsampling. The specific calculation process is shown in formula (15-17).

[0052] (15) (16) (17) In the formula, , This is an upsampling calculation. The final output of the feature enhancement component is calculated using formula (18).

[0053] (18) Secondly, skip connections are used to combine shallow detail features and deep semantic features at the same level; finally, the enhanced feature map is output. This feature map retains richer global structural information and local details, which helps to reconstruct the features of the occluded parts.

[0054] Step S6: Transformer Decoding and Prediction. First, a scaled dot product attention mechanism is used for calculation. The similarity score between the object query vector and the enhanced feature map is obtained through a dot product. Then, this score is scaled by dividing by the square root of the dimensions of the query vector and the key vector to avoid gradient vanishing. Next, the score is converted into a probability distribution of attention weights through normalization. Finally, these weights are used to perform a weighted summation of the enhanced feature map to obtain the attention calculation result. This attention calculation result is then input into the category prediction branch and the bounding box prediction branch, respectively. The category prediction branch outputs the pedestrian prediction category probability through an activation function, while the bounding box prediction branch outputs the predicted bounding box coordinates through a feedforward network.

[0055] Specifically, the enhanced features Input the Transformer decoder. The decoder interacts with a set of learnable object query vectors and encoded features, and is responsible for transforming the encoded information into pedestrian detection results. The calculation formula is shown in (19). (19) In the formula, , , For attention calculation, This is for calculation of the feedforward network.

[0056] The decoder utilizes the feature information in the enhanced coded representation and, through a series of calculation and reasoning processes, finally outputs the pedestrian detection result. The calculation is shown in formulas (20) and (21).

[0057] (20) (twenty one) in, To predict the category, i.e. whether it is a pedestrian. This is the bounding box of the pedestrian. and These are the category and bounding box weight matrices, respectively. Calculate the sigmoid function.

[0058] Step S7: Label assignment and loss calculation based on shape consistency metric. To improve the accuracy of label matching in dense scenes, a shape consistency metric is introduced into the Hungarian matching algorithm. The calculation process is as follows: First, the core of the Hungarian algorithm is that it is about to... The prediction matched to Find 5 true targets that minimize the total matching cost. Let the cost matrix be . Then find a permutation This satisfies formula (22): (twenty two) Then, the classification cost and bounding box cost between the predicted bounding box and the ground truth bounding box are calculated. The specific calculation is shown in formula (23): (twenty three) Due to the inherent complexity of dense pedestrian detection scenarios, particularly the challenge of highly overlapping bounding boxes, traditional matching criteria based on spatial distance or Intersection over Union (IOU) fall short, failing to accurately distinguish between pedestrians who are spatially adjacent but essentially represent different individuals. Therefore, this invention proposes a shape consistency metric. See Formula (24), as one of the cost calculations, to address the problem of inaccurate allocation caused by the matching strategy ignoring shape.

[0059] (twenty four) Specifically, the shape consistency measure comprehensively reflects the predicted box by weighting and fusing three geometric indicators. With real frame Consistency across dimensions. The size of the predicted bounding box is its length. With width The actual frame size corresponds to the length. With width The three indicators are defined as follows: Used to assess the similarity of two frames in terms of length and width differences; Used to assess the closeness of the length and width; This is used to assess the similarity of the areas of the two frames. , and The weight coefficients corresponding to the above three items are summed to 1, thus balancing the contribution of each geometric factor to the overall similarity.

[0060] The final cost calculation formula is shown in (25).

[0061] (25) The total loss function is calculated based on the matching result, and all parameters of the model are iteratively optimized through backpropagation.

[0062] In this embodiment, the inference process is basically the same as the forward computation part of the training process described above, but it does not require data augmentation (only standardization), label matching, and loss calculation steps. The input image to be detected undergoes feature extraction, multi-scale filtering, encoding, augmentation, decoding, and prediction head processing in sequence, directly outputting the final detection bounding box and class confidence score. The final detection result can be obtained through threshold filtering.

[0063] To verify the effectiveness of the method of the present invention, it was validated on the WiderPerson validation set, and the validation results are shown in Table 1.

[0064] Table 1 WiderPerson Validation Results

[0065] As shown in Table 1, the WiderPerson validation results demonstrate that the proposed method achieves significant performance advantages. In terms of average precision (AP), the proposed method achieves 60.5%, significantly higher than Deformable-DETR's 55.6% and the original DETR's 45.2%, exhibiting higher detection accuracy. In terms of log-mean false negative (MR-2), the proposed method reduces the MR-2 to 55.2%, far lower than Deformable-DETR's 60.3% and the original DETR's 75.2%, indicating that it effectively improves target recall. The results validate that the scale-adaptive mechanism and consistency optimization strategy employed in this invention can more effectively handle multi-scale variations of pedestrians in complex scenes.

[0066] The application scenarios of this invention are as follows: Public safety and security scenarios: Security monitoring in key areas: Applicable to security checkpoints and waiting halls of airports, high-speed rail stations, and subway stations, etc. It can identify pedestrians of different scales (such as people queuing nearby and passengers walking in the distance) as well as targets obscured by luggage or others, and output the location and movement trajectory of pedestrians in real time to assist security personnel in quickly screening suspicious persons and preventing emergencies such as stampedes and illegal intrusions.

[0067] Security dispatch for large-scale events: For large gatherings such as concerts, sporting events, and festival celebrations, it can continuously monitor the distribution density and movement trends of dense crowds within the venue, automatically identify risky behaviors such as overcrowding and abnormal collisions, provide real-time data support for on-site security command, help accurately dispatch police forces, delineate evacuation routes, and ensure the safe and orderly conduct of events.

[0068] Border and port control: In border and port clearance areas, it can adapt to the pedestrian detection needs in complex environments, distinguish between normal clearance personnel and suspicious personnel, and can stably output test results even in crowded situations where some bodies are obscured, assisting border guards in improving inspection efficiency and control accuracy.

[0069] Intelligent transportation and autonomous driving scenarios: Pedestrian detection on urban roads: Adapted to scenarios such as urban intersections, zebra crossings, and bus stops, it can accurately detect pedestrians nearby, small-scale pedestrians crossing the road at a distance, and pedestrian targets partially obscured by vehicles or guardrails. It provides accurate pedestrian location and motion status information for autonomous vehicles or driver assistance systems, triggering safety decisions such as deceleration and avoidance, and reducing the risk of traffic accidents.

[0070] Highway emergency detection: In areas with high pedestrian traffic, such as highway service areas and toll stations, it can quickly identify targets such as pedestrians and maintenance personnel crossing the lane. Even at night or in low visibility conditions such as rain or snow, it can improve detection accuracy by enhancing feature representation, providing dual protection for road monitoring systems and vehicle safety systems.

[0071] Safety monitoring for shared mobility: In-vehicle monitoring systems applied to ride-hailing and car-sharing vehicles can detect the status of passengers inside the vehicle (such as whether there are children traveling alone or whether the vehicle is overloaded) and the dynamics of pedestrians outside the vehicle (such as the distribution of pedestrians around the vehicle when getting on and off), helping the platform and drivers avoid safety risks, and providing data support for dispute tracing.

[0072] Smart city management scenarios: Traffic flow control in densely populated areas: For urban pedestrian streets, commercial plazas, scenic spots and other places, the number of pedestrians, dwell time and flow path in different areas can be counted in real time to generate traffic flow heat maps. This provides data basis for urban management departments to optimize traffic signal timing, adjust public transportation capacity and delineate temporary control areas, thereby alleviating traffic congestion and crowding.

[0073] Municipal facility safety monitoring: In public facilities such as bridges, tunnels, and underpasses, it can continuously monitor pedestrian traffic, identify abnormal behaviors such as climbing over guardrails and loitering for extended periods, and adapt to scenarios with uneven lighting and dense crowds that obstruct the view, providing timely warnings of safety hazards and assisting municipal departments in carrying out facility maintenance and safety management.

[0074] Intelligent management of communities and industrial parks: Suitable for large residential communities, industrial parks, campuses and other closed or semi-closed areas. It can detect the entry and exit of people at entrances and exits, the movement of pedestrians within the park, and identify unauthorized personnel entering restricted areas (such as equipment rooms and restricted areas in teaching buildings). It can work with access control systems and monitoring centers to achieve intelligent security management and improve the security of the area.

[0075] It should be noted that the method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0076] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Example 2

[0077] See Figure 6 Embodiment 2 of the present invention also provides a scale-adaptive pedestrian detection device based on Transformer, employing the scale-adaptive pedestrian detection method based on Transformer described in the above embodiments, including: Data preprocessing module 100 is used to perform augmentation operations on image data in the training set; The multi-scale feature extraction module 200 is used to extract features from the input image after enhancement using a backbone network, select the output feature map of a specified stage of the backbone network, and construct a multi-scale feature pyramid with a set spatial resolution and semantic level. The multi-scale feature adaptive filtering module 300 is used to process the feature map of the multi-scale feature pyramid through an adaptive filtering network, divide the feature map of different scales into image blocks and flatten them into feature sequences, and generate the filtering weights of the image blocks after capturing the context dependency relationship. Important image blocks are filtered and retained according to the weights to form a set of filtered multi-scale feature blocks. The scale-based Transformer encoding module 400 is used to flatten and encode the selected multi-scale feature block set, group the features according to their original scale, input each group of features into an independent Transformer encoder for feature encoding, and output the encoded features at the corresponding scale. The feature enhancement network processing module 500 is used to reconstruct the encoded features into a feature map, input it into the feature enhancement network, and output the enhanced feature map through multi-resolution feature capture, fusion and feature repair operations. The Transformer decoding and prediction module 600 is used to input the enhanced feature map into the Transformer decoder. The decoder interacts with the encoded features through the object query vector and outputs the pedestrian prediction category and bounding box.

[0078] In this embodiment, the enhancement operations of the data preprocessing module 100 include random horizontal flipping, random cropping, and scaling.

[0079] In this embodiment, the backbone network of the multi-scale feature extraction module 200 is a ResNet network, and the output feature maps of the last three stages of the ResNet network are selected to construct a multi-scale feature pyramid.

[0080] In this embodiment, the multi-scale feature adaptive filtering module 300 includes an adaptive filtering network comprising a two-layer Long Short-Term Memory (LSTM) network and a fully connected layer. The context dependency capture process satisfies the following formula:

[0081] The formula for the selection weight of the generated image patches is:

[0082] In the formula, This indicates that the first LSTM layer is at time step The hidden state, This indicates that the second LSTM layer is at time step The hidden state, This indicates the hidden state of the concatenation of two LSTM layers. ; The scalar value is the output of the fully connected layer. This is the weight matrix of the fully connected layer. Bias for fully connected layers; For the first The selection weight of each image patch; For the corresponding number k The scalar values ​​obtained after mapping image patches through a fully connected layer k It is the time step index of the feature sequence.

[0083] In this embodiment, the positional encoding in the scaled Transformer encoding module 400 is the same as the positional encoding method in the DETR framework, and the fusion of the flattened vector and the positional encoding satisfies the formula:

[0084] In the formula, This is the set of multi-scale feature blocks after filtering. The vector obtained by flattening the filtered multi-scale feature block set; for height, for width, Number of channels; This is the position encoding matrix; The encoder input vector; This is the encoded feature for the corresponding scale in the final output.

[0085] In this embodiment, the feature enhancement network processing module 500 includes: First, the dimensions of the encoded features are reconstructed to obtain the reconstructed feature map; then, a convolution operation is performed on the reconstructed feature map to generate the initial features. The spatial resolution of the feature map is gradually reduced through multi-layer downsampling operations; residual bottleneck calculation is performed on the feature map after the last layer of downsampling; the spatial resolution of the feature map is gradually restored through multi-layer upsampling operations, and the top-level upsampling result is directly fused with the downsampling feature of the corresponding layer; the enhanced feature map is obtained through the fused feature map.

[0086] In this embodiment, the Transformer decoding and prediction module 600 includes: The scaling dot product attention mechanism is used for calculation, and the similarity score between the object query vector and the enhanced feature map is obtained by dot product. Scaling is achieved by dividing the similarity score by the square root of the dimensions of the query vector and the key vector; The scores are converted into attention weights in the form of a probability distribution through normalization. Finally, the attention weights are weighted and summed on the enhanced feature map to obtain the attention calculation result. The attention calculation results are input into the category prediction branch and the bounding box prediction branch, respectively. The category prediction branch outputs the pedestrian prediction category probability through an activation function, and the bounding box prediction branch outputs the predicted bounding box coordinates through a feedforward network.

[0087] This embodiment also includes: The label assignment and loss calculation module 700 is used to introduce a shape consistency measure into the Hungarian matching algorithm, construct a matching cost matrix by combining classification cost and bounding box cost, achieve optimal matching between predicted boxes and ground truth boxes through the Hungarian algorithm, calculate the total loss and iteratively optimize model parameters through backpropagation. The matching cost matrix is ​​calculated according to the following formula:

[0088] In the formula, For the first i The prediction box and the first j The classification cost of a single ground truth bounding box. For bounding box cost, As a measure of shape consistency, M This represents the number of real pedestrian targets in the image. j>M The time indicates that the predicted bounding box matches an empty target.

[0089] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment in Embodiment 1 of this application, and the resulting technical effects are the same as those in the method embodiment of this application. For details, please refer to the description in the method embodiment shown above in this application, and it will not be repeated here. Example 3

[0090] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium storing program code of a Transformer-based scale-adaptive pedestrian detection method. The program code includes instructions for executing the Transformer-based scale-adaptive pedestrian detection method of Embodiment 1 or any possible implementation thereof.

[0091] Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)). Example 4

[0092] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor can execute the scale-adaptive pedestrian detection method based on Transformer in Embodiment 1 or any possible implementation thereof by calling the program instructions.

[0093] Specifically, a processor can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0094] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0095] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0096] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A scale-adaptive pedestrian detection method based on Transformer, characterized in that, Includes the following steps: Data preprocessing: Perform augmentation operations on the image data in the training set; Multi-scale feature extraction: A backbone network is used to extract features from the input image after enhancement. The output feature maps of the backbone network at a specified stage are selected to form a multi-scale feature pyramid with a set spatial resolution and semantic level. Multi-scale feature adaptive filtering: The feature map of the multi-scale feature pyramid is processed by an adaptive filtering network, the feature map of different scales is divided into image blocks and flattened into feature sequences. After capturing the context dependency, the filtering weight of the image blocks is generated. Important image blocks are filtered and retained according to the weight, forming a set of filtered multi-scale feature blocks. Multi-scale Transformer encoding: The filtered multi-scale feature block set is flattened and its position is encoded. The features are grouped according to their original scale, and each group of features is input into an independent Transformer encoder for feature encoding. The encoded features of the corresponding scale are output. Feature enhancement network processing: The encoded features are reconstructed into a feature map, which is then input into the feature enhancement network. Through multi-resolution feature capture, fusion, and feature repair operations, the enhanced feature map is output. Transformer Decoding and Prediction: The enhanced feature map is input into the Transformer decoder. The decoder interacts with the encoded features through the object query vector and outputs the pedestrian prediction category and bounding box.

2. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, The enhanced operations include random horizontal flipping, random cropping, and scaling.

3. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, The backbone network is a ResNet network, and the output feature maps of the last three stages of the ResNet network are selected to construct a multi-scale feature pyramid.

4. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, The adaptive filtering network comprises a two-layer Long Short-Term Memory (LSTM) network and a fully connected layer. The context dependency capture process satisfies the following formula: ； The formula for the selection weight of the generated image patches is: ； In the formula, This indicates that the first LSTM layer is at time step The hidden state, This indicates that the second LSTM layer is at time step The hidden state, Represents the hidden state of the splicing of a two-layer LSTM ; The scalar value is the output of the fully connected layer. This is the weight matrix of the fully connected layer. Bias for fully connected layers; For the first The selection weight of each image patch; For the corresponding number k The scalar values ​​obtained after mapping image patches through a fully connected layer k It is the time step index of the feature sequence.

5. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, The positional encoding is the same as the positional encoding method in the DETR framework, and the fusion of the flattened vector and the positional encoding satisfies the formula: ； In the formula, This is the set of multi-scale feature blocks after filtering. The vector obtained by flattening the filtered multi-scale feature block set; for height, for width, Number of channels; This is the position encoding matrix; The encoder input vector; This is the encoded feature for the corresponding scale in the final output.

6. The scale-adaptive pedestrian detection method based on Transformer according to claim 5, characterized in that, During the processing of the feature enhancement network: First, the dimensions of the encoded features are reconstructed to obtain the reconstructed feature map; then, a convolution operation is performed on the reconstructed feature map to generate the initial features. The spatial resolution of the feature map is gradually reduced through multi-layer downsampling operations; residual bottleneck calculation is performed on the feature map after the last layer of downsampling; the spatial resolution of the feature map is gradually restored through multi-layer upsampling operations, and the top-level upsampling result is directly fused with the downsampling feature of the corresponding layer; the enhanced feature map is obtained through the fused feature map.

7. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, The output process of the Transformer decoder is as follows: The scaling dot product attention mechanism is used for calculation, and the similarity score between the object query vector and the enhanced feature map is obtained by dot product. Scaling is achieved by dividing the similarity score by the square root of the dimensions of the query vector and the key vector; The scores are converted into attention weights in the form of a probability distribution through normalization. Finally, the attention weights are weighted and summed on the enhanced feature map to obtain the attention calculation result. The attention calculation results are input into the category prediction branch and the bounding box prediction branch, respectively. The category prediction branch outputs the pedestrian prediction category probability through an activation function, and the bounding box prediction branch outputs the predicted bounding box coordinates through a feedforward network.

8. The scale-adaptive pedestrian detection method based on Transformer according to claim 1, characterized in that, Also includes: Label assignment and loss calculation based on shape consistency metric: A shape consistency metric is introduced into the Hungarian matching algorithm. A matching cost matrix is ​​constructed by combining the classification cost and the bounding box cost. The optimal matching between the predicted box and the ground truth box is achieved through the Hungarian algorithm. The total loss is calculated and the model parameters are iteratively optimized through backpropagation.

9. The scale-adaptive pedestrian detection method based on Transformer according to claim 7, characterized in that, The matching cost matrix is ​​calculated according to the following formula: ； In the formula, For the first i The prediction box and the first j The classification cost of a single ground truth bounding box. For bounding box cost, As a measure of shape consistency, M This represents the number of real pedestrian targets in the image. j>M The time indicates that the predicted bounding box matches an empty target.

10. A scale-adaptive pedestrian detection device based on Transformer, employing the scale-adaptive pedestrian detection method based on Transformer as described in any one of claims 1 to 9, characterized in that, include: The data preprocessing module is used to perform augmentation operations on the image data in the training set; The multi-scale feature extraction module is used to extract features from the input image after enhancement using a backbone network, and select the output feature map of the backbone network at a specified stage to form a multi-scale feature pyramid with a set spatial resolution and semantic level. The multi-scale feature adaptive filtering module is used to process the feature map of the multi-scale feature pyramid through an adaptive filtering network, divide the feature map of different scales into image blocks and flatten them into feature sequences. After capturing the context dependency, the filtering weights of the image blocks are generated, and important image blocks are filtered and retained according to the weights to form a set of filtered multi-scale feature blocks. The scale-based Transformer encoding module is used to flatten and encode the selected multi-scale feature block set, group the features according to their original scale, input each group of features into an independent Transformer encoder for feature encoding, and output the encoded features at the corresponding scale. The feature enhancement network processing module is used to reconstruct the encoded features into a feature map, input it into the feature enhancement network, and output the enhanced feature map through multi-resolution feature capture, fusion and feature repair operations. The Transformer decoding and prediction module is used to input the enhanced feature map into the Transformer decoder. The decoder interacts with the encoded features through the object query vector and outputs the pedestrian prediction category and bounding box.

Images