A global self-adaptive cross-modal single-stage pedestrian detection method based on GPA_VLPD

Abstract

The application provides a cross-modal single-stage pedestrian detection method based on GPA_VLPD global self-adaption, class text is mapped into text features by using a pre-trained visual language model CLIP Text Encoder; pseudo labels are obtained by using a CLIP Image Encoder; then, a training set and the text features are input into a GPA_VLPD feature extraction module to obtain a GPA_VLPD model suitable for detecting pedestrians; finally, the model after training is used for inference detection on images; and if pedestrians are detected, the pedestrians are framed and output.

Description

Technical Field

[0001] This invention belongs to the fields of machine vision, autonomous driving and pedestrian video surveillance technology, and particularly relates to a cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptive. Background Technology

[0002] Pedestrian detection is a popular task in computer vision, aiming to quickly classify and locate pedestrians in images or videos in complex and ever-changing environments. In today's increasingly intelligent world, many applications rely on pedestrian detection, such as pedestrian search, action recognition, and video surveillance. Especially in autonomous driving, intelligent vehicles need to quickly detect pedestrians and react accordingly; otherwise, their lives may be threatened. Therefore, the inference speed and detection accuracy of pedestrian detection algorithms are crucial.

[0003] Pedestrian detection faces two main challenges: firstly, pedestrians are affected by non-human objects or nearby pedestrians in complex street environments, leading to intra- or inter-class occlusion; secondly, pedestrians from different directions and at greater distances appear as small targets with significantly different scales from normal targets in surveillance cameras. These two challenges greatly disrupt the generally regular shapes of pedestrians, reducing distinguishable features. The most challenging scenario is the heavy detection set, where pedestrians are severely occluded and have very small scales, resulting in compact and blurry features lacking sufficient detail. In this case, human-shaped occluders, due to their similarity in shape to pedestrians, are easily misdetected by the detector, further increasing the difficulty of pedestrian detection. Therefore, addressing these problems requires detectors with strong feature extraction capabilities and the ability to distinguish and detect pedestrians under complex conditions.

[0004] To address the aforementioned issues, two-stage methods based on Faster R-CNN and Cascade Mask R-CNN have been widely applied to various pedestrian detection tasks due to their excellent feature extraction and processing capabilities. These two-stage methods typically use HRNet or ResNeXt as the backbone extraction network, then pass the extracted features through a Region Proposal Network (RPN) to coarsely predict potential targets, followed by rigorous refinement of the feature bounding boxes within the proposed regions to generate the final prediction score. F2DNet also uses a fast suppression head to help address false positives produced by the detection head. Besides common two-stage object detection methods, there are also methods that utilize unique human target features (such as human keypoints and masks of visible parts) for pedestrian detection. However, the inference speed of these methods is limited by their network architecture, making them difficult to apply in real-world scenarios.

[0005] To simplify the network structure and accelerate inference speed, researchers developed a single-stage method that performs only one round of prediction and successfully applied it to pedestrian detection. However, the single-stage method's accuracy on various test subsets is significantly lower than that of the two-stage method F2DNet. While the single-stage method has an advantage in inference speed, it cannot maintain detection accuracy. To alleviate this problem, VLPD improves the overall architecture of CSP by introducing a pre-trained visual semantic model CLIP. This model obtains additional visual semantic labels through pseudo-labels mapped across modalities to identify the semantic class of the display context. During the training phase, a prototype semantic contrastive learning method is used to help the network better distinguish between pedestrians and context. VLPD shows significant improvement in accuracy on various test subsets of Citypersons. However, compared to the two-stage method F2DNet, VLPD still has room for improvement in detection accuracy. Summary of the Invention

[0006] Pedestrian detection is a challenging problem due to the complexity and diversity of detection environments and targets, requiring a balance between accuracy and speed. Current pedestrian detection methods are mainly divided into two-stage and single-stage approaches. Two-stage methods achieve superior accuracy by utilizing methods for generating and selecting candidate bounding boxes, but their speed is compromised. Single-stage methods directly predict on the extracted feature maps, significantly improving speed, but their accuracy is far inferior to two-stage methods. This invention aims to improve the pedestrian detection accuracy of single-stage methods to a level comparable to two-stage methods while maintaining their detection speed. The feature extraction module of VLPD is redesigned, incorporating a newly designed lightweight module, GPA, and a GPA-VLPD algorithm is proposed to achieve fast and accurate pedestrian identification and localization in complex environments. This invention enables single-stage pedestrian detection methods to be better applied in real-world scenarios, advancing advancements in fields such as autonomous driving and video surveillance.

[0007] The specific technical solution adopted is as follows:

[0008] A cross-modal single-stage pedestrian detection method based on GPA-VLPD global adaptation is proposed: The pre-trained visual language model CLIP's Text Encoder maps categorical text to text features; pseudo-labels are obtained through CLIP's Image Encoder; the training set and text features are then input into the GPA-VLPD feature extraction module to obtain a GPA-VLPD model suitable for pedestrian detection; finally, the trained model is used for inference detection on images; if a pedestrian is detected, a bounding box is selected and output.

[0009] Furthermore, before training begins, based on the template and category, the CLIP model's Text Encoder maps the categories in pedestrian detection to text feature vectors of a specified size. The generated text feature vectors are then used to calculate the cosine similarity with the image features through a dot product, thereby achieving cross-modal fusion of images and text.

[0010] Furthermore, the GPA-VLPD feature extraction module includes a simplified channel and spatial attention module (GPA) for adaptively refining the feature map. It uses a parallel computation structure of channel attention and spatial attention to obtain channel weights and spatial weights. The channel weights are used to adjust the feature response of each channel, and the spatial weights are used to adjust the feature response at different spatial locations. Then, the channel weights and spatial weights are fused using an additive reactivation method to obtain a unified 3D attention map. Finally, the attention map and the input feature are multiplied pixel-by-pixel to achieve adaptive feature adjustment in both channel and spatial aspects.

[0011] Furthermore, in the simplified channel and spatial attention module GPA:

[0012] For channel attention:

[0013] A ch (X)=F ex (F sq ,W)=σ[W2δ(W1(F GP (x)))]

[0014] Use global equal pooling F GP The spatial information of the global feature map is compressed into a one-dimensional vector of size C×1×1: This vector represents the global importance of the channel feature; it is directly forwarded to a shared network with two fully connected layers and one hidden layer, and the dimensionality reduction rate r of the dimensionality reduction layer is set to 16; the shared network is used to learn how to weight the feature response of each channel; after processing by the shared network, in order to add it pixel-by-pixel with the spatial features, the feature vector is expanded to the size of the input features using the σ(·) tensor expansion function, resulting in the channel feature vector A. ch (X);

[0015] For spatial attention:

[0016] A sp (X)=σ(W([F avg (X); F max (X)]))

[0017] First, using the feature channels as axes, the input features are subjected to global average pooling and max pooling to obtain two 2D spatial attention features. Then, the two generated spatial attention features are concatenated to generate a spatial attention map. A 7×7 convolution operation is used to reduce parameters and computational cost; finally, the generated 2D spatial attention map is expanded to the size of the input features using a dilation function to obtain the spatial feature vector A. sp (X);

[0018] Spatial channel fusion: synthesizes the outputs of the channel branches and spatial branches in a parallel layout.

[0019] GMA(X)=Z⊙X=(F SG (A ch +A sp ))⊙X

[0020] The "+" operator adds the channel feature weights and spatial feature weights pixel by pixel, achieving a balanced fusion of channel and spatial information; F SG is the sigmoid activation function, which limits the output to between 0 and 1, representing the weight of each position; "⊙" is the pixel-wise multiplication of the feature map with the 3D attention map.

[0021] Furthermore, in the GPA-VLPD feature extraction module, VLPD uses ResNet-50 as the backbone and initializes the CLIP weights of the pre-trained visual language model; VLPD does not perform downsampling operation in Stage 5, but uses dilated convolution to increase the receptive field, and adds SE-POSTblock to each layer of the last stage of ResNet-50.

[0022] The simplified channel and spatial attention module GPA are added after the p3 and p4 output layers of VLPD to form the GPA-VLPD feature extraction module.

[0023] Furthermore, in the training phase, the overall network architecture of the GPA-VLPD feature extraction module includes:

[0024] The first part consists of a feature extraction module and a CSP detection head. First, the backbone is initialized using CLIP-ResNet50 pre-trained weights. Then, features are extracted from the input image, and the feature vector output from the last layer of the backbone is fused with the text vector generated by CLIP. The p3, p4, and p5 layers of the backbone are deconvolved to the same size and concatenated. The feature vectors that have fused image and text information are upsampled and also concatenated. Finally, the CSP detection head performs inference and prediction directly on the feature map containing visual semantic information.

[0025] The second part involves unsupervised comparative learning of the p5 feature map that incorporates semantic information and the feature map generated by CLIP Image-Encoder that also incorporates semantic information.

[0026] The third part involves performing prototype semantic comparison learning on the feature map after p3 deconvolution, the bounding box, and the p5 feature map that incorporates semantic information. This brings targets of the same category closer together and moves targets of different categories further apart, thereby improving the model's ability to distinguish pedestrians and the environment.

[0027] Furthermore, during the inference phase, the GPA-VLPD feature extraction module retains only the first part and the corresponding weights obtained during training.

[0028] Furthermore, the training dataset was constructed as follows: First, the Citypersons high-resolution dataset was divided into a training set and a test set; the training set was randomly cropped to the same size and corresponding label files were generated; then, the training set was subjected to data augmentation operations of random brightness changes and random flipping.

[0029] And, an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation as described above.

[0030] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation as described above.

[0031] Compared to existing technologies, the GPA-VLPD proposed by this invention and its preferred embodiment, which has the detection accuracy of a two-stage pedestrian detection method and the inference speed of a single-stage pedestrian detection method, is more suitable for application in real life, so as to promote the progress of autonomous driving, pedestrian video surveillance and other fields. Attached Figure Description

[0032] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0033] Figure 1 This is a template and category diagram for embodiments of the present invention;

[0034] Figure 2 This is a schematic diagram of the GPA module in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the SE-POST module according to an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the GPA-VLPD feature extraction module in an embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of the GPA-VLPD network architecture according to an embodiment of the present invention;

[0038] Figure 6 , Figure 7 , Figure 8 Three images were randomly selected from the test set;

[0039] Figure 9 This is the folder structure for storing the training set and test set in an embodiment of the present invention;

[0040] Figure 10 , Figure 11 This is a diagram showing the detection results of pedestrian test sets under different conditions according to an embodiment of the present invention;

[0041] Figure 12 , Figure 13 , Figure 14 Corresponding to Figure 6 , Figure 7 , Figure 8 The image shows the test results. Detailed Implementation

[0042] In the following, specific embodiments of this application will be described in detail with reference to the accompanying drawings. Based on these detailed descriptions, those skilled in the art will be able to clearly understand and implement this application. Without departing from the principles of this application, features from various embodiments can be combined to obtain new implementations, or certain features from some embodiments can be substituted to obtain other preferred implementations.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation: The detection scheme provided by the embodiments of this invention consists of four steps: image preprocessing, obtaining visual and semantic features using CLIP, model training, and pedestrian detection.

[0045] First, the Citypersons high-resolution dataset (1024×2048) was divided into training and testing sets. To save training time and computational resources, the training set was randomly cropped to the same size (640×1024) and corresponding location label files were generated. To enhance the robustness of the model, data augmentation operations such as random brightness changes and random flipping were performed on the training set. Next, the pre-trained visual language model CLIP's Text Encoder was used to map the categorical text to text features. Pseudo-labels were obtained using CLIP's Image Encoder. Then, the training set and text features were input into the GPA-VLPD network to obtain a GPA-VLPD model suitable for pedestrian detection. Finally, the model was used to perform inference detection on the Citypersons test set. If a pedestrian was detected, the pedestrian was selected with a green box and output; otherwise, the same result image as the input image was output.

[0046] The main technical details of this solution are described below:

[0047] 1. Cross-modal data processing

[0048] CLIP is a multimodal model based on contrastive learning. Benefiting from its large text-image pair training set, CLIP has powerful zero-shot classification capabilities. Therefore, before training begins, based on the template and category, the CLIP model's Text Encoder maps the categories in pedestrian detection to text feature vectors of size 1024×10 (10 being the number of categories). Figure 1 This demonstrates the specific components of the template and categories before text mapping, as well as the pre-trained weights of CLIP. The generated text feature vectors are then used to calculate cosine similarity with image features via dot product, thereby achieving cross-modal fusion of images and text, which helps improve the category discrimination ability of the backbone.

[0049] 2. GPA Module Principle

[0050] This invention proposes a highly simplified channel and spatial attention module (GPA), such as Figure 2 As shown, adaptive feature refinement is performed on the feature map. Inspired by BAM, CBAM, and Polarized Self-Attention, GPA uses a structure that computes channel attention and spatial attention in parallel to obtain channel weights and spatial weights. Channel weights are used to adjust the feature response of each channel, enhancing useful information and reducing noise, while spatial weights are used to adjust the feature response at different spatial locations, improving sensitivity to important regions. Then, the channel weights and spatial weights are fused using an additive reactivation method to obtain a unified 3D attention map. Finally, the attention map and the input feature are multiplied pixel by pixel to achieve adaptive feature adjustment in both channel and spatial aspects.

[0051] For channel attention:

[0052] A ch (X)=F ex (F sq ,W)=σ[W2δ(W1(F GP (X)))]

[0053] To alleviate channel dependency and enhance the ability of channel units to utilize out-of-region context, a global flat pooling F is used. GP The spatial information of the global feature map is compressed into a one-dimensional vector of size (C×1×1): This vector represents the global importance of the channel feature. To reduce model complexity and parameter overhead, this vector is directly forwarded to a shared network with two fully connected layers and one hidden layer, and the reduction rate r of the dimensionality reduction layer is set to 16, referencing SE attention. The shared network learns how to weight the feature responses of each channel. After processing by the shared network, to perform pixel-wise addition with the spatial features, the feature vector is expanded to the size of the input features using the σ(·) tensor expansion function, resulting in the channel feature vector A. ch (X).

[0054] For spatial attention:

[0055] To strengthen the spatial connections between input features, the input features are first subjected to global average pooling and max pooling along the feature channels to obtain two 2D spatial attention features. Then, these two spatial attention features are concatenated to generate a spatial attention map. By using a 7×7 convolution operation, the number of parameters and computational cost are greatly reduced, which is beneficial for establishing high-dimensional spatial feature correlations. Finally, the generated 2D spatial attention map is expanded to the size of the input features using a dilation function to obtain the spatial feature vector A. sp (X).

[0056] A sp (X)=σ(W([F avg (X); F max (X)]))

[0057] Spatial channel fusion: The outputs of the aforementioned channel branches and spatial branches are synthesized in a parallel layout.

[0058] GMA(X)=Z⊙X=(F SG (A ch +A sp ))⊙X

[0059] The "+" sign adds the channel feature weights and spatial feature weights pixel by pixel, achieving a balanced fusion of channel and spatial information. F SG The '-' is the sigmoid activation function, which limits the output to between 0 and 1, representing the weight at each position. It also adds non-linearity to the weighted feature vector that integrates channel and spatial information, enabling it to better fit the input features. The '⊙' multiplies the feature map with the 3D attention map pixel by pixel, greatly improving the connection between various features in channels and space, and making it more conducive to extracting effective features of the target.

[0060] 3. GPA-VLPD Feature Extraction Module

[0061] VLPD selects the classic ResNet-50 as its backbone and initializes the CLIP weights of the pre-trained visual language model. ResNet-50 has a simple structure and fast inference speed, but its feature extraction capability is slightly inferior to backbones like HRNetv2. Furthermore, to reduce feature information loss during downsampling, VLPD does not perform downsampling in Stage 5.

[0062] To compensate for the loss of receptive field due to the lack of downsampling, VLPD selects dilated convolution in Stage 5 to increase the receptive field, which to some extent limits the expressive power of the backbone. Inspired by BoTNet and SENet, such as Figure 3As shown, GPA-VLPD adds an SE-POSTblock to each layer of the last stage of ResNet-50. With a small increase in parameters and computation, it better mines the relationships between channels, adaptively adjusts channel information, and improves backbone performance. GPA-VLPD maintains the overall backbone architecture largely unchanged, and the original VLPD training formula and CLIP initialization weights remain applicable, effectively addressing the issue of insufficient backbone capability in VLPD.

[0063] By comparing the experimental results of VLPD with the two-stage method F2DNet, it was found that VLPD's ability to detect severely occluded targets in complex environments is slightly insufficient. VLPD reduced the miss rate of the heavily occluded subset by 1.62% by adding a Prototype Semantic Contrast (PSC) module during training, but we believe this is not the limit of the PSC module, so we explored the PSC module in more depth. Experiments revealed that, as... Figure 4 The feature input of PSC is changed by Modified to Afterwards, although the miss rate of other test subsets increased, the miss rate of the heavily occluded subset decreased by 0.11. Furthermore, because the PSC input size decreased from (768, 160, 320) to (256, 160, 320) (H and W were 640 and 1280 respectively during training), the computational cost during training was reduced, making it more compatible with training hardware. To compensate for the lack of semantic information in the PSC with only one feature layer and to further improve the detector's ability to detect severely occluded targets, a GPA module with global adaptive capabilities was added after the output layers of the feature extraction modules p3 and p4. Through the above series of operations, the feature extraction module of GPA-VLPD was finally formed.

[0064] 4. Overall network architecture and process

[0065] During the training phase, the overall network architecture is divided into three parts. For example... Figure 4 and Figure 5As shown, the first part consists of a feature extraction module and a CSP detection head. This part first initializes the backbone using CLIP-ResNet50 pre-trained weights, then extracts features from the input image, fusing the feature vector output from the last layer of the backbone with the text vector generated by CLIP. To obtain multi-scale information and better detect pedestrians of different sizes, p3, p4, and p5 of the backbone are deconvolved to the same size and concatenated. The feature vectors that fuse image and text information are also upsampled and concatenated. Finally, the CSP detection head performs inference and prediction directly on the feature map containing visual semantic information. The second part involves unsupervised comparative learning between the p5 feature map (which incorporates semantic information) and the feature map generated by the CLIP Image-Encoder, which also incorporates semantic information, to improve the learning ability of the feature extraction module. The third part involves prototypical semantic comparative learning between the deconvolved p3 feature map, the bounding boxes, and the p5 feature map (which incorporates semantic information). This brings targets of the same category closer together and keeps targets of different categories further apart, thereby improving the model's ability to distinguish pedestrians and the environment. During the inference phase, only the first part needs to be retained. Therefore, although training takes more time, it does not affect the model's inference speed. In fact, this model is very suitable for transfer learning because only the weights of the first part need to be retained in practical applications. Additional generated text features only need to be generated and saved before inference, without affecting the network's inference speed.

[0066] The computer experimental environment used in this embodiment is as follows: Ubuntu 20.04, CPU: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90HGz, GPU: GeForce GTX 4090, Memory: 128G, Python: 3.8.10, PyTorch: 1.10.0

[0067] 1. Dataset Creation

[0068] The Citypersons dataset is a pedestrian detection dataset specifically built upon the CityScapes dataset. It selects 5000 high-quality images from CityScapes and annotates the bounding boxes of pedestrians within them. This dataset records 19654 people in 18 different cities under various weather conditions, with an average of 7 pedestrians per image. The training set contains 2975 images, and the test set contains 500 images. The images can be downloaded at [link to download]; the annotation files can be downloaded at [link to download]; and the CLIP pre-trained weights can be downloaded at [link to download].

[0069] All dataset images are .png format images with a size of 1024×2048. Figure 6 , Figure 7and Figure 8 These are three images randomly selected from the test set.

[0070] In this method, all training images are resized to 640×1024 pixels before being fed into the model for training. Furthermore, data augmentation techniques such as random cropping, flipping, and lighting adjustments are performed to improve the model's robustness. The folder structure for storing the training and test sets is as follows: Figure 9 As shown:

[0071] 2. Define training hyperparameters and evaluation metrics

[0072] To ensure fairness in comparison with other single-stage pedestrian detection methods, the hyperparameter settings in this network are kept consistent. The main hyperparameters are as follows: Initial learning rate: 2e-4, BatchSize: 4, epoch: 200, lr_policy: step, λ1: 100, λ2: 1e-3. The complete hyperparameter set can be downloaded from [link to download].

[0073] MR -2 By adjusting different detection confidence thresholds, different relationships between log(MR) and FPPI (MR = 1 - Recall, FPPI = FP / N) are obtained. Using FPPI as the x-axis and log(MR) as the y-axis, nine FPPI values ​​within the range [0.01, 100] are uniformly selected to obtain their corresponding nine log(MR) values. These y-axis values ​​are then averaged, and finally, the average is restored to a percentage of MR using exponential operations. This yields the MR value used to quantify the MR-FPPI curve. -2 The smaller the value of this indicator, the higher the detector performance.

[0074] To more accurately evaluate GAP-VLPD's ability to detect pedestrians under various complex conditions, detection was performed on various test subsets. The different test subsets were defined by the height and visibility of pedestrians in the images, as shown in Tables 1 and 2.

[0075] Table 1

[0076] Height [50,inf] [50,75] [50,inf] [20,inf] Visibility [0.65,inf] [0.65,inf] [0.2,0.65] [0.2.inf]

[0077] Table 2

[0078] Height [50,inf] [50,inf] [50,inf] [20,inf] Visibility [0,0.65] [0.65,0.9] [0.9,1] [0.2,1]

[0079] 3. Model Training

[0080] After defining all the parameters, start training the model. The training command is as follows:

[0081] CUDA_VISIBLE_DEVICES=0,1python-mtorch.distributed.launch--nproc_per_node2--master_port45622trainval_distributed.py

[0082] The following data will be output during training:

[0083] Epoch1beginEpoch1end,AvgLossis0.015213,Timeused281.0sec.

[0084] Epoch 1 has the lowest loss: 0.0152127

[0085] ----------Epoch2begin

[0086] [Epoch 2 / 300, Batch 80 / 500]$<Totalloss:0.014988> cls:0.001920,reg:0.000329,off:0.005166,ps:0.753704,pc:0.035747,Time:0.566,lr:0.000133

[0087] Where cls, reg, and off are the class loss, regression loss, and positional offset loss generated by the CSP detection head, respectively. Figure 4 As shown, ps is the unsupervised contrastive learning loss. pc is the prototype semantic contrastive learning loss. Total loss calculation function for:

[0088]

[0089] After each epoch, the average loss, minimum loss, and time taken for that epoch are calculated. If the loss for this epoch is the lowest, the parameters of the current model are saved as best.pt. The total training time is 15.6 hours.

[0090] 4. Model Testing

[0091] After training, pedestrian detection was performed on a test set defined under different conditions based on the saved best.pt file. The detection results are as follows: Figure 10 and Figure 11As shown, GPA-VLPD's miss rate is lower than that of the two-stage method F2DNet on all test subsets. This demonstrates GPA-VLPD's powerful feature extraction capabilities and its ability to identify pedestrians in complex environments. Meanwhile, GPA-VLPD achieves an FPS of 11.5, maintaining the inference speed of single-stage pedestrian detection methods. GPA-VLPD, which balances detection accuracy and inference speed, is more suitable for real-world applications.

[0092] Figure 12 , Figure 13 and Figure 14 They are Figure 6 , Figure 7 and Figure 8 The inference detection results, as shown by the output green detection boxes, clearly demonstrate the accuracy of GPA-VLPD in pedestrian detection, even identifying pedestrians who are mostly obscured by utility poles and vehicles.

[0093] The existing two-stage pedestrian detection method F2DNet outperforms the single-stage pedestrian detection method VLPD in terms of detection accuracy across various test subsets. F2DNet's miss rates on the Reasonable and HO test subsets are 8.7 and 32.6, respectively, significantly better than VLPD's 9.4 and 34.88. However, in real-world applications, pedestrian detection methods require a balance between detection accuracy and inference speed. F2DNet's FPS of 2.27 is far lower than VLPD's 11.7 FPS, thus necessitating a more balanced pedestrian detection model that balances detection accuracy and inference speed. GPA-VLPD, by reconstructing VLPD's feature extraction module and adding a simplified channel and spatial attention module (GPA), achieves a reduction in miss rates to 8.69 and 32.47 on the Reasonable and HO test subsets, respectively, while maintaining an FPS of 11.5. GPA-VLPD's detection accuracy is comparable to F2DNet, and even on the Small test subset, GPA-VLPD's miss rate reaches 10.54, surpassing F2DNet's 11.3, while maintaining a similar detection speed to VLPD. Therefore, GPA-VLPD combines fast inference speed with high detection accuracy. GPA-VLPD performs well on various test subsets, particularly showing a 0.4% and 2.4% reduction in miss rate compared to VLPD on the small target and severe occlusion test subsets, respectively. This demonstrates GPA-VLPD's robustness in complex detection environments, enabling it to better distinguish between small pedestrians and occluded objects. In conclusion, the GPA-VLPD proposed in this invention, with its detection accuracy of a two-stage pedestrian detection method and inference speed of a single-stage pedestrian detection method, is more suitable for real-world applications, driving progress in fields such as autonomous driving and pedestrian video surveillance.

[0094] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0096] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0099] This patent is not limited to the above-described preferred embodiment. Anyone can derive other forms of cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation under the guidance of this patent. All equivalent changes and modifications made within the scope of this patent application shall be covered by this patent.

Claims

1. A cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation, characterized in that: The pre-trained visual language model CLIP's Text Encoder maps categorical text to text features; pseudo-labels are obtained through CLIP's Image Encoder; then the training set and text features are input into the GPA-VLPD feature extraction module to obtain a GPA-VLPD model suitable for pedestrian detection; finally, the trained model is used to perform inference detection on the image; if a pedestrian is detected, it is selected and output. The GPA-VLPD feature extraction module includes a simplified channel and spatial attention module (GPA) for adaptive feature refinement of the feature map. It uses a parallel computation structure of channel attention and spatial attention to obtain channel weights and spatial weights. The channel weights are used to adjust the feature response of each channel, and the spatial weights are used to adjust the feature response at different spatial locations. Then, the channel weights and spatial weights are fused using an additive reactivation method to obtain a unified 3D attention map. Finally, the attention map and the input feature are multiplied pixel-by-pixel to achieve adaptive feature adjustment in both channel and spatial aspects. In the aforementioned simplified channel and spatial attention module GPA: For channel attention: Use global equal pooling Compress the spatial information of the global feature map to a size of [size missing]. One-dimensional vector: This vector represents the global importance of the channel features; it is directly forwarded to a shared network with two fully connected layers and one hidden layer, and the dimensionality reduction rate r of the dimensionality reduction layer is set to 16; the shared network is used to learn how to weight the feature responses of each channel; after processing by the shared network, in order to add it pixel-by-pixel with the spatial features, it utilizes... The tensor expansion function expands the feature vector to the size of the input feature vector, resulting in the channel feature vector. ; For spatial attention: First, using the feature channels as axes, the input features are subjected to global average pooling and max pooling to obtain two 2D spatial attention features. Then, the two generated spatial attention features are concatenated to generate a spatial attention map. , ; through a convolution kernel for The convolution operation is performed to reduce parameters and computation; finally, the generated 2D spatial attention map is expanded to the size of the input features using a dilation function to obtain the spatial feature vector. ; Spatial channel fusion: Combines the outputs of channel branches and spatial branches in a parallel layout. The "+" sign adds the channel feature weights and spatial feature weights pixel by pixel, achieving a balanced fusion of channel and spatial information. It's the sigmoid activation function, which limits the output to between 0 and 1, representing the weight at each position; "This involves multiplying the feature map pixel by pixel with the 3D attention map." 2. The cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation according to claim 1, characterized in that: Before training begins, based on the template and category, the CLIP model's Text Encoder maps the categories in pedestrian detection to text feature vectors of a specified size. The generated text feature vectors are then used to calculate the cosine similarity with the image features through a dot product, thereby achieving cross-modal fusion of images and text.

3. The cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation according to claim 1, characterized in that: In the GPA-VLPD feature extraction module, VLPD uses ResNet-50 as the backbone and initializes the CLIP weights of the pre-trained visual language model. VLPD does not perform downsampling in Stage 5, but uses dilated convolution to increase the receptive field. SE-POST blocks are added to each layer of the last stage of ResNet-50. The simplified channel and spatial attention module GPA are added after the p3 and p4 output layers of VLPD to form the GPA-VLPD feature extraction module.

4. The cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation according to claim 3, characterized in that: During the training phase, the overall network architecture of the GPA-VLPD feature extraction module includes: The first part consists of a feature extraction module and a CSP detection head. First, the backbone is initialized using CLIP-ResNet50 pre-trained weights. Then, features are extracted from the input image, and the feature vector output from the last layer of the backbone is fused with the text vector generated by CLIP. The p3, p4, and p5 layers of the backbone are deconvolved to the same size and concatenated. The feature vectors that have fused image and text information are upsampled and also concatenated. Finally, the CSP detection head performs inference and prediction directly on the feature map containing visual semantic information. The second part involves unsupervised comparative learning of the p5 feature map that incorporates semantic information and the feature map generated by CLIP Image-Encoder that also incorporates semantic information. The third part involves performing prototype semantic comparison learning on the feature map after p3 deconvolution, the bounding box, and the p5 feature map that incorporates semantic information. This brings targets of the same category closer together and moves targets of different categories further apart, thereby improving the model's ability to distinguish pedestrians and the environment.

5. The cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation according to claim 4, characterized in that: During the inference phase, the GPA-VLPD feature extraction module retains only the first part and the corresponding weights obtained during training.

6. The cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation according to claim 1, characterized in that: The dataset used for training was constructed as follows: First, the Citypersons high-resolution dataset was divided into a training set and a test set; the training set was randomly cropped to the same size and corresponding label files were generated; then, the training set was subjected to data augmentation operations of random brightness changes and random flipping.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation as described in any one of claims 1-6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of a cross-modal single-stage pedestrian detection method based on GPA_VLPD global adaptation as described in any one of claims 1-6.

Images