Multi-view scale perception pedestrian detection system and method

Abstract

The invention discloses a multi-view scale perception pedestrian detection system and method, and belongs to the technical field of computer vision, and the system comprises an initial feature extraction layer, a single-view two-dimensional detection layer, a multi-view scale perception feature extraction module, a shared weight supervision module, and a multi-view feature decoding layer. Acquiring pedestrian visible light images simultaneously captured by a plurality of cameras in the same scene; and inputting the multi-view pedestrian visible light image into the optimized multi-view scale perception pedestrian detection system to obtain a predicted multi-view scene-level pedestrian occupancy image, and the bright spot of the predicted multi-view scene-level pedestrian occupancy image being the position of the pedestrian in the scene. According to the method, the multi-view scale sensing information is fused, and the pedestrian position information of the single-view projection features is fully utilized through a shared weight supervision mechanism, so that the accuracy of multi-view pedestrian detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-view scale perception pedestrian detection system and method. Background Art

[0002] As people pay more and more attention to the safety issues in public places, multi-view pedestrian detection has attracted extensive attention from researchers because it can predict the number of people and crowd distribution in unconstrained scenes. As an important topic in the field of computer vision, multi-view pedestrian detection has a wide range of applications in real-world monitoring system engineering, such as crowd flow monitoring, crowd analysis, security management, and traffic monitoring.

[0003] Despite significant progress in object detection, especially in localization accuracy and robustness in various scenarios, occlusion remains a challenging problem. Multi-view pedestrian detection systems enhance the resistance to occlusion by integrating information from different views. Traditional multi-view pedestrian detection predicts scene-level pedestrian occupancy maps by integrating image data from different views. However, scale differences caused by perspective distortion (i.e., different distances from the camera and different body sizes of different pedestrians in the same view) remain a challenge. Existing methods fail to fully consider the contextual scale awareness of projected features at each view, which limits the improvement of multi-view pedestrian detection performance. Previous methods usually use the same convolution kernel and aggregation operation, ignoring the fact that scale changes continuously in the image. Some researchers have tried to solve the scale problem by combining feature maps from images of different resolutions or using convolution kernels of different sizes, but failed to effectively handle rapid scale changes. In addition, previous multi-view detection methods only use the supervision information of single-view 2D detection, and do not fully utilize the pedestrian position supervision information of single-view projected features, which also limits the improvement of multi-view pedestrian detection performance. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-view scale-aware pedestrian detection system and method, which can reduce the adverse effects of the continuous change of multi-view image scale on the detection performance and make full use of the pedestrian position supervision information of the single-view projection features, thereby improving the detection accuracy of the multi-view pedestrian detection scheme.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a multi-view scale-perception pedestrian detection system, including an initial feature extraction layer, on which a single-view two-dimensional detection layer, a multi-view scale-perception feature extraction module and a shared weight supervision module are connected at the same time, and a common multi-view feature decoding layer is connected to the multi-view scale-perception feature extraction module and the shared weight supervision module, wherein the shared weight supervision module includes a single-view projection feature decoding layer and a shared weight map generation layer.

[0006] The further improvement of the technical solution of the present invention is that the specific structure of each layer is as follows:

[0007] Initial feature extraction layer: from input to output, it includes 2 convolutional layers, 1 maximum pooling layer, 2 convolutional layers, 1 maximum pooling layer, 4 convolutional layers, 1 maximum pooling layer, 4 convolutional layers, 1 maximum pooling layer, and 4 convolutional layers;

[0008] Single-view 2D detection layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, and 1 convolution layer;

[0009] Multi-view scale-aware feature extraction module: from input to output, it includes four scale parallel structures of different scales, one channel connection operation and one convolution layer. Among them, the first scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the second scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the third scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the fourth scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output;

[0010] Single-view projection feature decoding layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer;

[0011] Shared weight map generation layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer;

[0012] Multi-view feature fusion decoding layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, and 1 convolution layer.

[0013] A multi-view scale perception pedestrian detection method, the specific steps are as follows:

[0014] Step 1: Obtain camera parameters, multi-view visible light images from the same scene, real multi-view pedestrian head position maps, real multi-view pedestrian foot position maps, and real multi-view scene-level pedestrian occupancy maps;

[0015] Step 2: Input multiple viewing angle visible light images into the initial feature extraction layer in sequence to extract initial features, and obtain multiple viewing angle initial feature maps respectively;

[0016] Step 3: Input the initial feature maps of multiple viewpoints into the single view two-dimensional detection layer in sequence for two-dimensional detection, and obtain the pedestrian head position map and foot position map of multiple viewpoints;

[0017] Step 4: Project multiple view initial feature maps in sequence to obtain multiple view projection feature maps;

[0018] Step 5: Input multiple view projection feature maps into the multi-view scale-aware feature extraction module in sequence to extract scale-aware features, obtain multiple view scale-aware feature maps and connect them by channel;

[0019] Step 6: Input multiple view projection feature maps into the shared weight supervision module in sequence to obtain a shared weight map;

[0020] Step 7: Multiply the multiple view-scale perception feature maps with the shared weight map to obtain weighted multiple view-scale perception feature maps;

[0021] Step 8: Input the weighted multi-view scale perception feature maps into the multi-view feature decoding layer for multi-view feature decoding to obtain a multi-view scene-level pedestrian occupancy map.

[0022] The further improvement of the technical solution of the present invention is that step 3 is specifically as follows:

[0023] Step 3.1: Input the initial feature maps of multiple viewpoints into the single view 2D detection layer in sequence for 2D detection, and obtain the predicted pedestrian head position maps and foot position maps of multiple viewpoints;

[0024] Step 3.2: The loss between the predicted pedestrian head position map and foot position map of multiple perspectives and the actual pedestrian head position map and foot position map of multiple perspectives is calculated as:

[0025]

[0026] in, and They are respectively the pedestrian head position map and foot position map of the predicted camera view c; and They are the pedestrian head position map and foot position map of the real camera view c respectively; the Gaussian kernel f(·) is used for blurring; ||·||2 indicates the calculation of the Euclidean distance.

[0027] The further improvement of the technical solution of the present invention is that the specific steps of step 5 are as follows:

[0028] Step 5.1: Input multiple view projection feature maps into the four-scale parallel structure in sequence to obtain the intermediate scale perception feature map;

[0029] Step 5.2: Connect the intermediate scale-aware feature map and the original projection feature map by channel and then input them into the 1×1 convolutional layer to obtain the multi-view scale-aware feature map.

[0030] The further improvement of the technical solution of the present invention is that the specific steps of step 5.1 are as follows:

[0031] Step 5.11: Project the multi-view feature map F obtained in step 3 p Input four scale parallel structures in sequence to calculate multi-scale features The multi-scale features obtained by the parallel structure with input scale i Calculated as:

[0032]

[0033] Where, for each scale i of the camera view c, P ave (·,i) average the projection features of each view into k(i)×k(i) blocks; is a convolutional network with a kernel size of 1, which can combine context features across channels without changing its dimension. B represents bilinear interpolation, which is used to upsample the context feature array to Same size;

[0034] Step 5.12: Define contrast features:

[0035] Step 5.13: In the multi-view scale-aware feature extraction module, the contrast feature provides important information to understand the local scale of each image region. The contrast feature is used as an auxiliary network The input weight is defined as Used to calculate the weight map assigned to each different scale The weight map of scale i for camera view c is defined as follows:

[0036]

[0037] is a 1×1 convolutional layer followed by a sigmoid function to avoid division by zero, and uses weights to calculate the intermediate scale perceptual features:

[0038]

[0039] Here, [·|·] represents the channel-level connection operation, which is the element-wise product between the weight map and the feature map.

[0040] The further improvement of the technical solution of the present invention is that the specific steps of step 6 are as follows:

[0041] Step 6.1: Input the multi-view projection feature maps into the projected single-view feature decoding layer in sequence for single-view feature decoding to obtain the predicted single-view scene-level pedestrian occupancy map. Calculate the loss between the predicted single-view scene-level pedestrian occupancy map and the real single-view scene-level pedestrian occupancy map. The pedestrian position supervision information loss term of the single-view projection feature is defined as follows:

[0042]

[0043] in, is the predicted single-view scene-level pedestrian occupancy map in camera view c; g c is the pedestrian occupancy map at the single-view scene level in the real camera view c; g is the pedestrian occupancy map at the real multi-view scene level; M c is the mask for single-view scene-level detection in camera view c, which can be obtained from the camera internal parameters; g c It can be obtained by g and M c Multiply to obtain; is element-wise multiplication;

[0044] Step 6.2: Connect the single-view scene-level pedestrian occupancy maps of each view by channel and then input them into the shared weight map generation layer to obtain the shared weight map.

[0045] A further improvement of the technical solution of the present invention is that step 8 is specifically as follows:

[0046] The weighted multi-view scale perception feature maps are input into the multi-view feature decoding layer for multi-view feature decoding to obtain the predicted multi-view scene-level pedestrian occupancy map. The predicted multi-view scene-level pedestrian occupancy map and the real multi-view scene-level pedestrian occupancy map are compared to calculate the loss. The supervision information loss term of the final output multi-view scene-level pedestrian occupancy map is defined as follows:

[0047]

[0048] The overall loss for training is defined as follows:

[0049]

[0050] Among them, α and β are hyperparameters used to adjust the importance of the two losses in training; C is the number of camera views.

[0051] Due to the adoption of the above-mentioned technical solution, the technical progress achieved by the present invention is as follows: by setting a multi-view scale-aware feature extraction module to adaptively encode the contextual information of the continuous scale changes of multiple view images, the design of this module is simple and efficient in alleviating the scale change problem in multi-view detection, and the extracted scale-aware feature map can effectively improve the final detection accuracy of the detection network model to reduce the adverse effects of the scale change problem of multi-view images on the feature extraction process. In addition, by introducing the pedestrian position supervision information of the single-view projection feature, using two specific projections of the single-view feature decoding layer and the shared weight map generation layer to obtain a shared weight map with pedestrian scene-level position supervision information, it helps to better integrate the multi-view pedestrian features and improve the accuracy of multi-view pedestrian image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a schematic diagram of the structure of the detection system of the present invention;

[0054] Figure 2 It is a schematic flow chart of the detection method of the present invention;

[0055] Figure 3 It is a schematic diagram of using the multi-view pedestrian detection method of the present invention to perform pedestrian detection. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with embodiments:

[0057] like Figure 1 The figure shows a schematic diagram of the structure of a multi-view scale perception pedestrian detection system, including an initial feature extraction layer, on which a single-view two-dimensional detection layer, a multi-view scale perception feature extraction module and a shared weight supervision module are connected, and a common multi-view feature decoding layer is connected to the multi-view scale perception feature extraction module and the shared weight supervision module, wherein the shared weight supervision module includes a single-view projection feature decoding layer and a shared weight map generation layer. The specific structure of each module of the system is as follows:

[0058] Initial feature extraction layer: It is the remaining part of the VGG19 network after removing the fully connected layer and the Softmax layer, including 16 convolutional layers and 4 maximum pooling layers. From input to output, there are 2 3×3 convolutional layers, 1 maximum pooling layer, 2 3×3 convolutional layers, 1 maximum pooling layer, 4 3×3 convolutional layers, 1 maximum pooling layer, 4 3×3 convolutional layers, 1 maximum pooling layer and 4 3×3 convolutional layers. The number of feature map channels generated by each convolutional layer is 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, and 512, respectively. The step size of the pooling layer is 2;

[0059] Single-view 2D detection layer: 1 convolution layer with a kernel size of 1×1 and 64 output channels, 1 ReLU layer, and 1 convolution layer with a kernel size of 1×1 and 2 output channels;

[0060] Multi-view scale perception feature extraction module: In this embodiment, it includes four scale parallel structures with scales of 1, 2, 3, and 6, 1 channel connection operation, and 1 1×1 convolution layer. Among them, the first scale is 1 1×1 average pooling layer, 1 1×1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 1×1 convolution layer, and 1 multiplication operation from input to output; the second scale is 1 2×2 average pooling layer, 1 1×1 convolution layer, and 1 2×2 convolution layer from input to output. layer, 1 upsampling layer, 1 subtraction operation, 1 1×1 convolution layer, and 1 multiplication operation; the third scale is 1 3×3 average pooling layer, 1 1×1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 1×1 convolution layer, and 1 multiplication operation from input to output; the fourth scale is 1 6×6 average pooling layer, 1 1×1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 1×1 convolution layer, and 1 multiplication operation from input to output;

[0061] Single view projection feature decoding layer: from input to output, it includes 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, a ReLU layer, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, a ReLU layer, 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, a padding size of 2, and a dilation rate of 2, a ReLU layer, 1 convolution kernel size of 3×3, an output channel number of 256, a padding size of 4, and a dilation rate of 4, a ReLU layer, 1 convolution kernel size of 3×3, an output channel number of 128, a padding size of 8, and a dilation rate of 8, a ReLU layer, 1 convolution kernel size of 3×3, an output channel number of 64, a ReLU layer, and a convolution layer with a convolution kernel size of 1×1 and an output channel number of 1;

[0062] Shared weight map generation layer: from input to output, it includes 1 convolution layer with a convolution kernel size of 3×3 and 256 output channels, 1 ReLU layer, 1 convolution layer with a convolution kernel size of 3×3 and 256 output channels, 1 ReLU layer, 1 convolution layer with a convolution kernel size of 3×3 and 128 output channels, 1 ReLU layer, 1 convolution layer with a convolution kernel size of 1×1 and 1 output channel;

[0063] Multi-view feature fusion decoding layer: from input to output, it includes 1 convolution layer with a convolution kernel size of 3×3 and an output channel number of 512, a ReLU layer, 1 convolution layer with a convolution kernel size of 3×3, an output channel number of 512, a padding size of 2, and a dilation rate of 2, a ReLU layer, and 1 convolution layer with a convolution kernel size of 3×3, an output channel number of 1, a padding size of 4, and a dilation rate of 4.

[0064] Relying on the above system, a multi-view scale perception pedestrian detection method can be implemented, such as Figure 2 As shown in the figure, multiple visible light images of pedestrians captured by multiple cameras in the same scene are input into the constructed multi-view pedestrian detection system for direct end-to-end training. The real multi-view pedestrian head position map and foot position map, the real single-view scene-level pedestrian occupancy map, and the real multi-view scene-level pedestrian occupancy map are used as supervision to calculate the loss. The multi-view pedestrian detection system is optimized using the gradient descent optimization algorithm to obtain the optimal parameters of the multi-view pedestrian detection system. The specific steps are as follows:

[0065] Step 1: Obtain camera parameters, multi-view visible light images from the same scene, real multi-view pedestrian head position maps, real multi-view pedestrian foot position maps, and real multi-view scene-level pedestrian occupancy maps;

[0066] Step 2: Input multiple viewing angle visible light images into the initial feature extraction layer in sequence to extract initial features, and obtain multiple viewing angle initial feature maps respectively;

[0067] Step 3: Input the initial feature maps of multiple viewpoints into the single view two-dimensional detection layer in sequence for two-dimensional detection, and obtain the pedestrian head position map and foot position map of multiple viewpoints;

[0068] Step 3.1: Input the initial feature maps of multiple viewpoints into the single view 2D detection layer in sequence for 2D detection, and obtain the predicted pedestrian head position maps and foot position maps of multiple viewpoints;

[0069] Step 3.2: Calculate the loss of the predicted pedestrian head position map and foot position map from multiple perspectives and the real pedestrian head position map and foot position map from multiple perspectives. Here, the supervision information loss term of single-view two-dimensional detection is defined as follows:

[0070]

[0071] in, and They are respectively the pedestrian head position map and foot position map of the predicted camera view c; and are the pedestrian head position map and foot position map of the real camera view c; Gaussian kernel f(·) is used for blurring; ||·||2 represents the calculation of Euclidean distance. On the one hand, the foot detection loss generates homogeneous supervision on the ground plane. On the other hand, the head detection loss can generate heterogeneous supervision, thereby further improving the system performance. The head and foot detection losses can provide additional supervision for the output of the main task of the system, thereby further improving the system performance.

[0072] Step 4: Project multiple view initial feature maps in sequence to obtain multiple view projection feature maps;

[0073] Step 5: Input multiple view projection feature maps into the multi-view scale-aware feature extraction module in sequence to extract scale-aware features, obtain multiple view scale-aware feature maps and connect them by channel;

[0074] Step 5.1: Input multiple view projection feature maps into the four-scale parallel structure in sequence to obtain the intermediate scale perception feature map; input the multi-view projection feature map F obtained in step 4 p Input four scale parallel structures in sequence to calculate multi-scale features The multi-scale features obtained by the parallel structure with input scale i Calculated as:

[0075]

[0076] Where, for each scale i of the camera view c, P ave (·,i) average the projection features of each view into k(i)×k(i) blocks; is a convolutional network with a kernel size of 1, which can combine context features across channels without changing its dimensionality. B represents bilinear interpolation, which is used to upsample the context feature array to In practice, we choose S = 4 different scales with corresponding block sizes k(i)∈{1,2,3,6}, as this setting shows better performance.

[0077] The scale-aware features are combined with the original multi-view projection feature map Concatenation is the simplest approach, but this does not take into account the scale differences of different images. To address this problem, a learning prediction weight map is used to determine the relative influence of each scale perceptual feature. To do this, first define the contrast feature:

[0078]

[0079] These features capture the difference between a specific location and its neighboring locations and are used to learn the weights of scale-aware features, which are often important visual cues for saliency. These contrast features provide important information to understand the local scale of each image region. Therefore, they are used as auxiliary networks The input weight is Used to calculate the weights assigned to each different scale Each such network outputs a weight map of a specific scale.

[0080]

[0081] is a 1×1 convolutional layer followed by a sigmoid function to avoid division by zero. These weights are then used to compute our intermediate scale perceptual features as

[0082]

[0083] Here, [·|·] represents the channel-level connection operation, which is the element-wise product between the weight map and the feature map.

[0084] Step 5.2: Concatenate the intermediate scale-aware feature map with the original features by channel, and finally use a 1×1 convolution layer to reduce the number of channels to the number of original feature channels to obtain the scale-aware feature map of the camera view. Concatenate the scale-aware feature maps of all views by channel to obtain the final multi-view scale-aware feature map. A multi-view scale-aware module is designed for all feature maps of each view to extract scale-aware contextual information for each view. The weights of the subnetwork are shared among all input camera views to handle possible variable camera views.

[0085] Step 6: Input multiple view projection feature maps into the shared weight supervision module in sequence to obtain a shared weight map;

[0086] Step 6.1: Input multiple view projection feature maps into the projected single view feature decoding layer in sequence for single view feature decoding to obtain the predicted single view scene-level pedestrian occupancy map. Calculate the loss between the predicted single view scene-level pedestrian occupancy map and the real single view scene-level pedestrian occupancy map. Here, the pedestrian position supervision information loss term of the single view projection feature is defined as follows:

[0087]

[0088] in, is the predicted single-view scene-level pedestrian occupancy map in camera view c; g c is the pedestrian occupancy map at the single-view scene level in the real camera view c; g is the pedestrian occupancy map at the real multi-view scene level; M c is the mask for single-view scene-level detection in camera view c, which can be obtained from the camera internal parameters; g c It can be obtained by g and M c Multiply to obtain; is element-wise multiplication.

[0089] Step 6.2: Connect the single-view scene-level pedestrian occupancy maps of each view by channel and input them into the shared weight map generation layer to obtain the shared weight map;

[0090] Step 7: Multiply the multiple view-scale perception feature maps with the shared weight map to obtain weighted multiple view-scale perception feature maps;

[0091] Step 8: Input the weighted multi-view scale perception feature map into the multi-view feature decoding layer for multi-view feature decoding to obtain a multi-view scene-level pedestrian occupancy map. Input the weighted multi-view scale perception feature map into the multi-view feature decoding layer for multi-view feature decoding to obtain a predicted multi-view scene-level pedestrian occupancy map. Calculate the loss between the predicted multi-view scene-level pedestrian occupancy map and the actual multi-view scene-level pedestrian occupancy map. The final output multi-view scene-level pedestrian occupancy map supervision information loss term is defined as follows:

[0092]

[0093] The overall loss for training is defined as follows:

[0094]

[0095] Among them, α and β are hyperparameters used to adjust the importance of the two losses in training; C is the number of camera views.

[0096] like Figure 3 FIG. 1 is a schematic diagram of using the multi-view pedestrian detection method of the present invention to perform scene-level pedestrian detection. Figure 3 When the multi-view pedestrian detection method of the present invention is used for pedestrian detection, multiple pedestrian visible light images captured simultaneously by multiple cameras of the target scene are input into the established multi-view scale-aware pedestrian detection system, and the trained and optimized multi-view pedestrian detection system is used to perform scene-level pedestrian detection prediction. The obtained predicted multi-view scene-level pedestrian occupancy map can display the position of pedestrians in the scene.

[0097] Aiming at the problems of continuous scale changes in existing multi-view pedestrian detection and insufficient use of pedestrian position information of single-view projection features, the present invention proposes a multi-view scale-aware pedestrian detection method and system, which can effectively solve the problem of continuous scale changes in multi-view pedestrian detection and make more full use of pedestrian position information of single-view projection features. By extracting features from multiple visible light images of pedestrians captured by multiple cameras in the same scene, the obtained initial feature maps of multiple perspectives are input into the single-perspective two-dimensional detection layer for two-dimensional detection, and the predicted pedestrian head position map and foot position map of multiple perspectives are compared with the actual pedestrian head position map and foot position map of multiple perspectives for supervision calculation loss; then the initial feature maps of multiple perspectives are projected to obtain multiple perspective projection feature maps; on the one hand, the multiple perspective projection feature maps are input into the shared weight supervision module to obtain the predicted single-perspective scene-level pedestrian occupancy map and shared weight; the predicted single-perspective scene-level pedestrian occupancy map and the actual single-perspective scene-level pedestrian occupancy map are compared with the actual single-perspective scene-level pedestrian occupancy map for supervision calculation loss, so that the pedestrian position information of the single-perspective projection feature is more fully utilized; on the other hand, the multiple perspective projection feature maps are input into the multi-perspective scale-aware feature extraction module for scale-aware feature extraction, and the obtained multiple perspective scale-aware feature maps integrate the contextual information of the continuously changing scale, thereby improving the detection accuracy of the multi-perspective pedestrian detection network.

[0098] The embodiments described above are merely descriptions of preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-view scale perception pedestrian detection system, characterized by: It includes an initial feature extraction layer, on which a single-view two-dimensional detection layer, a multi-view scale-aware feature extraction module and a shared weight supervision module are simultaneously connected, and a common multi-view feature decoding layer is connected to the multi-view scale-aware feature extraction module and the shared weight supervision module, wherein the shared weight supervision module includes a single-view projection feature decoding layer and a shared weight map generation layer.

2. The multi-view scale perception pedestrian detection system according to claim 1, characterized in that: The specific structure of each layer is as follows: Initial feature extraction layer: from input to output, it includes 2 convolutional layers, 1 maximum pooling layer, 2 convolutional layers, 1 maximum pooling layer, 4 convolutional layers, 1 maximum pooling layer, 4 convolutional layers, 1 maximum pooling layer, and 4 convolutional layers; Single-view 2D detection layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, and 1 convolution layer; Multi-view scale-aware feature extraction module: from input to output, it includes four scale parallel structures of different scales, one channel connection operation and one convolution layer. Among them, the first scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the second scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the third scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; the fourth scale is 1 average pooling layer, 1 convolution layer, 1 upsampling layer, 1 subtraction operation, 1 convolution layer, and 1 multiplication operation from input to output; Single-view projection feature decoding layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer; Shared weight map generation layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, 1 convolution layer; Multi-view feature fusion decoding layer: from input to output, it includes 1 convolution layer, 1 ReLU layer, 1 convolution layer, 1 ReLU layer, and 1 convolution layer.

3. A multi-view scale perception pedestrian detection method, characterized in that Applied to the system as claimed in claim 1 or 2, characterized in that: the specific steps are as follows: Step 1: Obtain camera parameters, multi-view visible light images from the same scene, real multi-view pedestrian head position maps, real multi-view pedestrian foot position maps, and real multi-view scene-level pedestrian occupancy maps; Step 2: Input multiple viewing angle visible light images into the initial feature extraction layer in sequence to extract initial features, and obtain multiple viewing angle initial feature maps respectively; Step 3: Input the initial feature maps of multiple viewpoints into the single view two-dimensional detection layer in sequence for two-dimensional detection, and obtain the pedestrian head position map and foot position map of multiple viewpoints; Step 4: Project multiple view initial feature maps in sequence to obtain multiple view projection feature maps; Step 5: Input multiple view projection feature maps into the multi-view scale-aware feature extraction module in sequence to extract scale-aware features, obtain multiple view scale-aware feature maps and connect them by channel; Step 6: Input multiple view projection feature maps into the shared weight supervision module in sequence to obtain a shared weight map; Step 7: Multiply the multiple view-scale perception feature maps with the shared weight map to obtain weighted multiple view-scale perception feature maps; Step 8: Input the weighted multi-view scale perception feature maps into the multi-view feature decoding layer for multi-view feature decoding to obtain a multi-view scene-level pedestrian occupancy map.

4. The multi-view scale perception pedestrian detection method according to claim 3, characterized in that: Step 3 is as follows: Step 3.1: Input the initial feature maps of multiple viewpoints into the single view 2D detection layer in sequence for 2D detection, and obtain the predicted pedestrian head position maps and foot position maps of multiple viewpoints; Step 3.2: The loss between the predicted pedestrian head position map and foot position map of multiple perspectives and the actual pedestrian head position map and foot position map of multiple perspectives is calculated as: in, and They are respectively the pedestrian head position map and foot position map of the predicted camera view c; and They are the pedestrian head position map and foot position map of the real camera view c respectively; the Gaussian kernel f(·) is used for blurring; ||·||2 indicates the calculation of the Euclidean distance.

5. The multi-view scale perception pedestrian detection method according to claim 3, characterized in that: Step 5 The specific steps are as follows: Step 5.1: Input multiple view projection feature maps into the four-scale parallel structure in sequence to obtain the intermediate scale perception feature map; Step 5.2: Connect the intermediate scale-aware feature map and the original projection feature map by channel and then input them into the 1×1 convolutional layer to obtain the multi-view scale-aware feature map.

6. The multi-view scale perception pedestrian detection method according to claim 5, characterized in that: Step 5.1 The specific steps are as follows: Step 5.11: Project the multi-view feature map F obtained in step 3 p Input four scale parallel structures in sequence to calculate multi-scale features The multi-scale features obtained by the parallel structure with input scale i Calculated as: Where, for each scale i of the camera view c, P ave (·,i) average the projection features of each view into k(i)×k(i) blocks; is a convolutional network with a kernel size of 1, which can combine context features across channels without changing its dimension. B represents bilinear interpolation, which is used to upsample the context feature array to Same size; Step 5.12: Define contrast features: Step 5.13: In the multi-view scale-aware feature extraction module, the contrast feature provides important information to understand the local scale of each image region. The contrast feature is used as an auxiliary network The input weight is defined as Used to calculate the weight map assigned to each different scale The weight map of scale i for camera view c is defined as follows: is a 1×1 convolutional layer followed by a sigmoid function to avoid division by zero, and uses weights to calculate the intermediate scale perceptual features: Here, [·|·] represents the channel-level connection operation, which is the element-wise product between the weight map and the feature map.

7. The multi-view scale perception pedestrian detection method according to claim 3, characterized in that: Step 6 The specific steps are as follows: Step 6.1: Input the multi-view projection feature maps into the projected single-view feature decoding layer in sequence for single-view feature decoding to obtain the predicted single-view scene-level pedestrian occupancy map. Calculate the loss between the predicted single-view scene-level pedestrian occupancy map and the real single-view scene-level pedestrian occupancy map. The pedestrian position supervision information loss term of the single-view projection feature is defined as follows: in, is the predicted single-view scene-level pedestrian occupancy map in camera view c; g c is the pedestrian occupancy map at the single-view scene level in the real camera view c; g is the pedestrian occupancy map at the real multi-view scene level; M c is the mask for single-view scene-level detection in camera view c, which can be obtained from the camera internal parameters; g c It can be obtained by g and M c Multiply to obtain; is element-wise multiplication; Step 6.2: Connect the single-view scene-level pedestrian occupancy maps of each view by channel and then input them into the shared weight map generation layer to obtain the shared weight map.

8. The multi-view scale perception pedestrian detection method according to claim 3, characterized in that: Step 8 is as follows: The weighted multi-view scale perception feature maps are input into the multi-view feature decoding layer for multi-view feature decoding to obtain the predicted multi-view scene-level pedestrian occupancy map. The predicted multi-view scene-level pedestrian occupancy map and the real multi-view scene-level pedestrian occupancy map are compared to calculate the loss. The supervision information loss term of the final output multi-view scene-level pedestrian occupancy map is defined as follows: The overall loss for training is defined as follows: Among them, α and β are hyperparameters used to adjust the importance of the two losses in training; C is the number of camera views.

Images