A dense vehicle counting method based on deep spatio-temporal network

Abstract

The application discloses a dense vehicle counting method based on a deep space-time network, and comprises the following steps: sending a counting image to the first ten layers of VGG-16 through MSCNet to extract initial image features; adopting a direction-based perspective coding module to code global features in four directions; designing a multi-scale hollow residual module for extracting different scale feature information, combining different layers in the module through a cascading method to form multiple hollow convolution rates, and solving the gradient disappearance problem caused by the deep network through residual connection; and learning the weight of different channel features through a channel perception attention module to enhance the vehicle features extracted in a mixed scene. The application counts vehicles through MSCNet, which is a single-column multi-branch network based on CNN and adopts an end-to-end cascading framework, can effectively mine the space-time data in traffic jam images, predict a high-quality density map and calculate the number of vehicles, and has high counting precision for vehicles.

Description

Technical Field

[0001] This invention relates to the field of vehicle counting technology, and more specifically to a dense vehicle counting method based on deep spatiotemporal networks. Background Technology

[0002] Object counting is a crucial component of scene understanding. It is used to count the number of objects in images or videos, such as vehicle counts in traffic scenes, crowd counts in public congestion scenes, and specific bacteria or cell counts in microscopic scenes. Object counting has wide applications in spatiotemporal traffic flow prediction, public safety management, and biological and medical research. In smart cities, vehicle counting plays a vital role in the urban planning and management of Intelligent Transportation Systems (ITS). Vehicle counting research is a multidisciplinary research direction that integrates computer vision, machine learning, deep learning, data mining, and pattern recognition.

[0003] Existing technologies have the following shortcomings: accurately estimating the number of dense crowds or vehicles in an image is a meaningful research task, which has been widely applied in many fields such as safety management and traffic prediction. Existing CNN (Convolution Neural Networks) based methods have poor feature extraction capabilities for long queues of vehicles in complex spatiotemporal scenarios, and the results are not significant. Therefore, we propose a dense vehicle counting method based on deep spatiotemporal networks to achieve accurate vehicle counting. Summary of the Invention

[0004] The purpose of this invention is to provide a dense vehicle counting method based on deep spatiotemporal networks to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a dense vehicle counting method based on deep spatiotemporal networks, the counting method comprising the following steps:

[0006] S1: Send the counted images to the first ten layers of VGG-16 via MSCNet to extract initial image features;

[0007] S2: A direction-based perspective coding module is used to encode global features in four directions;

[0008] S3: Design a multi-scale dilated residual module to extract feature information at different scales. By using a cascading method, different layers are combined within the module to form multiple dilated convolution rates. The gradient vanishing problem caused by deep networks is solved by using residual connections.

[0009] S4: Sampling and counting of vehicle targets at the range scale using a multi-scale void residual module;

[0010] S5: By learning the weights of different channel features through the channel-aware attention module, the vehicle features extracted in mixed scenarios are enhanced. By combining local and global features, features hidden in different channels are extracted.

[0011] In a preferred embodiment, the MSCNet includes a backbone network VGG16 and three modules: a direction-based perspective coding module DPCM, a multi-scale hole residual module MDRM, and a channel-aware attention module CAM.

[0012] In a preferred embodiment, step S4, sampling and counting vehicle targets at the range scale, includes the following steps:

[0013] S4.1: Utilize DPCM and MDRM to perform in-depth mining of spatiotemporal information;

[0014] S4.2: DPCM encodes perspective information in four directions;

[0015] S4.3: Scale features of dense extraction range variation in MDRM;

[0016] S4.4: CAM mines channel correlations to enhance vehicle counting features in human-vehicle hybrid scenarios.

[0017] Our method directly converts the input image into a high-quality estimated density map, and then integrates to obtain the total number of vehicles.

[0018] In a preferred embodiment, in step S1, the first 10 layers and 3 pooling layers of the VGG16 network are retained as the backbone network.

[0019] The model first sends the input image A to the pre-trained VGG16 backbone network to obtain the feature map f. b As shown in equation (1):

[0020] f b =F VGG (A) (1)

[0021] VGG16 was used to extract initial features from image A.

[0022] In a preferred embodiment, the DPCM processes long-distance continuous perspective changes, and the DPCM encodes perspective information in four directions;

[0023] DPCM uses the feature map f output from VGG16 b As input, it includes four convolutional layers: left, right, lower, and upper.

[0024] Each convolutional layer consists of a convolutional kernel of size C×ω and a ReLU activation function. The output of this kernel is superimposed on the next slice to form a new slice. The convolutional layer operation continues until all slices have been processed. The feedforward calculation is defined as shown in Equation (2):

[0025]

[0026] Among them, L W (*) indicates the left-hand operation (Conv+ReLU), f W i This is the i-th feature slice along the W direction;

[0027] Perspective information is encoded from four directions, and feature maps f are output. c .

[0028] In a preferred embodiment, the MDRM is used to extract continuously varying scale features in an image. The MDRM contains three dilated convolutional layers with dilation rates of 1, 2, and 3, respectively. A 1×1 filter-sized convolutional layer is added before each dilated convolutional layer.

[0029] A standard convolutional kernel with a 3×3 filter size is used to fuse all the connection features from the previous dilated convolutional layers, reducing the number of channels at the final output, and a ReLU activation function is applied after each dilated convolutional layer;

[0030] Each dilated convolutional layer in M ​​is tightly connected to other dilated convolutional layers, and each layer can access all subsequent layers and transmit the feature information that needs to be extracted.

[0031] After dense and continuous layer connections, the module gathers multiple receptive fields, resulting in increased scale diversity.

[0032] In a preferred embodiment, in dilated convolution, a small kernel with a k×k filter is amplified to k+(k-1)×(r-1), and the MDRM module has 3×3 convolution kernels with 3 different dilation rates, which are combined to form 6 receptive fields of different sizes.

[0033] The most important method for the MDRM module to utilize multi-scale information is to combine the multi-scale features with the original input features f. c Connecting them together adds the input characteristics to the output characteristics, i.e., f d ;

[0034] The function C(*) represents four feature maps f c The concatenation of f1, f2, and f3 is performed, and a 3×3 convolution (dilation rate = 1) operation is executed to complete channel fusion, as shown in equation (3):

[0035] fd =f c +C([f c (3)

[0036] After performing multi-scale dilated convolution operations, the initial feature information is preserved, and the image features f are integrated. c and multi-scale feature information.

[0037] In a preferred embodiment, after DPCM and MDRM, the output characteristic f d Sent to CAM;

[0038] DPCM and MDRM extract continuous multi-scale spatial information and perform feature fusion in the spatial dimension;

[0039] CAM extracts features hidden in different channels of the feature map along the channel dimension, obtains the weight of each feature channel through learning, enhances the counting features through the weights, and suppresses features that are not important to the counting task. The whole process is expressed as Equation (4):

[0040] f e =f d +Φ(GAP(f d ,C))⊙f d (4)

[0041] Feature map f d The size is W×H×C, and global average pooling is used to calculate f. d Divide into 1×1×C blocks on average.

[0042] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0043] 1. This invention counts vehicles using MSCNet, a single-column multi-branch CNN-based network with an end-to-end cascaded framework. It can effectively mine spatiotemporal data in traffic congestion images, predict and generate high-quality density maps, and calculate the number of vehicles, resulting in high accuracy in vehicle counting.

[0044] 2. This invention uses DPCM to handle long-distance continuous perspective changes. It performs slicing and convolution operations in four directions to extract as much spatial information as possible that is hidden due to perspective. The density features of vehicle distribution are also hidden in the perspective information, effectively encoding the perspective change information in traffic congestion scenarios. It is particularly suitable for extracting the target features of continuous long-distance shapes, such as crowded queues of vehicles with strong spatial relationships but poor appearance cues.

[0045] 3. This invention obtains six receptive fields of different sizes by combining three void ratios, enabling more intensive detection of image features at different scales. The combination of DPCM and MDRM can better extract features of long-distance continuous large-scale changes in dense vehicles.

[0046] 4. This invention introduces global information through GAP, and some channels show high activation modes, which are related to the specific vehicle features that need to be learned. Therefore, the module can accurately extract important channel features, generate high-quality prediction density maps, and obtain a more accurate number of vehicles in human-vehicle mixed scenarios. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0048] Figure 1 This is an architecture diagram of the MSCNet for vehicle counting in this invention.

[0049] Figure 2 This is a diagram of the orientation-based perspective coding module of the present invention.

[0050] Figure 3 This is a schematic diagram of the multi-scale void residual module of the present invention.

[0051] Figure 4 This is a schematic diagram of the channel-aware attention module of the present invention.

[0052] Figure 5 This is a schematic diagram showing a visual comparison between the actual ground values ​​and the estimated results of this invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1

[0055] Please see Figure 1 As shown in this embodiment, a dense vehicle counting method based on deep spatiotemporal networks is described. The counting method includes the following steps:

[0056] The counting image is sent to the first ten layers of VGG-16 using MSCNet (Multi-Scale Dilated Convolutional Channel-Aware Network) to extract initial image features. To handle continuous perspective changes, a direction-based perspective coding module is employed to encode global features in four directions. A multi-scale dilated residual module is designed to extract feature information at different scales. A flexible cascading method is used to combine different layers within the module to form multiple dilated convolution rates, and residual connections are used to solve the gradient vanishing problem caused by deep networks. This module can sample targets at very large scales in a denser manner, thus handling the large-scale variation problem in vehicle counting. In addition, the method utilizes a channel-aware attention module to learn the weights of different channel features to enhance the vehicle features that need to be extracted in mixed scenes. This module can combine local and global features to extract important features hidden in different channels. Multiple experimental results based on benchmark datasets show that the proposed MSCNet network method for vehicle counting outperforms existing methods.

[0057] The specific counting operation logic is as follows:

[0058] 1) By utilizing DPCM and MDRM to deeply mine spatiotemporal information, the problem of long-distance continuous scale changes of queuing vehicles in vehicle counting is solved. DPCM encodes perspective information in four directions, which is suitable for extracting features of long-distance continuous changes; MDRM intensively extracts scale features of large-scale changes. This module combines three dilated convolution rates to obtain six receptive fields of different sizes.

[0059] 2) CAM is a module with channel awareness capabilities. This module can delve into channel correlations to enhance the useful features of vehicle counting in human-vehicle hybrid scenarios.

[0060] MSCNet is an end-to-end single-column multi-branch CNN model with broad scale diversity, capable of handling large-scale scale variations caused by continuous perspective effects, and fusing feature information from multiple channels.

[0061] The network model mainly consists of a backbone network (VGG16) and three modules: a direction-based perspective coding module (DPCM), a multi-scale holed residual module (MDRM), and a channel-aware attention module (CAM). The network structure is as follows: Figure 1 As shown.

[0062] Referring to the CSRNet network structure, we retain the first 10 layers and 3 pooling layers of the VGG16 network as our backbone network.

[0063] The model first sends the input image A to the pre-trained VGG16 backbone network to obtain the feature map f. bAs shown in equation (1):

[0064] f b =F VGG (A) (1)

[0065] VGG16 is used to extract initial features from image A. Its strong transfer learning ability is used in dense vehicle analysis. However, the features extracted by VGG16 have limitations. It only encodes the receptive field of a limited area in the entire image. This application attempts to deploy DPCM, MDRM and CAM to extract deeper salient spatiotemporal information and enhance the initial features.

[0066] Our method directly converts the input image into a high-quality estimated density map, and then integrates to obtain the total number of vehicles.

[0067] Example 2

[0068] Using DPCM to handle long-distance continuous perspective changes, such as Figure 2 As shown, DPCM encodes perspective information in four directions.

[0069] This module uses the feature map f output from VGG16. b The input consists of four convolutional layers: left, right, lower, and upper.

[0070] exist Figure 2 In the diagram, C, H, and W represent the channel, length, and width of the feature map. The tensor is first divided into W feature slices in the left (and right) layers, and then the first slice is sent to the convolutional layer.

[0071] Each convolutional layer consists of a convolutional kernel of size C×ω and a ReLU activation function. Traditional CNNs pass the output of this layer to the next layer for convolutional layer operations. However, this application superimposes the output with the next slice to form a new slice, and then continues to use convolutional layer operations until all slices have been processed. To illustrate the operation intuitively, the left layer is used as an example. The feedforward calculation is defined as shown in formula (2):

[0072]

[0073] Among them, L W (*) indicates the left-hand operation (Conv+ReLU), f W i Let be the i-th feature slice in the W direction.

[0074] Similarly, the other three layers (right, bottom, and top) operate identically except for the sliding and slicing directions. This structure is primarily used to encode perspective information from four directions and output feature maps f. c .

[0075] This module performs slicing and convolution operations in four directions to extract as much spatial information as possible that is hidden due to perspective effects. The density features of vehicle distribution are also hidden in the perspective information.

[0076] DPCM allows perspective information to be transferred between rows and columns of an image. The lower and upper layers transfer information along the H direction from top to bottom and from bottom to top, while the left and right layers transfer information along the W direction from right to left and from left to right.

[0077] Since perspective information needs to be transmitted between multiple rows and columns, this module also uses the residual method for learning, taking advantage of the advantages of residual networks. In addition, this module does not change the channel dimension of the input features and introduces global spatial information into the feature map.

[0078] In summary, these operations effectively encode perspective change information in traffic congestion scenarios, and are particularly suitable for extracting target features of continuous long-distance shapes, such as crowded queues of vehicles with strong spatial relationships but poor appearance cues.

[0079] Example 3

[0080] Large-scale scale variations are a major problem in dense vehicle counting. The scale variations of crowded vehicles are almost continuous across the entire image and cover a large range, making it essential to continuously extract scale variation features from the scene.

[0081] like Figure 3 As shown, MDRM is mainly used to extract continuously varying scale features in an image. This module contains three dilated convolutional layers with dilation rates of 1, 2 and 3, respectively. A 1×1 filter-sized convolutional layer is added before each dilated convolutional layer.

[0082] A standard convolutional kernel with a 3×3 filter size (dilation rate = 1) is used to fuse all the connection features from the previous dilated convolutional layers, reducing the number of channels at the end output, and a ReLU activation function is applied after each dilated convolutional layer.

[0083] Each dilated convolutional layer in the MDRM module is tightly connected to other dilated convolutional layers, and each layer can access all subsequent layers and transmit the feature information that needs to be extracted.

[0084] After dense and continuous layer connections, the module gathers multiple receptive fields, resulting in increased scale diversity, as shown in Table 1:

[0085]

[0086] Table 1

[0087] These three void ratios combine to form six different sizes of receptive fields.

[0088] For example, when the void ratio is 1, the receptive field size is 3×3; when the void ratio combination is 1 and 3, the receptive field size is 9×9.

[0089] Dilated convolution has been widely used and proven in segmentation tasks, and its model accuracy has been significantly improved. Compared with pooling layers, dilated convolution can preserve the spatial information of feature maps, while deconvolution layers can reduce the loss of spatial information, but increase complexity and cause execution delay.

[0090] Dilated convolution is a better option, as it uses sparse kernels instead of pooling and deconvolution layers, expanding the receptive field without increasing the number of parameters.

[0091] In dilated convolution, a small kernel with a k×k filter is amplified to k+(k-1)×(r-1). The MDRM module has 3×3 convolution kernels with 3 different dilation rates, which are combined to form 6 receptive fields of different sizes. They extract large-scale feature information in a dense and continuous manner. In addition, the MDRM module also designs residual connections to deal with the gradient vanishing problem.

[0092] In this module, the most important method for utilizing multi-scale information is to combine the multi-scale features with the original input features f. c Connecting the input characteristics, this operation adds the input characteristics to the output characteristics, i.e., f d .

[0093] The function C(*) represents four feature maps f c The f1, f2, and f3 are connected and a 3×3 convolution (hole rate = 1) operation is performed to complete channel fusion.

[0094] The operation is shown in equation (3): f d =f c +C([f c (3)

[0095] After performing multi-scale dilated convolution operations, more initial feature information is retained, thus enabling the integration of image features f. c and multi-scale feature information.

[0096] MDRM obtains six receptive fields of different sizes by combining three hole rates, enabling more dense detection of image features at different scales. The combination of DPCM and MDRM can better extract features of long-distance continuous large-scale changes in dense vehicles.

[0097] Example 4

[0098] After DPCM and MDRM, the output characteristic f d It is sent to the CAM.

[0099] like Figure 4 As shown, DPCM and MDRM extract continuous multi-scale spatial information and complete feature fusion in the spatial dimension, while CAM considers feature fusion in the channel dimension. Some works have also used such a module, which can extract important features hidden in different channels of the feature map.

[0100] The model learns the weights for each feature channel, then uses these weights to enhance useful counting features and suppress features that are not important to the counting task. The whole process can be expressed as equation (4): f e =f d +Φ(GAP(f d ,C))⊙f d (4)

[0101] Feature map f d The size is W×H×C (width×height×channel), and global average pooling (GAP) is used to divide f. d Divide into 1×1×C blocks on average.

[0102] The Avg function averages the information of all pixels in the space as a single value, masking the spatial distribution information. This operation encodes the entire spatial feature on each channel as a global feature, and then uses the function φ(*) to process the average block, which includes a bottleneck structure with two fully connected layers, a ReLU function and a Sigmoid function, with the aim of obtaining the correlation between channels.

[0103] This operation focuses on the correlation between channels, rather than the correlation in spatial distribution, and learns the weighting coefficients W. d (Sigmoid function, 0-1), then multiply the weight coefficient of each channel by f. d The original features on the channel dimension are used to enhance the original features on the channel dimension, and then f o With input features f d Connect them to enhance the representation of the feature map.

[0104] After CAM processing, the enhanced features f will be finally... e The data is passed to a decoder consisting of multiple convolutional layers to generate a predicted density map. Essentially, CAM performs attention operations on the channel dimension. This attention mechanism enables the model to pay more attention to channels with more feature information and suppress those channels that are not important.

[0105] Different characteristic channels introduce global information through GAP. Some channels show high activation modes, which are related to the specific vehicle features that need to be learned. Therefore, the module can accurately extract important channel features, generate high-quality prediction density maps, and obtain a more accurate number of vehicles in human-vehicle mixed scenarios.

[0106] Example 5

[0107] The MSCNet model was built in an experimental environment using Ubuntu 18.04 and PyTorch 11.1, with Intel Core processors as the primary hardware support. TM The system uses an i7 12700 4.7GHz processor and a GeForce RTX 3090. The task uses the L2 loss function and the Adam optimizer with a learning rate of 0.00001.

[0108] The model was tested using common evaluation metrics based on density plot counting: mean absolute error (MAE) and root mean square error (RMSE), defined as in formulas (5) and (6):

[0109]

[0110]

[0111] Where N is the total number of test images, C i GT C is the actual number of ground-based counts. i This is the prediction count. The grid mean absolute error (GAME) is also used for model evaluation during testing, and GAME is defined as in formula (7):

[0112]

[0113] Where N is the total number of test images, It is the predicted count of the input image within region l. This represents the ground truth count for the corresponding area. For each level L, GAME(L) uses 4. L Image segmentation is performed by covering the entire image with non-overlapping grids. Specifically, when L=0, GAME is equivalent to MAE.

[0114] This application compares with state-of-the-art methods on five different public datasets: the TRANCOS dataset, the VisDrone2021 crowd and vehicle dataset, and the ShanghaiTechPat_A&Pat_B dataset.

[0115] Experimental results show that the proposed MSCNet achieves satisfactory target counting capabilities on all five datasets. Table 2 shows the relevant statistics for the five datasets:

[0116]

[0117] Table 2

[0118] The TRANCOS dataset is a vehicle counting dataset for traffic congestion scenarios. It is a public transportation dataset consisting of images captured by road monitoring cameras, containing 1,244 images from different perspectives of traffic congestion scenarios and 46,796 annotated vehicles.

[0119] The VisDrone2021 crowd and vehicle datasets were derived from an object detection dataset with annotated bounding boxes. We modified the original datasets into two object counting datasets: the VisDrone2021 vehicle dataset and the VisDrone2021 crowd dataset.

[0120] The original dataset contains 11 object categories. We select objects from the "pedestrian" and "person" categories to form a crowd dataset (VisDrone2021 crowd dataset). The annotation operation for the new crowd dataset is defined by equation (8): People[X, Y] = [bbox left +bbox width / 2, bbox top (8)

[0121] Similar to the steps for creating the VisDrone2021 crowd dataset, we combine the categories of "car", "van", "truck", and "bus" to construct a vehicle dataset (VisDrone2021 vehicle dataset). The annotation operation for the new vehicle dataset is defined as Equation (9): Vehicle[X, Y] = [bbox left +bbox width / 2, bbox top +bbox height / 2] (9)

[0122] The ShanghaiTech dataset contains 1,198 images and a total of 330,165 annotated individuals. The dataset consists of two parts: Pat_A, which contains 482 images, and Pat_B, which contains 716 images. ShanghaiTechPat_A is a dataset with extremely crowded scenes and different perspectives, with a total of 241,677 annotated individuals. ShanghaiTechPat_B is a dataset with relatively sparse crowd scenes, with a total of 88,488 annotated individuals.

[0123] Table 3 shows the comparison results of MSCNet with other methods on the TRANCOS dataset:

[0124]

[0125] Table 3

[0126] The TRANCOS dataset is a challenging dataset with severe vehicle occlusion problems and highly variable perspectives, but existing methods have achieved good performance.

[0127] Our method achieves significant improvements on four different levels of GAME, with scores of 3.49 for GAME(0), 5.11 for GAME(1), 7.81 for GAME(2), and 14.01 for GAME(3), outperforming other methods. GAME(0) is equivalent to MAE.

[0128] We also found that the values ​​of MSCNet and CSRNet on GAME(0) are very close. It can be inferred that the output of dilated convolution contains more detailed vehicle feature information, the dilated convolution of MDRM can focus more on extracting vehicle features with large-scale changes, while DPCM and MDRM can be used to solve the problem of long-distance continuous scale changes in images.

[0129] Table 4 shows the comparison results of MSCNet with other methods on the VisDrone2021 crowd and vehicle datasets:

[0130]

[0131]

[0132] Table 4

[0133] In addition to vehicle counting, we also conducted experiments on a population dataset to demonstrate the robustness and generality of our proposed method. The vehicle dataset is a challenging dataset, with different types and numbers of vehicles in each image.

[0134] Experimental results show that MSCNet can handle this more complex dataset. Our method consistently demonstrates better counting performance than other methods, illustrating the importance of the Channel Attention Module (CAM) for more complex datasets. CAM can learn channel correlations to enhance useful features for vehicle counting in complex scenes. In the crowd dataset, which was collected in an outdoor scene with complex backgrounds and a wide range of crowd density distribution in the images.

[0135] Experimental results show that MSCNet can also handle this type of population dataset with large-scale scale variations.

[0136] Table 5 shows the comparison results of MSCNet with other methods on two challenging crowd counting datasets, Part A and Part B of the ShanghaiTech dataset:

[0137]

[0138] Table 5

[0139] The scale and perspective variations presented by these two datasets pose a challenge to many CNN-based counting models, with the target density of ShanghaiTechPart_A being greater than that of ShanghaiTechPart_B.

[0140] It can be observed that MSCNet achieves the best MAE and RMSE in Part_A, with an MAE of 58.9 and an RMSE of 94.1. In Part_B, MSCNet achieves an MAE of 7.5 and an RMSE of 11.2.

[0141] Compared with other methods, the average performance of this model is better. In particular, MSCNet has a significant performance improvement over MCNN. In addition, our method outperforms the state-of-the-art methods CSRNet and PCCNet on both Part_A and Part_B datasets. These results further demonstrate the robustness of the MSCNet model proposed in this application.

[0142] This invention also demonstrates the visualization results of experiments on the VisDrone2021 vehicle dataset, such as... Figure 5 As shown, each point represents an image randomly selected from the dataset. The visualization results show that the trends of the two curves are very similar, indicating that the error amplitude of the model is small and the minimum error is zero. The better the two curves fit, the better the performance of the model.

[0143] Ablation studies were conducted on the VisDrone2021 vehicle dataset, and the experimental results demonstrate the effectiveness of the proposed method.

[0144] The experiments considered using different backbone networks and MSCNet with and without CAM modules. VGG and ResNet were used as different backbone networks for comparison. Detailed experimental configuration information is as follows:

[0145] MSCNets with different backbone networks: represented as VGG-MSCNet and ResNet-MSCNet.

[0146] MSCNet with or without a Channel Attention Module (CAM) is indicated as MSCNet-no-CAM and ResNet-MSCNet-no-CAM.

[0147] As can be seen from Table 6, replacing VGG with ResNet does not result in better accuracy and leads to a decrease in MAE and RMSE:

[0148]

[0149] Table 6

[0150] The effectiveness of CAM was confirmed; it can extract useful features of vehicle counts hidden in different channels. The ResNet backbone network also performed poorly here, with a MAE of only 16.5 on the vehicle dataset, which is far lower than that of VGG-MSCNet.

[0151] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0152] It should be understood that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this application generally indicates an "or" relationship between the preceding and following related objects, but it may also indicate an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0153] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0154] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A dense vehicle counting method based on deep spatio-temporal network, characterized in that: The counting method comprises the following steps: S1: sending a counting image to the first ten layers of VGG16 through a multi-scale hollow convolution channel perception network MSCNet to extract initial image features; S2: using a direction-based perspective coding module to code the global features in four directions; S3: designing a multi-scale hollow residual module to extract different scale feature information, combining different layers in the module through a cascading method to form multiple hollow convolution rates, and solving the gradient disappearance problem caused by the deep network through a residual connection; S4: sampling and counting the range-scale vehicle targets through the multi-scale hollow residual module; S5: learning the weight of different channel features through a channel perception attention module, enhancing the vehicle features extracted in the mixed scene, combining the local features and the global features, and extracting the features hidden in different channels; In step S4, the sampling and counting of the range-scale vehicle targets comprise the following steps: S4.1: using a direction-based perspective coding module DPCM and a multi-scale hollow residual module MDRM to deeply mine the space-time information; S4.2: using the direction-based perspective coding module DPCM to code the perspective information in four directions; S4.3: using the multi-scale hollow residual module MDRM to densely extract the scale features of the range changes; S4.4: using the channel perception attention module CAM to mine the channel correlation and enhance the features of the vehicle counting in the human-vehicle mixed scene.

2. The method of claim 1, wherein the method is based on a deep spatio-temporal network. The MSCNet comprises a backbone network VGG16 and three modules: a direction-based perspective coding module DPCM, a multi-scale hollow residual module MDRM, and a channel perception attention module CAM.

3. The method of claim 1, wherein the method is based on a deep spatio-temporal network. In step S1, the first 10 layers of the VGG16 network and 3 pooling layers are reserved as the backbone network. The model first sends the input count image A into the pre-trained VGG16 backbone network to obtain a feature map f b As shown in formula (1): f b =F VGG (A)(1) The VGG16 is used to extract the initial features of the counting image A.

4. The method of claim 3, wherein the method is based on a deep spatio-temporal network. The DPCM processes long-distance continuous perspective changes, and the DPCM codes the perspective information in four directions; DPCM is applied to the feature maps f output from VGG16 b As input, it includes four convolutional layers: left, right, bottom, and top. Each convolution layer is composed of a convolution kernel with a size of C x ω and a ReLU activation function, and the output is superimposed with the next slice to form a new slice, and then the convolution layer operation is continued until all the slices are processed, and the feedforward calculation is defined as formula (2): （2） wherein L W (*) denotes the left layer operation Conv + ReLU, f W i is the i-th feature slice in the W direction; The perspective view information is encoded from four directions and the feature map f is output c .

5. The method of claim 4, wherein: The MDRM is used to extract the scale features of the continuous changes in the image, and the MDRM comprises three hollow convolution layers with hollow rates of 1, 2 and 3, and a 1x1 filter size convolution layer is added before each hollow convolution layer; A standard convolution kernel with a filter size of 3x3 is used to fuse all the connected features from the previous hollow convolution layers, reduce the number of channels of the end output, and apply a ReLU activation function after each hollow convolution layer; Each hollow convolution layer is closely connected with other hollow convolution layers, and each layer can access all subsequent layers and transmit the feature information to be extracted; After the dense and continuous layer connection, the module aggregates multiple receptive fields and increases the scale diversity.

6. The method of claim 5, wherein: In the operation of the multi-scale dilated convolution, the initial feature information is reserved, and the image feature fc and the multi-scale feature information are integrated. The MDRM module uses the most important method of multi-scale information to combine the multi-scale features with the original input features f c Connected, add the input characteristics to the output characteristics, that is, f d ; The function C(*) represents the connection of the four feature maps f c f1, f2, f3, and performs a 3x3 convolutional dilation rate = 1 operation to complete channel fusion, as shown in equation (3): （3） DPCM and MDRM extract continuous multi-scale spatial information in the spatial dimension and complete feature fusion.

7. The method of claim 6, wherein: After DPCM and MDRM, the output characteristic f d is sent to the CAM; CAM extracts features hidden in different channels of the feature map in the channel dimension, obtains the weight of each feature channel through learning, enhances the counting features through the weight, and suppresses the features that are not important to the counting task. The whole process is represented as formula (4): ​ （4） Feature map f d of size W x H x C, using global average pooling, f d is averaged into a 1 x 1 x C block.

Images