A video crowd counting method based on time-series interaction and global correlation network

Abstract

This invention discloses a video crowd counting method based on temporal interaction and a global association network, belonging to the field of crowd counting technology. It involves acquiring a continuous video sequence of a preset length containing pedestrians, forming a sample set with T consecutive video frames and the ground truth density map corresponding to each of the T consecutive video frames. Based on the sample set, a training model including an encoding module, a dual-branch feature fusion module, a channel-guided cross-branch feature fusion module, and a feature integration module is trained to obtain a temporal interaction and global association network used to generate the predicted density map of T consecutive frames. The T consecutive video frames to be estimated are input into the trained temporal interaction and global association network to obtain the predicted density map of T consecutive frames. The crowd estimation result is generated by summing the results frame by frame. This invention effectively improves the accuracy and robustness of crowd counting in video scenes through dual-branch parallel processing and cross-dimensional feature fusion.

Description

Technical Field

[0001] This invention relates to the field of crowd counting technology, and specifically to a video crowd counting method based on temporal interaction and global correlation networks. Background Technology

[0002] With the rapid development of intelligent surveillance systems and the increasing frequency of large-scale public events, video crowd counting has become a key technology in areas such as public safety, traffic management, and crowd control. The goal of video crowd counting is to accurately estimate the number of people in each video frame.

[0003] In existing research, video crowd counting methods can be divided into short-term temporal difference modeling methods and long-term spatiotemporal correlation modeling methods. Among them, short-term temporal difference modeling methods rely on auxiliary modal information such as optical flow or inter-frame difference, which are sensitive to occlusion and have difficulty in characterizing complex dynamic changes. Long-term spatiotemporal correlation modeling methods use 3D convolution or attention to model spatiotemporal relationships, but there is a high degree of similarity between adjacent frames, which easily leads to information redundancy. This not only increases the computational burden, but also weakens the model's ability to perceive key temporal information.

[0004] Therefore, there is an urgent need for a video crowd counting method that can simultaneously take into account the local temporal dependencies between adjacent frames and the global spatiotemporal correlation characteristics, and effectively regulate the contribution of temporal interaction information at different scales, so as to achieve the synergistic enhancement of temporal modeling and global context awareness in dense crowd scenes. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a video crowd counting method based on temporal interaction and a global correlation network.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A video crowd counting method based on temporal interaction and global association networks is proposed. Steps S1 to S2 construct a temporal interaction and global association network for estimating the number of people in a video frame. Step A generates estimation results for the number of people in T consecutive video frames to be estimated.

[0008] Step S1: Obtain a continuous video sequence of a preset length containing pedestrians, and form an extended video frame dataset by performing the same data augmentation operation on each T consecutive video frames. Each video frame in the extended video frame dataset corresponds to a ground truth density map, thereby forming a sample set with T consecutive video frames and the ground truth density maps corresponding to each video frame in the T consecutive video frames as samples.

[0009] Step S2: Construct a model to be trained, which includes an encoding module, a first upsampling module, a dual-branch feature fusion module, a channel-guided cross-branch feature fusion module, and a feature integration module. Based on the sample set, take T consecutive video frames in the sample as input and the ground truth density map corresponding to each of the T consecutive video frames in the sample as output, train the model to be trained, and obtain a temporal interaction and global association network for generating the predicted value density map of T consecutive frames.

[0010] Step A: Input the T consecutive video frames to be estimated into the trained temporal interaction and global association network to obtain the prediction density map of the T consecutive frames, and generate the number of people estimation results by summing the frames one by one.

[0011] Furthermore, the input end of the encoding module constitutes the input end of the temporal interaction and global association network, and the output end of the encoding module is sequentially connected in series with the first upsampling module, the dual-branch feature fusion module, the channel-guided cross-branch feature fusion module, and the feature integration module. The output end of the feature integration module constitutes the output end of the temporal interaction and global association network. Moreover, the dual-branch feature fusion module includes a parallel temporal interaction branch and a global association branch.

[0012] The encoding module receives T consecutive video frames, generates multi-scale spatial features corresponding to each video frame through layer-by-layer downsampling, and outputs the high-level strong semantic spatial features in each multi-scale spatial feature to the first upsampling module. The first upsampling module improves the spatial resolution of each high-level strong semantic feature through upsampling and outputs the upsampled high-level strong semantic features to the temporal interaction branch and the global association branch. The temporal interaction branch interacts and fuses the channel information between adjacent video frames to form hybrid channel features. The global association branch models the autocorrelation of the high-level strong semantic spatial features in terms of channel dimension, spatial dimension, and temporal dimension to form multi-dimensional global association aggregated features. The channel-guided cross-branch feature fusion module receives the hybrid channel features and multi-dimensional global association aggregated features, and forms cascaded spatial reconstruction features through cascaded spatial reconstruction and adaptive channel enhancement. The feature integration module receives the cascaded spatial reconstruction features, performs lightweight channel compression and feature integration through a multi-layer convolutional network, and outputs a density map of predicted values ​​for T consecutive video frames.

[0013] Furthermore, the timing interaction branch includes a first gated timing channel interaction module, a second gated timing channel interaction module, and a second upsampling module. The first gated timing channel interaction module and the second gated timing channel interaction module have the same structure.

[0014] The input terminal of the first gated timing channel interaction module constitutes the input terminal of the timing interaction branch, and the output terminal of the first gated timing channel interaction module is connected in series with the second gated timing channel interaction module and the second upsampling module. The output terminal of the second upsampling module constitutes the output terminal of the timing interaction branch.

[0015] The first gated temporal channel interaction module applies a bidirectional temporal cyclic shift operation based on gate weights to the high-level strong semantic features through a cascaded forward temporal channel interaction module and a reverse temporal channel interaction module, generating a first temporal interaction feature; the second gated temporal channel interaction module applies another bidirectional temporal cyclic shift operation based on gate weights to the first temporal interaction feature, generating a second temporal interaction feature; the second upsampling module performs an upsampling operation on the second temporal interaction feature, generating a hybrid channel feature.

[0016] Furthermore, the forward temporal channel interaction module includes a deep convolution module, a grouping gating module, a layer normalization module, a weighted fusion module, and a channel attention module;

[0017] The input of the deep convolution module forms the input of the forward temporal channel interaction module, and the output of the deep convolution module is connected to the input of the layer normalization module and the input of the group gating module; the input of the weighted fusion module is connected to the output of the group gating module and the output of the layer normalization module; the input of the channel attention module is connected to the output of the weighted fusion module; the output of the channel attention module forms the output of the forward temporal channel interaction module.

[0018] The depthwise convolution module generates local features through a single depthwise separable convolution; the grouping gating module divides the local features along the channel dimension to form several local feature groups, and adaptively adjusts the spatial weights of each channel in each local feature group to generate gated temporal channel interaction features; the layer normalization module normalizes the local features to generate normalized local features; the weighted fusion module fuses the gated temporal channel interaction features and the normalized local features to generate a first fused feature; and the channel attention module enhances the first fused feature to generate a positive temporal interaction feature.

[0019] Furthermore, the group gating module includes a partitioning module, a gating output module, a residual module, a bidirectional timing channel shifting module, and a splicing module;

[0020] The input terminal of the partitioning module constitutes the input terminal of the group gating module, and the output terminal of the partitioning module is connected to the input terminal of the gating output module, the input terminal of the residual module, and the input terminal of the splicing module; the output terminal of the gating output module is connected to the input terminal of the residual module and the input terminal of the bidirectional timing channel shift module; the input terminal of the splicing module is also connected to the output terminal of the bidirectional timing channel shift module and the output terminal of the residual module; the output terminal of the splicing module constitutes the output terminal of the group gating module.

[0021] The segmentation module divides local features into retained features and interactive features along the channel dimension according to a preset ratio, and further divides the interactive features into several interactive feature groups evenly.

[0022] For each group of features to be interacted with, the gated output module sequentially uses a loop filling operation and a 3D convolution operation to generate a single-channel feature map corresponding to the group of features to be interacted with, and then generates a gated weight map corresponding to the single-channel feature map through an activation function. After obtaining the gated weight map corresponding to each group of features to be interacted with, the gated weight maps of each group of features to be interacted with are copied and stitched along the channel dimension to generate the channel-level gated weights corresponding to the features to be interacted with. Then, the modulated interactive features are generated by multiplying the features to be interacted with the channel-level gated weights element by element.

[0023] The residual module is used to remove modulation interaction features from the feature group to be interacted with, generating residual features; the bidirectional timing channel shift module is used to divide the modulation interaction features into an even number of gated output feature groups along the channel dimension, and perform forward timing cyclic shift operation or reverse timing cyclic shift operation on each gated output feature group using a preset shift step size to generate shifted gated output features; the splicing module adds the shifted gated output features and residual features element by element, and splices them with the retained features along the channel dimension to obtain the gated timing channel interaction features.

[0024] Furthermore, the global correlation branch includes a multi-dimensional global correlation aggregation module and a third upsampling module, and the multi-dimensional global correlation aggregation module includes a channel module, a spatial module, a splicing module, a temporal module, and a fusion module;

[0025] The input terminals of the channel module and the spatial module together constitute the input terminal of the multidimensional global correlation module; the input terminal of the splicing module is connected to the output terminals of the channel module and the spatial module, and the input terminal of the timing module is connected to the output terminal of the splicing module; the input terminal of the third upsampling module is connected to the output terminal of the timing module; the output terminal of the third upsampling module constitutes the output terminal of the multidimensional global correlation module.

[0026] The channel module is used to generate a channel-level global representation using global weighted average pooling, and multiply the channel-level global representation by its transpose along the channel dimension. After normalization by the activation function, a channel autocorrelation matrix is ​​generated. This channel autocorrelation matrix is ​​then multiplied by high-level strong semantic spatial features to generate channel-enhanced features. The spatial module is used to generate a spatial-level global representation using channel-level weighted average pooling, and multiply the spatial-level global representation by its transpose along the channel dimension. After normalization by the activation function, a spatial autocorrelation matrix is ​​generated. This spatial autocorrelation matrix is ​​then multiplied by high-level strong semantic spatial features to generate spatial-enhanced features. The concatenation module is used to... The channel enhancement feature and the spatial enhancement feature are added together to obtain the channel-space fusion feature. The channel-space fusion feature is then residually connected with the high-level strong semantic spatial feature to generate a spliced ​​fusion feature. The temporal module generates a global temporal feature based on the high-level strong semantic feature and the spliced ​​fusion feature using a global temporal attention mechanism, and generates a temporal reconstruction feature through convolutional mapping. The fusion module is used to generate a first spatiotemporal related global feature by fusing the temporal reconstruction feature and the channel-space fusion feature element-wise. The third upsampling module is used to upsample the first spatiotemporal related global feature to generate a multidimensional global related aggregate feature.

[0027] Furthermore, the channel-guided cross-branch feature fusion module includes a cascaded third gated timing channel interaction module and a fourth gated timing channel interaction module, as well as a cascaded first channel-guided local space reconstruction module and a second channel-guided local space reconstruction module.

[0028] Among them, the structures of the third gated timing channel interaction module and the fourth gated timing channel interaction module are the same as those of the first gated timing channel interaction module; the structures of the first channel guided local space reconstruction module and the second channel guided local space reconstruction module are the same; and the first channel guided local space reconstruction module includes an aggregation module, a channel weight generation module, and a weighting module.

[0029] The third gated temporal channel interaction module is used to generate third temporal interaction features; the aggregation module is used to apply multiple deep convolutional operations with different receptive fields to the multidimensional global correlation aggregation features, and perform weighted fusion and normalization processing through learnable weights to generate salient spatial features; the channel weight generation module is used to perform spatial convolution operations and global average pooling on the third temporal interaction features to obtain global channel representations, and extract channel weights through a bottleneck structure composed of two fully connected layers, and further utilize residual connections and The activation function yields the channel weights; the weighting module is used to weight the channel weights and significant spatial features along the channel dimension, and generate the first spatial reconstruction features through a single convolution and residual connection;

[0030] The fourth gated timing channel interaction module is used to generate a fourth timing interaction feature based on the third timing interaction feature; the second channel guided local space reconstruction module is used to generate cascaded space reconstruction features based on the first space reconstruction feature and the fourth timing interaction feature.

[0031] Furthermore, the feature fusion module is used to stitch together cascaded spatial reconstruction features and fourth temporal interaction features along the channel dimension, and to generate a continuous T-frame prediction value density map using multi-layer convolution operations.

[0032] The beneficial effects of adopting the above technical solution are as follows:

[0033] (1) This invention fully explores the complementary information between temporal interaction features and multidimensional global association aggregation features through temporal interaction and global association network. It not only effectively realizes the fusion of temporal interaction features and multidimensional global association aggregation features, but also significantly improves the model prediction accuracy.

[0034] (2) This invention effectively improves the efficiency of information utilization between adjacent frames through adaptive channel filtering and temporal information interaction mechanism, and realizes more complete temporal feature fusion. Compared with the prior art, this invention can more comprehensively model the temporal dependency relationship between consecutive frames, thereby significantly enhancing the model's temporal modeling ability and overall robustness in complex scenarios.

[0035] (3) This invention introduces the global correlation of three dimensions—channel, space, and time—into the feature representation to achieve adaptive modeling of multi-dimensional global dependencies. Compared with existing modeling methods that only focus on a single dimension or local area, this invention can capture the overall video context information in different dimensions, highlight key features while suppressing redundant responses, and effectively improve the temporal consistency and density prediction accuracy of the model while maintaining linear computational complexity.

[0036] (4) By constructing a unidirectional temporal modulation mechanism between the temporal interaction branch and the global association branch, the present invention realizes the dynamic fusion of cross-branch information, which not only enhances the local inductive ability of the global branch, but also improves the model's ability to perceive the temporal dimension context, thereby significantly improving the robustness and discriminability of feature expression. Attached Figure Description

[0037] Figure 1 This is a flowchart of the invention;

[0038] Figure 2 This is a framework diagram of the temporal interaction and global association network of this invention;

[0039] Figure 3 This is a framework diagram of the first gated timing channel interaction module of the present invention;

[0040] Figure 4 This is a framework diagram of the multidimensional global association aggregation module of the present invention;

[0041] Figure 5 This is a framework diagram of the channel-guided local space reconstruction module of the present invention;

[0042] Figure 6 These are comparison images of the invention's effects in outdoor scenarios;

[0043] Figure 7 These are comparison images showing the effects of this invention in indoor settings. Detailed Implementation

[0044] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] refer to Figure 1 A video crowd counting method based on temporal interaction and global association networks is proposed. Steps S1 to S2 construct a temporal interaction and global association network for estimating the number of people in a video frame. Step A generates estimation results for the number of people in T consecutive video frames to be estimated.

[0046] Step S1: Obtain a continuous video sequence of a preset length that includes pedestrians. Perform random cropping of the same region on each of the T consecutive video frames and perform a uniform horizontal flip on the cropped region with a probability of 0.5 to achieve data augmentation and form an extended video frame dataset. Each video frame in the extended video frame dataset corresponds to a ground truth density map, thereby forming a sample set with the T consecutive video frames and the ground truth density maps corresponding to each video frame in the T consecutive video frames as samples.

[0047] Furthermore, during the data augmentation process, missing frames at sequence boundaries can be filled in by copying the first or last frame to ensure the integrity of the input sequence.

[0048] Step S2: Construct a model to be trained, which includes an encoding module, a first upsampling module, a dual-branch feature fusion module, a channel-guided cross-branch feature fusion module, and a feature integration module. Based on the sample set, take T consecutive video frames in the sample as input and the ground truth density map corresponding to each of the T consecutive video frames in the sample as output, train the model to be trained, and obtain a temporal interaction and global association network for generating the predicted value density map of T consecutive frames.

[0049] Step A: Input the T consecutive video frames to be estimated into the trained temporal interaction and global association network to obtain the prediction density map of the T consecutive frames, and generate the number of people estimation results by summing the frames one by one.

[0050] Further, refer to Figure 2The input end of the encoding module constitutes the input end of the temporal interaction and global association network. The output end of the encoding module is sequentially connected in series with the first upsampling module, the dual-branch feature fusion module, the channel-guided cross-branch feature fusion module, and the feature integration module. The output end of the feature integration module constitutes the output end of the temporal interaction and global association network. The dual-branch feature fusion module includes a parallel temporal interaction branch and a global association branch.

[0051] The encoding module receives T consecutive video frames, generates multi-scale spatial features corresponding to each video frame through layer-by-layer downsampling, and outputs the high-level strong semantic spatial features in each multi-scale spatial feature to the first upsampling module. The first upsampling module improves the spatial resolution of each high-level strong semantic feature through upsampling and outputs the upsampled high-level strong semantic features to the temporal interaction branch and the global association branch. The temporal interaction branch interacts and fuses the channel information between adjacent video frames to form hybrid channel features. The global association branch models the autocorrelation of the high-level strong semantic spatial features in terms of channel dimension, spatial dimension, and temporal dimension to form multi-dimensional global association aggregated features. The channel-guided cross-branch feature fusion module receives the hybrid channel features and multi-dimensional global association aggregated features, and forms cascaded spatial reconstruction features through cascaded spatial reconstruction and adaptive channel enhancement. The feature integration module receives the cascaded spatial reconstruction features, performs lightweight channel compression and feature integration through a multi-layer convolutional network, and outputs a density map of predicted values ​​for T consecutive video frames.

[0052] Specifically, firstly, take T consecutive RGB video frames... Simultaneously, the input is fed into the ConvNeXt-T encoder for feature extraction, generating multi-scale spatial features corresponding to each video frame, among which... These represent the number of frames, number of channels, height, and width of the video frame, respectively. Then, an upsampling operation is used to increase the spatial resolution of each multi-scale spatial feature from 1 / 32 to 1 / 16, obtaining a higher-resolution multi-scale spatial feature representation. Next, the high-level strong semantic spatial features from each upsampled multi-scale spatial feature are output to the temporal interaction branch and the global association branch for processing. The temporal interaction branch and the global association branch share the high-level strong semantic features, denoted as […]. and .

[0053] In this embodiment, each video frame is an input image. Therefore, when the ConvNeXt-T model processes the input image, it will go through four computational stages in sequence. The size of the feature map output by each stage decreases step by step, while the number of channels increases step by step, thereby capturing information at different scales from details to semantics.

[0054] Further, refer to Figure 3 The timing interaction branch includes a first gated timing channel interaction module, a second gated timing channel interaction module, and a second upsampling module. The first gated timing channel interaction module and the second gated timing channel interaction module have the same structure.

[0055] The input terminal of the first gated timing channel interaction module constitutes the input terminal of the timing interaction branch, and the output terminal of the first gated timing channel interaction module is connected in series with the second gated timing channel interaction module and the second upsampling module. The output terminal of the second upsampling module constitutes the output terminal of the timing interaction branch.

[0056] The first gated temporal channel interaction module applies a bidirectional temporal cyclic shift operation based on gate weights to the high-level strong semantic features through cascaded forward and reverse temporal channel interaction modules, generating a first temporal interaction feature. The second gated temporal channel interaction module applies another bidirectional temporal cyclic shift operation based on gate weights to the first temporal interaction feature, generating a second temporal interaction feature. The second upsampling module performs an upsampling operation on the second temporal interaction feature, generating a hybrid channel feature. .

[0057] Further, refer to Figure 3 The forward temporal channel interaction module includes a deep convolution module, a grouping gating module, a layer normalization module, a weighted fusion module, and a channel attention module;

[0058] The input of the deep convolution module forms the input of the forward temporal channel interaction module, and the output of the deep convolution module is connected to the input of the layer normalization module and the input of the group gating module; the input of the weighted fusion module is connected to the output of the group gating module and the output of the layer normalization module; the input of the channel attention module is connected to the output of the weighted fusion module; the output of the channel attention module forms the output of the forward temporal channel interaction module.

[0059] The deep convolutional module is used to receive high-level strong semantic features and, through... Depth convolution generates local features The grouping gating module is used to divide local features along the channel dimension, forming several local feature groups, and adaptively adjusts the spatial weights of each channel in each local feature group to generate gated temporal channel interaction features. The layer normalization module is used to normalize local features to generate normalized local features; the weighted fusion module is used to fuse gated temporal channel interaction features and normalized local features to generate a first fused feature; the channel attention module is used to enhance the first fused feature to generate a positive temporal interaction feature. .

[0060] In this embodiment, the only difference between the reverse timing channel interaction module and the forward timing channel interaction module is that the group gating module is replaced with the anti-group gating module.

[0061] Further, refer to Figure 3 The group gating module includes a partitioning module, a gating output module, a residual module, a bidirectional timing channel shifting module, and a splicing module;

[0062] The input terminal of the partitioning module constitutes the input terminal of the group gating module, and the output terminal of the partitioning module is connected to the input terminal of the gating output module, the input terminal of the residual module, and the input terminal of the splicing module; the output terminal of the gating output module is connected to the input terminal of the residual module and the input terminal of the bidirectional timing channel shift module; the input terminal of the splicing module is also connected to the output terminal of the bidirectional timing channel shift module and the output terminal of the residual module; the output terminal of the splicing module constitutes the output terminal of the group gating module.

[0063] The division module is used to divide according to a preset ratio. The local features are divided into two parts along the channel dimension, where the first part is divided into two parts. The local features of each channel constitute the features to be interacted with. The remaining Local features of each channel constitute preserved features The interactive features are further divided into several interactive feature groups evenly according to the following formula:

[0064] ;

[0065] in, This refers to the number of video frames. In this embodiment... According to the formula above, the features to be interacted with are... They are evenly divided into 4 groups of features to be interacted with, denoted as follows: , , and .

[0066] For each group of features to be interacted with, the gated output module sequentially uses a cyclic padding operation and a 3D convolution operation to generate a single-channel feature map corresponding to the group of features to be interacted with, and then generates a gated weight map corresponding to the single-channel feature map through an activation function. The cyclic padding operation refers to padding the end of the sequence with... Frame features are concatenated to the beginning of the sequence, and the beginning of the sequence is also appended. Frame features are concatenated to the end of the sequence, thus making full use of the adjacent information of the boundary frames.

[0067] In this embodiment, for and A one-frame cyclic padding method is used, which involves copying the feature of the last frame of the sequence and concatenating it to the beginning of the sequence, while simultaneously copying the feature of the first frame of the sequence and concatenating it to the end of the sequence; for and A two-frame cyclic padding method is adopted, which copies the features of the last two frames of the sequence to the beginning of the sequence, and at the same time copies the features of the first two frames of the sequence to the end of the sequence, in order to enhance the utilization of the temporal correlation information of the boundary frames.

[0068] Furthermore, using a temporal dilation rate of 1 and a convolution kernel of... The 3D convolution pairs the first two feature groups to be interacted with. and Perform a 3D convolution operation to obtain and The corresponding single-channel feature maps; using a temporal dilation rate of 2 and a convolution kernel of... The 3D convolution pairs the last two feature groups to be interacted with and Perform a 3D convolution operation to obtain and The corresponding single-channel feature maps; and then utilize Activation function processing generates a gated weight map corresponding to each single-channel feature map. :

[0069]

[0070] in, Represents the hyperbolic tangent activation function; Indicates the time-series inflation rate of convolution; Indicates a cyclic fill operation; Representation of features The Grouped.

[0071] Then, for each gate weight graph conduct The number of channels obtained by copying and concatenating is: Channel-level gating weights and the channel-level gating weights are combined with The modulation interaction features are obtained by multiplying each channel element-wise. :

[0072]

[0073] in, Indicates a splicing operation; Indicates a repeated copy operation; , , , Four feature groups to be interacted with , , , The corresponding gating weight diagrams.

[0074] Simultaneously, residual features are obtained by removing modulated interaction features from the features to be interacted with. ,Right now .

[0075] Subsequently, the modulation interaction features are input to the bidirectional timing channel shift module. The module divides the modulation interaction features into an even number of gated output feature groups along the channel dimension, and performs a forward timing cyclic shift operation or a reverse timing cyclic shift operation on each gated output feature group using a preset shift step size to generate shift-gated output features. Specifically, forward timing cyclic shift refers to passing the channel features of the current frame backward along the time dimension to the position of its subsequent frames, i.e., moving part of the channel features of the current frame to the next frame or a subsequent frame; correspondingly, reverse timing cyclic shift refers to passing the channel features of the current frame forward along the time dimension to the position of its preceding frame.

[0076] Taking forward timing cyclic shift as an example, according to the preset ratio Modulation interaction features The sequence is divided into four gated output feature groups. For the first two gated output feature groups, they are cyclically shifted forward and backward along the time dimension with a step size of 1. This means shifting the index of the current frame in the time series in the positive direction, moving the first group of channel features to the position of the next frame, and the second group of channel features to the position of the previous frame. For the last two gated output feature groups, they are cyclically shifted forward and backward along the time dimension with a step size of 2. The cyclic shift means that channel features shifted out from one side of the sequence are added to the other side to fill the gaps created by the shift.

[0077] After completing the bidirectional temporal channel shift, the stitching module adds the shift-gated output features and residual features element-wise, and then stitches them together with the retained features along the channel dimension to obtain the gated temporal channel interaction features. .

[0078] In this embodiment, the difference between the reverse timing channel interaction module and the forward timing channel interaction module is that the cyclic shift performs the shift in the opposite direction. For example, the four gated output feature groups in the forward timing channel interaction module are shifted by forward 1, reverse 1, forward 2, and reverse 2 respectively; while the four gated output feature groups in the reverse timing channel interaction module are shifted by reverse 1, forward 1, reverse 2, and forward 2 respectively.

[0079] Further, refer to Figure 4 The global correlation branch includes a multi-dimensional global correlation aggregation module and a third upsampling module, and the multi-dimensional global correlation aggregation module includes a channel module, a spatial module, a splicing module, a temporal module, and a fusion module;

[0080] The input terminals of the channel module and the spatial module together constitute the input terminal of the multidimensional global correlation module; the input terminal of the splicing module is connected to the output terminals of the channel module and the spatial module, and the input terminal of the timing module is connected to the output terminal of the splicing module; the input terminal of the third upsampling module is connected to the output terminal of the timing module; the output terminal of the third upsampling module constitutes the output terminal of the multidimensional global correlation module.

[0081] First, the channel module generates a channel-level global representation using global weighted average pooling. The channel-level global representation is multiplied by its transpose along the channel dimension, and then normalized by the activation function to generate the channel autocorrelation matrix. Then, the channel autocorrelation matrix is ​​multiplied with the high-level strong semantic space features to generate channel-enhanced features. :

[0082] ;

[0083] ;

[0084] ;

[0085] in, This represents globally weighted average pooling; weights High-level strong semantic features Normalized in the spatial dimension, representing spatial location. Weight at each location; This indicates the current channel c. Feature map In spatial location The response value at that location, Indicates the spatial position in each channel The weight value calculated at the location; This represents matrix multiplication.

[0086] Simultaneously, the spatial module generates a spatial-level global representation using channel-level weighted average pooling. The spatial-level global representation is multiplied by its transpose along the channel dimension, and then normalized by an activation function to generate the spatial autocorrelation matrix. Then, the spatial autocorrelation matrix is ​​multiplied with the high-level strong semantic spatial features to generate spatially enhanced features. :

[0087] ;

[0088] ;

[0089] ;

[0090] in, This indicates channel-level weighted average pooling; weights High-level strong semantic features Normalized along the channel dimension, representing the channel position. Weight at each location; Indicates the current spatial location Feature map In the passageway The response value at that location, Indicates the current spatial location In the passage The weight value calculated at that point.

[0091] Subsequently, the stitching module adds the channel enhancement features and spatial enhancement features to obtain the channel-spatial fusion features. Furthermore, the channel-space fusion features are residually connected with high-level strong semantic space features to generate spliced ​​and fused features. ;

[0092] ;

[0093] ;

[0094] Furthermore, the temporal module generates global temporal features based on high-level strong semantic features and splicing fusion features, utilizing a global temporal attention mechanism. It then employs a multi-head attention mechanism to calculate the global temporal correlation between frames, and subsequently... Convolution generates temporal reconstruction features This enhances information exchange between frames.

[0095] Specifically, a global temporal attention mechanism is used to stitch and fuse features. Perform global weighted average pooling and obtain the query result through linear transformation. ,key At the same time, high-level strong semantic features are used as values. The calculation expression is as follows:

[0096]

[0097]

[0098]

[0099] in, It is a learnable parameter matrix used to enhance the model's fitting ability; express operate.

[0100] Furthermore, in global temporal attention, by querying consecutive frames... AND key Multiplying them yields the attention matrix, which is then multiplied by the values. Multiplication yields the global temporal features, calculated as follows:

[0101]

[0102] in, This represents the activation function.

[0103] Finally, the fusion module generates the first spatiotemporal correlation global feature by element-wise addition and fusion of the temporal reconstruction features and the channel-spatial fusion features. :

[0104] ;

[0105] The third upsampling module is used to perform upsampling operations on the first spatiotemporal correlated global features to generate multidimensional global correlated aggregated features. .

[0106] In this embodiment, through a multi-dimensional global correlation modeling mechanism, global dependencies can be adaptively captured in three dimensions: channel, space, and time. This highlights information-rich feature regions, enabling the features of the current frame to integrate semantic relationships between channels, contextual information of spatial location, and temporal dynamics between frames, thereby improving the comprehensiveness and robustness of feature representation.

[0107] Further, refer to Figure 5The channel-guided cross-branch feature fusion module includes a cascaded third gated timing channel interaction module and a fourth gated timing channel interaction module, as well as a cascaded first channel-guided local space reconstruction module and a second channel-guided local space reconstruction module.

[0108] Among them, the structures of the third gated timing channel interaction module and the fourth gated timing channel interaction module are the same as those of the first gated timing channel interaction module; the structures of the first channel guided local space reconstruction module and the second channel guided local space reconstruction module are the same; and the first channel guided local space reconstruction module includes an aggregation module, a channel weight generation module, and a weighting module.

[0109] The third gated timing channel interaction module is used to generate third timing interaction features. The aggregation module is used to aggregate features based on multidimensional global association. Multiple deep convolutional operations with different receptive fields are applied, and weighted fusion and normalization are performed using learnable weights to generate salient spatial features. :

[0110] ;

[0111] ;

[0112] in, express Activation function; Representation layer normalization; These are learnable parameters; This represents a depthwise convolution with a kernel size of 3 and a dilation rate of 1.

[0113] Simultaneously, the channel weight generation module performs the third temporal interaction feature analysis. Spatial convolution and global average pooling yield the global channel representation. Furthermore, channel weights are extracted through a bottleneck structure consisting of two fully connected layers, and residual connections are further utilized. Activation function obtains channel weights :

[0114] ;

[0115] ;

[0116] in, Indicates global average pooling; This represents a bottleneck structure, which includes a Convolution compresses the number of channels, followed by ReLU activation, and then... Convolution recovers the number of channels.

[0117] Furthermore, the weighting module weights the channel weights and salient spatial features along the channel dimension to enhance the response of salient channels and suppress interference from irrelevant channels; and through Convolution and residual connections generate first spatial reconstruction features guided by temporal interaction features. :

[0118] ;

[0119] The fourth gated timing channel interaction module is used to generate a fourth timing interaction feature based on the third timing interaction feature. The second channel guides the local spatial reconstruction module to generate cascaded spatial reconstruction features based on the first spatial reconstruction features and the fourth temporal interaction features. This enables cascaded spatial enhancement and channel recalibration.

[0120] Furthermore, the feature fusion module is used to concatenate spatial reconstruction features and fourth temporal interaction features along the channel dimension, and uses four-layer convolutional operations to generate a continuous T-frame prediction density map. ;

[0121] The specific structure of the feature fusion module is as follows:

[0122] ;

[0123] in, Indicates having Each convolutional kernel has a kernel size of [number]. Expansion rate convolutional layers, for Activation function.

[0124] Preferably, in step S2, the ConvNeXt-T model is initialized using weights pre-trained on ImageNet-22K, and the model to be trained is trained using the mean squared error loss function. The optimizer for training the temporal interaction and global association networks is AdamW, with an initial learning rate of 1e-5, a weight decay of 1e-4, and a maximum number of iterations of 300.

[0125] The technical effects of this embodiment will be further explained in detail below with reference to comparative experiments.

[0126] from Figure 6 and Figure 7It can be observed that the video crowd counting method provided in this embodiment can more accurately reflect the crowd distribution in different scenarios. It can effectively suppress overestimation in dense areas and reduce missed detections in sparse areas. The generated density map is more continuous in spatial distribution and more consistent with the real crowd distribution. This shows that the proposed temporal interaction and global association mechanism can make full use of cross-frame information and global context information, thereby improving the counting accuracy and robustness of the model in complex scenarios.

[0127] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A video crowd counting method based on temporal interaction and global correlation networks, characterized in that, Construct a temporal interaction and global association network for estimating the number of people in a video frame according to steps S1 to S2, and generate the estimation results of the number of people in the T consecutive video frames to be estimated according to step A: Step S1: Obtain a continuous video sequence of a preset length containing pedestrians, and form an extended video frame dataset by performing the same data augmentation operation on each T consecutive video frames. Each video frame in the extended video frame dataset corresponds to a ground truth density map, thereby forming a sample set with T consecutive video frames and the ground truth density maps corresponding to each video frame in the T consecutive video frames as samples. Step S2: Construct a model to be trained, which includes an encoding module, a first upsampling module, a dual-branch feature fusion module, a channel-guided cross-branch feature fusion module, and a feature integration module. Based on the sample set, take T consecutive video frames in the sample as input and the ground truth density map corresponding to each of the T consecutive video frames in the sample as output, train the model to be trained, and obtain a temporal interaction and global association network for generating the predicted value density map of T consecutive frames. Step A: Input the T consecutive video frames to be estimated into the trained temporal interaction and global association network to obtain the prediction density map of the T consecutive frames, and generate the number of people estimation results by summing the frames one by one.

2. The video crowd counting method based on temporal interaction and global correlation network according to claim 1, characterized in that, The input end of the encoding module constitutes the input end of the temporal interaction and global association network. The output end of the encoding module is sequentially connected in series with the first upsampling module, the dual-branch feature fusion module, the channel-guided cross-branch feature fusion module, and the feature integration module. The output end of the feature integration module constitutes the output end of the temporal interaction and global association network. The dual-branch feature fusion module includes a parallel temporal interaction branch and a global association branch. The encoding module is used to receive T consecutive video frames, generate multi-scale spatial features corresponding to each video frame through layer-by-layer downsampling operation, and output the high-level strong semantic spatial features in each multi-scale spatial feature to the first upsampling module; the first upsampling module is used to improve the spatial resolution of each high-level strong semantic feature through upsampling operation, and output each upsampled high-level strong semantic feature to the temporal interaction branch and the global association branch. The temporal interaction branch is used to interact and fuse the channel information between adjacent video frames to form a hybrid channel feature; The global association branch is used to model the autocorrelation of high-level strong semantic space features in terms of channel dimension, spatial dimension and temporal dimension to form multi-dimensional global association aggregated features; the channel-guided cross-branch feature fusion module is used to receive mixed channel features and multi-dimensional global association aggregated features, and form cascaded spatial reconstruction features through cascaded spatial reconstruction and adaptive channel enhancement. The feature integration module is used to receive cascaded spatial reconstruction features, perform lightweight channel compression and feature integration through a multi-layer convolutional network, and output a density map of predicted values ​​for T consecutive video frames.

3. The video crowd counting method based on temporal interaction and global correlation network according to claim 2, characterized in that, The timing interaction branch includes a first gated timing channel interaction module, a second gated timing channel interaction module, and a second upsampling module. The first gated timing channel interaction module and the second gated timing channel interaction module have the same structure. The input terminal of the first gated timing channel interaction module constitutes the input terminal of the timing interaction branch, and the output terminal of the first gated timing channel interaction module is connected in series with the second gated timing channel interaction module and the second upsampling module. The output terminal of the second upsampling module constitutes the output terminal of the timing interaction branch. The first gated temporal channel interaction module applies a bidirectional temporal cyclic shift operation based on gate weights to the high-level strong semantic features through a cascaded forward temporal channel interaction module and a reverse temporal channel interaction module, generating a first temporal interaction feature; the second gated temporal channel interaction module applies another bidirectional temporal cyclic shift operation based on gate weights to the first temporal interaction feature, generating a second temporal interaction feature; the second upsampling module performs an upsampling operation on the second temporal interaction feature, generating a hybrid channel feature.

4. The video crowd counting method based on temporal interaction and global correlation network according to claim 3, characterized in that, The forward temporal channel interaction module includes a deep convolution module, a grouping gating module, a layer normalization module, a weighted fusion module, and a channel attention module; The input of the deep convolution module forms the input of the forward temporal channel interaction module, and the output of the deep convolution module is connected to the input of the layer normalization module and the input of the group gating module; the input of the weighted fusion module is connected to the output of the group gating module and the output of the layer normalization module; the input of the channel attention module is connected to the output of the weighted fusion module; the output of the channel attention module forms the output of the forward temporal channel interaction module. The depthwise convolution module generates local features through a single depthwise separable convolution; the grouping gating module divides the local features along the channel dimension to form several local feature groups, and adaptively adjusts the spatial weights of each channel in each local feature group to generate gated temporal channel interaction features; the layer normalization module normalizes the local features to generate normalized local features; the weighted fusion module fuses the gated temporal channel interaction features and the normalized local features to generate a first fused feature; and the channel attention module enhances the first fused feature to generate a positive temporal interaction feature.

5. The video crowd counting method based on temporal interaction and global correlation network according to claim 4, characterized in that, The group gating module includes a partitioning module, a gating output module, a residual module, a bidirectional time-series channel shifting module, and a splicing module; The input terminal of the partitioning module constitutes the input terminal of the group gating module, and the output terminal of the partitioning module is connected to the input terminal of the gating output module, the input terminal of the residual module, and the input terminal of the splicing module; the output terminal of the gating output module is connected to the input terminal of the residual module and the input terminal of the bidirectional timing channel shift module; the input terminal of the splicing module is also connected to the output terminal of the bidirectional timing channel shift module and the output terminal of the residual module; the output terminal of the splicing module constitutes the output terminal of the group gating module. The segmentation module divides local features into retained features and interactive features along the channel dimension according to a preset ratio, and further divides the interactive features into several interactive feature groups evenly. For each group of features to be interacted with, the gated output module sequentially uses a loop filling operation and a 3D convolution operation to generate a single-channel feature map corresponding to the group of features to be interacted with, and then generates a gated weight map corresponding to the single-channel feature map through an activation function. After obtaining the gated weight map corresponding to each group of features to be interacted with, the gated weight maps of each group of features to be interacted with are copied and stitched along the channel dimension to generate the channel-level gated weights corresponding to the features to be interacted with. Then, the modulated interactive features are generated by multiplying the features to be interacted with the channel-level gated weights element by element. The residual module is used to remove modulation interaction features from the feature group to be interacted with, and generate residual features; the bidirectional timing channel shift module is used to divide the modulation interaction features into an even number of gated output feature groups along the channel dimension, and perform forward timing cyclic shift operation or reverse timing cyclic shift operation on each gated output feature group using a preset shift step size, to generate shifted gated output features. The splicing module adds the shift-gated output features and residual features element by element, and splices them with the retained features along the channel dimension to obtain the gated temporal channel interaction features.

6. The video crowd counting method based on temporal interaction and global correlation network according to claim 2, characterized in that, The global correlation branch includes a multi-dimensional global correlation aggregation module and a third upsampling module, and the multi-dimensional global correlation aggregation module includes a channel module, a spatial module, a splicing module, a temporal module, and a fusion module; The input terminals of the channel module and the spatial module together constitute the input terminal of the multidimensional global correlation module; the input terminal of the splicing module is connected to the output terminals of the channel module and the spatial module, and the input terminal of the timing module is connected to the output terminal of the splicing module; the input terminal of the third upsampling module is connected to the output terminal of the timing module; the output terminal of the third upsampling module constitutes the output terminal of the multidimensional global correlation module. The channel module is used to generate a channel-level global representation using global weighted average pooling, and multiplies the channel-level global representation with its transpose along the channel dimension. After normalization by an activation function, a channel autocorrelation matrix is ​​generated. The channel autocorrelation matrix is ​​then multiplied with high-level strong semantic spatial features to generate channel-enhanced features. The spatial module is used to generate a spatial-level global representation using channel-level weighted average pooling, and multiplies the spatial-level global representation with its transpose along the channel dimension. After normalization by an activation function, a spatial autocorrelation matrix is ​​generated. The spatial autocorrelation matrix is ​​then multiplied with high-level strong semantic spatial features to generate spatial-enhanced features. The splicing module is used to add the channel enhancement features and the spatial enhancement features to obtain the channel-space fusion features, and to perform residual connection between the channel-space fusion features and the high-level strong semantic spatial features to generate spliced ​​fusion features; The temporal module generates global temporal features based on high-level strong semantic features and splicing fusion features, and generates temporal reconstruction features through convolutional mapping; the fusion module is used to generate the first spatiotemporal related global features by adding the temporal reconstruction features and the channel-space fusion features element by element; the third upsampling module is used to perform upsampling operation on the first spatiotemporal related global features to generate multidimensional global related aggregate features.

7. The video crowd counting method based on temporal interaction and global correlation network according to claim 5, characterized in that, The channel-guided cross-branch feature fusion module includes a cascaded third gated timing channel interaction module and a fourth gated timing channel interaction module, as well as a cascaded first channel-guided local space reconstruction module and a second channel-guided local space reconstruction module. Among them, the structures of the third gated timing channel interaction module and the fourth gated timing channel interaction module are the same as those of the first gated timing channel interaction module; the structures of the first channel guided local space reconstruction module and the second channel guided local space reconstruction module are the same; and the first channel guided local space reconstruction module includes an aggregation module, a channel weight generation module, and a weighting module. The third gated temporal channel interaction module is used to generate third temporal interaction features; the aggregation module is used to apply multiple deep convolutional operations with different receptive fields to the multidimensional global correlation aggregation features, and perform weighted fusion and normalization processing through learnable weights to generate salient spatial features; the channel weight generation module is used to perform spatial convolution operations and global average pooling on the third temporal interaction features to obtain global channel representations, and extract channel weights through a bottleneck structure composed of two fully connected layers, and further utilize residual connections and The activation function yields the channel weights; the weighting module is used to weight the channel weights and significant spatial features along the channel dimension, and generate the first spatial reconstruction features through a single convolution and residual connection; The fourth gated timing channel interaction module is used to generate a fourth timing interaction feature based on the third timing interaction feature; the second channel guided local space reconstruction module is used to generate cascaded space reconstruction features based on the first space reconstruction feature and the fourth timing interaction feature.

8. The video crowd counting method based on temporal interaction and global correlation network according to claim 7, characterized in that, The feature fusion module is used to stitch together cascaded spatial reconstruction features and fourth temporal interaction features along the channel dimension, and to generate a continuous T-frame prediction value density map using multi-layer convolution operations.

Images