A video crowd counting method, device and computer readable storage medium

Abstract

This invention discloses a video crowd counting method, apparatus, and computer-readable storage medium, relating to the field of video crowd counting technology. The method includes acquiring a sequence of video frames to be processed; for each frame in the video frame sequence, generating a motion guide map based on the current frame and its temporally adjacent frames; extracting features from multiple consecutive frames in the video frame sequence and their corresponding motion guide maps to obtain static appearance features and dynamic motion features for each frame; fusing the static appearance features and dynamic motion features through a pre-constructed multi-path fusion network to obtain the final features; outputting a crowd density map of the current frame based on the final features, and calculating the estimated number of people in the current frame based on the crowd density map; the multi-path fusion network includes a dual-path reconstruction module and a global temporal context aggregation module.

Description

Technical Field

[0001] This invention relates to a video crowd counting method, apparatus, and computer-readable storage medium, belonging to the field of video crowd counting technology. Background Technology

[0002] With the rapid development of intelligent surveillance systems and the increasing frequency of large-scale public events, video crowd counting has become a key technology in areas such as public safety, traffic management, and crowd control. The goal of video crowd counting is to accurately estimate the number of people in each video frame. Current mainstream methods address this problem by estimating a crowd density map and summing the density maps.

[0003] Existing research on video crowd counting can be broadly categorized into two types: methods based on spatiotemporal correlation and methods based on auxiliary modality feature fusion. However, two main problems exist: First, the fusion mechanism of motion and appearance features is inadequate. Existing methods typically employ shallow or loosely coupled strategies to integrate dynamic motion information with static appearance features, lacking explicit decoupling and deep interaction at the feature level. Second, there is a lack of global modeling capabilities for long-term temporal dependencies. Existing auxiliary modality methods are mostly limited to characterizing short-term temporal changes, failing to effectively capture the overall motion trends and long-term temporal dependencies of the crowd across multiple time steps. Summary of the Invention

[0004] The purpose of this invention is to provide a video crowd counting method, device, and computer-readable storage medium. By starting from the perspective of "motion-appearance decoupling", a motion guidance graph is introduced and a multi-path fusion network is constructed to alleviate the background noise interference problem and improve the crowd counting accuracy in complex scenes.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution.

[0006] In a first aspect, the present invention provides a video crowd counting method, comprising:

[0007] Obtain the video frame sequence to be processed;

[0008] For each frame in the video frame sequence, a motion guidance map is generated based on the current frame and its temporally adjacent frames.

[0009] Feature extraction is performed on multiple consecutive frames and their corresponding motion guidance maps in the video frame sequence to obtain the static appearance features and dynamic motion features of each frame.

[0010] The static appearance features and dynamic motion features are fused through a pre-constructed multi-path fusion network to obtain the final features;

[0011] The crowd density map of the current frame is output based on the final features, and the estimated number of people in the current frame is calculated based on the crowd density map.

[0012] The pre-built multipath fusion network includes:

[0013] The dual-path reconstruction module is used to achieve the collaborative fusion and enhancement of the static appearance features and dynamic motion features from feature recalibration and spatial context association, so as to obtain fused and enhanced features;

[0014] The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

[0015] Further, generating the motion guidance map based on the current frame and its temporally adjacent frames includes:

[0016] Convert the current frame and adjacent temporally sequenced frames into single-channel grayscale images;

[0017] Based on the converted single-channel grayscale image, the pixel-level absolute difference map between the current frame and the temporally adjacent frames is calculated respectively.

[0018] The pixel-level absolute difference map is binarized, and pixels below a preset threshold are filtered out to obtain a forward binarized difference map and a backward binarized difference map.

[0019] Based on the forward binarized difference map and the backward binarized difference map, a motion guidance map is obtained by performing a pixel-by-pixel logical AND operation, retaining only the regions that change significantly in the forward binarized difference map and the backward binarized difference map.

[0020] Furthermore, the dual-path reconstruction module is based on motion guidance and includes a channel-space reconstruction submodule and a global attention reconstruction submodule; wherein,

[0021] The channel-space reconstruction submodule is used to generate channel attention weights and spatial attention weights based on the dynamic motion features; the static appearance features are multiplied element-wise with the channel attention weights and spatial attention weights to obtain differential guidance features;

[0022] The global attention reconstruction submodule uses the static appearance feature as the query and the dynamic motion feature as the key and value to perform global attention calculation, and then merges and refines the calculated output feature with the static appearance feature to obtain motion guidance feature.

[0023] The differential guidance feature and the motion guidance feature are concatenated along the channel dimension to obtain the fused enhancement feature.

[0024] Furthermore, the global temporal context aggregation module includes two parallel three-dimensional convolutional branches with different temporal dilation rates, which respectively extract short-range local dynamics and long-range global temporal correlations from the continuous static appearance features; the temporal features extracted by the two three-dimensional convolutional branches are concatenated along the channel dimension and a global aggregation operation is performed to obtain the global spatiotemporal context features; the global spatiotemporal context features and the fusion enhancement features are fused through element-level addition to obtain the final features.

[0025] Furthermore, the two parallel 3D convolutional branches with different temporal dilation rates include the same kernel size; one of the 3D convolutional branches has a temporal receptive field of 3, and the other 3D convolutional branch has a receptive field of 5.

[0026] Further, outputting the crowd density map of the current frame based on the final features includes:

[0027] The final feature and the dynamic motion feature are input into the pre-constructed multi-path fusion network for deep integration, and then fed into the progressive regression head; the progressive regression head uses a three-layer network architecture with decreasing channels to refine and generate the population density map layer by layer.

[0028] Furthermore, features are extracted from multiple consecutive frames in the video frame sequence and their corresponding motion guide maps using a pre-constructed feature coding network; wherein the feature coding network consists of multiple cascaded feature extraction stages, and the weights are shared when extracting features from the current frame and the motion guide map.

[0029] Furthermore, the feature encoding network is a visual Transformer model or a convolutional neural network model with the global average pooling layer and fully connected layer removed; wherein, the visual Transformer model includes the ShiftViT-T model; and the convolutional neural network model includes the ResNet model, the VGG model, and the ConvNext model.

[0030] Secondly, the present invention also provides a video crowd counting device, comprising:

[0031] The data acquisition module is configured to acquire the video frame sequence to be processed.

[0032] The motion guide graph generation module is configured to generate a motion guide graph for each frame in the video frame sequence, based on the current frame and its temporally adjacent frames.

[0033] The feature extraction module is configured to: extract features from multiple consecutive frames in the video frame sequence and their corresponding motion guidance maps to obtain the static appearance features and dynamic motion features of each frame;

[0034] The feature fusion module is configured to fuse the static appearance features and dynamic motion features through a pre-built multi-path fusion network to obtain the final features.

[0035] The people counting module is configured to: output the crowd density map of the current frame based on the final features, and calculate the estimated number of people in the current frame based on the crowd density map;

[0036] The pre-built multipath fusion network includes:

[0037] The dual-path reconstruction module is used to achieve the collaborative fusion and enhancement of the static appearance features and dynamic motion features from feature recalibration and spatial context association, so as to obtain fused and enhanced features;

[0038] The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

[0039] Thirdly, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the aforementioned video crowd counting method.

[0040] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0041] This invention proposes a video crowd counting method that employs a motion-guided multi-path fusion network. This fully leverages the complementary information between static appearance structure and dynamic motion cues, effectively achieving explicit decoupling and deep synergy between the two types of features. In video crowd counting tasks, it exhibits superior performance, stronger generalization ability, and higher robustness. Unlike traditional models that implicitly mix or strongly couple motion information with static features, this invention, from the perspective of "motion-appearance decoupling," constructs a unified framework capable of simultaneously and synergistically modeling spatial appearance features and temporal dynamic evolution patterns. This effectively mitigates background noise interference, thereby significantly improving the accuracy and spatiotemporal consistency of model density estimation. Furthermore, the multi-path fusion network proposed in this invention... The proposed multi-path fusion network includes a dual-path reconstruction module, which works collaboratively from two dimensions: feature-level adaptive calibration and global spatial context enhancement. This overcomes the limitations of traditional strong coupling between appearance and motion features, effectively decoupling static and dynamic features. This allows the model to accurately focus on the core area of ​​crowd movement, significantly improving robustness in crowded and occluded scenes. Furthermore, the proposed multi-path fusion network also includes a global temporal context aggregation module. This module employs a parallel multi-dilation rate 3D convolution strategy to provide receptive fields at different time scales. With lower parameter count and computational cost, this module simultaneously models both local short-range dynamics and global long-range evolution trends, effectively enhancing the smoothness and consistency of video crowd density estimation in the temporal dimension. Attached Figure Description

[0042] Figure 1 The diagram shown is a flowchart of the video crowd counting method provided in an embodiment of the present invention;

[0043] Figure 2 The diagram shown is a schematic representation of the overall framework of the multipath fusion network for the video crowd counting method provided in this embodiment of the invention.

[0044] Figure 3 The image shown is a schematic diagram illustrating the generation of a motion guidance map for a video crowd counting method provided in an embodiment of the present invention.

[0045] Figure 4 The diagram shown is a schematic representation of the motion-guided dual-path reconstruction module of the video crowd counting method provided in an embodiment of the present invention.

[0046] Figure 5 The diagram shows the structure of the global temporal context aggregation module of the video crowd counting method provided in this embodiment of the invention. Detailed Implementation

[0047] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0048] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0049] Example 1

[0050] See Figure 1 This embodiment provides a video crowd counting method, which specifically includes:

[0051] Step S1: Obtain the video frame sequence to be processed;

[0052] First, all frames in the video to be processed are divided into segments along the time dimension, forming training, validation, and test sets. For each consecutive video frame sequence input into the model from the training set, data augmentation operations are performed, specifically including applying uniform random cropping and horizontal flipping. Random cropping simulates changes in the field of view, differences in camera position, or partial occlusion by extracting different local regions from the video frame. Horizontal flipping mirrors the image horizontally, eliminating overfitting of the model to specific directions.

[0053] As a specific implementation method, this embodiment extracts continuous data for each training sample. T The video image is processed and randomly cropped to half the original resolution at a fixed spatial location, and then randomly flipped horizontally with a probability of 0.5.

[0054] Step S2: For each frame in the video frame sequence, generate a motion guide map based on the current frame and its temporally adjacent frames;

[0055] See Figure 3 To explicitly characterize instantaneous motion regions in a video sequence, a motion guide map is generated for each frame's RGB three-channel image in the video frame sequence, using the current frame and its preceding and following adjacent frames. Specifically, this includes:

[0056] Convert the current frame and adjacent frames in time sequence into single-channel grayscale images;

[0057] Calculate the pixel-level absolute difference between the current frame and its temporally adjacent frames;

[0058] The pixel-level absolute difference map is binarized, and pixels below a preset threshold are filtered out to obtain a forward binarized difference map and a backward binarized difference map.

[0059] Based on the forward binarized difference map and the backward binarized difference map, a motion guidance map is obtained by performing a pixel-by-pixel logical AND operation, retaining only the regions that show significant changes in the forward binarized difference map and the backward binarized difference map.

[0060] This embodiment uses a binarized three-frame difference map as a motion guide map to characterize the regions of significant motion change in the current frame. The significant regions refer to the pixels in the current frame that have changed compared to their preceding and following adjacent frames. See also... Figure 3 Figures (a), (b), and (c) in the image represent the grayscale images of the current frame and its absolute frame difference with the preceding and following temporally adjacent frames, respectively. For each frame in the input sequence, the three-channel RGB image is first processed... Convert to single-channel grayscale image Where H and W represent the height and width of the image of the i-th input frame, respectively. Then, the temporally adjacent frames before and after the current i-th input frame are calculated. and pixel-level absolute difference as well as Specifically, as shown in the following expression:

[0061] (1),

[0062] Since the directly obtained difference image may still be affected by factors such as lighting changes, background jitter, and slight camera movements, the difference result is further binarized. By setting a threshold, pixel changes below the threshold are considered noise and suppressed, thereby making the moving areas more concentrated and clear in spatial distribution. This binarization operation not only enhances the contrast of the foreground area but also significantly reduces the interference of irrelevant background changes on the subsequent feature learning process. Preferably, this embodiment sets a threshold. It will be lower than The pixel value is set to 0, higher than The pixel value is set to 255 to obtain a clean motion mask, as shown in the following expression:

[0063] (2),

[0064] In the formula, This represents the pixel value of the current i-th input frame after binarization at pixel position (x, y) with its neighboring frames; Indicates the index of adjacent frames, However, relying solely on unidirectional inter-frame differencing can still produce ghosting effects in low frame rate videos or still crowd scenes, such as... Figure 3 (e) and (f) in the text.

[0065] To further improve the robustness of motion representation, this embodiment utilizes both forward and backward difference results, and retains only the regions that are determined to be significantly changed in both difference results by pixel-by-pixel intersection, as shown in the following expression:

[0066] (3),

[0067] In the formula, This represents the motion guide graph of the generated i-th input frame; This represents the forward binarized difference graph between the i-th input frame and its preceding frame. This represents the backward binarized difference graph of the i-th input frame and its subsequent frames.

[0068] This strategy effectively filters out false detections caused by transient noise or non-uniform changes, explicitly and clearly capturing transient motion change regions between frames with almost no additional computational overhead. By setting a binarization threshold and a bidirectional intersection strategy, interference from illumination changes and dynamic background noise is effectively filtered out, providing the network with clean motion prior information. See also... Figure 3 The (d) in the image makes the final three-frame difference map more stable and closer to the motion distribution of real people.

[0069] This step helps reduce interference from transient noise or non-uniform changes, making the spatial distribution of the moving region more stable. The final generated three-frame difference map is a single-channel binary image, mainly used to describe the positional distribution of the foreground moving region. Considering that the subsequent feature encoding network is usually based on a three-channel model structure pre-trained on a large-scale image dataset, this invention uses channel duplication to expand the single-channel difference map into a three-channel form. The processed three-frame difference map serves as an auxiliary motion modality, inputting it in parallel with the original RGB frames into the feature encoding network, providing clear and reliable motion prior information for subsequent motion-appearance decoupling feature fusion.

[0070] Step S3: Extract features from multiple consecutive frames in the video frame sequence and their corresponding motion guidance maps to obtain the static appearance features and dynamic motion features of each frame;

[0071] As a specific implementation method, in this embodiment, multiple consecutive frames in the video frame sequence and their corresponding motion guide maps are extracted using a pre-constructed feature coding network. The feature coding network consists of multiple cascaded feature extraction stages, which share weights when extracting features from the current frame and the motion guide map.

[0072] In this embodiment, the feature encoding network uses the ShiftViT-T model with the global average pooling layer and fully connected layers removed. It should be understood that the above model is merely an example, and other feature encoding networks, such as ResNet, VGG, and ConvNext, may be used in other embodiments.

[0073] After feature extraction via a pre-constructed feature encoding network, the static appearance features and dynamic motion features of each frame at multiple scales are obtained in a unified feature space.

[0074] For the ShiftViT-T model, this embodiment uses weights pre-trained on ImageNet-1K for initialization. During model training, Mean Squared Error (MSE) is used as the objective function, and the AdamW optimizer is employed to reduce training loss. Regarding hyperparameter settings, a linear learning rate warm-up strategy is used for the first 15 epochs of training, followed by cosine annealing for learning rate decay. The initial learning rate is 1×10⁻⁶. -5 The weight decays to 1×10 -4 The maximum number of iterations is 300, and the model weights that perform best on the validation set are stored.

[0075] Step S4: The static appearance features and dynamic motion features are fused through a pre-constructed multi-path fusion network to obtain the final features;

[0076] The pre-built multipath fusion network includes:

[0077] The dual-path reconstruction module achieves the collaborative fusion and enhancement of the static appearance features and dynamic motion features through feature recalibration and spatial context association, resulting in fused and enhanced features;

[0078] The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

[0079] See Figure 4 Specifically, static appearance features and dynamic motion features are input into a motion-guided dual-path reconstruction module; the dual-path reconstruction module includes a channel-space reconstruction submodule, and the calculation process includes:

[0080] Step S4A1: To stabilize the training process and unify the feature space, layer normalization is used for static appearance features. and dynamic motion characteristics Perform layer normalization The static appearance features after layer normalization are obtained. and dynamic motion characteristics ;

[0081] Step S4A2: Convert the normalized static appearance features and dynamic motion characteristics It is projected onto a uniform channel dimension through a learnable linear layer. To ensure dimensional consistency in subsequent fusion operations, the following expression is used:

[0082] (4),

[0083] In the formula, express After having learnable parameters linear projection Subsequent frame features; express After having learnable parameters linear projection The subsequent frame difference features.

[0084] Step S4A3: The channel-spatial reconstruction submodule will extract frame difference features. Replace them with channel attention weights and spatial attention weights .

[0085] Among them, frame difference features First, global average pooling is applied in the spatial dimension. This achieves the separation of spatial information; secondly, through two... Convolutional layer The implemented bottleneck structure first reduces and then restores the channel dimension, two... Convolutional layer The ReLU activation function is used for connection, which can reduce the amount of computation and parameters while introducing nonlinearity to prevent overfitting, and channel attention weights. The output shape is Specifically, as shown in the following expression:

[0086] (5),

[0087] In the formula, This represents the Sigmoid activation function.

[0088] Meanwhile, spatial attention weights Through a combination of two Spatially varying modules composed of convolutional layers are generated, wherein two... Convolutional layer The ReLU activation function is used for connection in the middle. The first one... Convolutional layer Frame difference features The channel dimension is halved to reduce redundancy, the second Convolutional layer Further, the reduced channel dimensions are compressed into a single channel to capture the spatial importance map, highlighting key spatial locations and spatial attention weights. The output shape is Specifically, as shown in the following expression:

[0089] (6).

[0090] Step S4A4: After that, frame features By comparing with channel attention weights and spatial attention weights Element-wise multiplication is performed for refinement, thereby enhancing the network's sensitivity to dynamic crowd areas, as shown in the following expression:

[0091] (7),

[0092] In the formula, Indicates differential guided features; This represents element-wise multiplication.

[0093] Step S4A5: See Figure 4 The dual-path reconstruction module also includes a global attention reconstruction submodule, which converts the frame features obtained through projection in step S4A2 into a global attention reconstruction module. As the query Q, the frame difference feature obtained after projection Simultaneously serving as both key K and value V, as shown in the following expression:

[0094] (8),

[0095] In the formula, , as well as These represent the keys Q, K, and V used to generate the query Q, key K, and value V, respectively. Convolutional projection.

[0096] Step S4A6: To reduce computational complexity, this embodiment performs attention calculation before proceeding with the calculation. Convolutional layers halve the channel dimensions of Q and K.

[0097] Step S4A7: Perform attention calculation, as shown in the following expression:

[0098] (9),

[0099] In the formula, This represents the spatial attention matrix, which is mainly used to capture the correlation between static appearance features and dynamic motion features; This represents the features output after attention calculation. Indicates transpose. This represents the dimension of Q.

[0100] Step S4A8: Calculate the features output from the attention obtained in step S4A7. Features of the original frame By combining these features, we obtain the spliced ​​characteristics along the channel dimension. To preserve rich spatial information and through a lightweight post-processing network Further refinement is performed to generate motion-guided features. Specifically, as shown in the following expression:

[0101] (10)

[0102] As a specific implementation method, a lightweight post-processing network include Convolutional layers Convolutional layers, layer normalization, and the ReLU activation function.

[0103] Step S4A9: The refined differential guided features obtained in step S4A4 are... and motion guidance features By stitching along the channel dimension, we obtain the stitched features. And the channel dimension is restored to its original value through a linear projection layer. This generates fusion enhancement features. Specifically, as shown in the following expression:

[0104] (11),

[0105] In the formula, Indicates by Channel mapping layers composed of convolutions are used to restore the channel dimensions to their original form. , This indicates the changed channel dimension.

[0106] The dual-path design leverages motion cues in the difference graph and operates at three complementary levels: channel recalibration, local spatial emphasis, and global spatial correlation modeling. Through the channel-space dual-path modeling mechanism, the model adaptively highlights important channels and suppresses redundant channels under the guidance of the temporal context. This ensures that the features of the current frame incorporate temporal changes from previous and subsequent frames, improving the representational power and stability of the current frame features.

[0107] The fusion enhancement features obtained from the above steps The input is fed into the global time series context aggregation module, and the calculation process includes:

[0108] Step S4B1: This step constructs a global temporal context aggregation module, mainly including two parallel 3D convolutional branches with different temporal dilation rates, which respectively extract short-range local dynamics and long-range global temporal correlations from the continuous static appearance features. As a specific implementation, this embodiment constructs temporal receptive fields of 3 frames and 5 frames respectively, and combines the temporal features extracted by the two parallel branches... and The features are obtained by stitching along the channel dimension. ,in, Indicates the time convolution kernel size And a bottleneck block with an expansion rate of 1, Indicates the time convolution kernel size And a bottleneck block with an expansion rate of 2.

[0109] Step S4B2: Perform aggregation operation along the time dimension using a three-dimensional average pooling layer, as shown in the following expression:

[0110] (12)

[0111] In the formula, Represents global spatiotemporal context features; This represents a three-dimensional global average pooling layer.

[0112] Step S4B3: Apply the global spatiotemporal context features obtained in step S4B2 and the fusion enhancement features obtained in step S4A9 Element-wise addition is used for fusion, effectively injecting global temporal information into the spatial features of each frame. This achieves deep fusion of features across weights, spatial dimensions, and temporal dimensions, as shown in the following expression:

[0113] (13)

[0114] In the formula, This indicates element-wise addition; This represents the final feature, which serves as the input to the subsequent decoder.

[0115] Step S5: Output the crowd density map of the current frame based on the final features, and calculate the estimated number of people in the current frame based on the crowd density map;

[0116] See Figure 5 To further capture microscopic details, this embodiment will finalize the features. and dynamic motion characteristics The data is then upsampled again to 1 / 8 of the original resolution and fed into a second identical multipath fusion network for deep integration. After completion, redundant difference map features are discarded, and the final features are fed into a progressive regression head, such as... Figure 2 As shown in the trapezoid on the right, the regression head uses a three-layer network architecture with decreasing channels to refine and generate the population density map layer by layer. The specific structure is shown in the following expression:

[0117] (14)

[0118] In the formula, Indicates having Number of input channels A two-dimensional convolutional layer with k output channels and a kernel size of k; GELU represents the Gaussian error linear unit activation function.

[0119] In actual testing, the continuous video sequences in the test set were compared with the binarized three-frame differences extracted according to the same rules. Figure 1 The same input is fed into the optimal network model to obtain the crowd prediction density map for consecutive frames.

[0120] By performing a global spatial integral summation on the pixel values ​​of the crowd prediction density map, the accurate estimated number of people contained in each current frame can be output.

[0121] Example 2

[0122] Based on Embodiment 1, this embodiment proposes a video crowd counting device, comprising:

[0123] The data acquisition module is configured to acquire the video frame sequence to be processed.

[0124] The motion guide map generation module is configured to: generate a motion guide map for each frame in the video frame sequence, based on the current frame and its temporally adjacent frames; wherein the motion guide map is used to characterize the region of significant motion change in the current frame;

[0125] The feature extraction module is configured to extract features from multiple consecutive frames and their corresponding motion guide maps in the video frame sequence through a feature coding network to obtain the static appearance features and dynamic motion features of each frame.

[0126] The feature fusion module is configured to fuse the static appearance features and dynamic motion features through a pre-built multi-path fusion network to obtain the final features.

[0127] The people counting module is configured to: output the crowd density map of the current frame based on the final features, and calculate the estimated number of people in the current frame based on the crowd density map;

[0128] The pre-built multipath fusion network includes:

[0129] The dual-path reconstruction module achieves the collaborative fusion and enhancement of the static appearance features and dynamic motion features through feature recalibration and spatial context association, resulting in fused and enhanced features;

[0130] The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

[0131] Example 3

[0132] Based on Embodiment 1, this embodiment proposes a computer-readable storage medium storing a computer program / instruction thereon. When the computer program / instruction is executed by a processor, it implements the steps of the video crowd counting method proposed in Embodiment 1.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for counting people in a video, characterized in that, include: Obtain the video frame sequence to be processed; For each frame in the video frame sequence, a motion guidance map is generated based on the current frame and its temporally adjacent frames. Feature extraction is performed on multiple consecutive frames and their corresponding motion guidance maps in the video frame sequence to obtain the static appearance features and dynamic motion features of each frame. The static appearance features and dynamic motion features are fused using a pre-built multi-path fusion network to obtain the final features; The crowd density map of the current frame is output based on the final features, and the estimated number of people in the current frame is calculated based on the crowd density map. The pre-built multipath fusion network includes: The dual-path reconstruction module is used to achieve the collaborative fusion and enhancement of the static appearance features and dynamic motion features from feature recalibration and spatial context association, so as to obtain fused and enhanced features; The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

2. The video crowd counting method according to claim 1, characterized in that, The generation of the motion guidance map based on the current frame and its temporally adjacent frames includes: Convert the current frame and adjacent frames in time sequence into single-channel grayscale images; Based on the converted single-channel grayscale image, the pixel-level absolute difference map between the current frame and the temporally adjacent frames is calculated respectively. The pixel-level absolute difference map is binarized, and pixels below a preset threshold are filtered out to obtain a forward binarized difference map and a backward binarized difference map. Based on the forward binarized difference map and the backward binarized difference map, a motion guidance map is obtained by performing a pixel-by-pixel logical AND operation, retaining only the regions that change significantly in the forward binarized difference map and the backward binarized difference map.

3. The video crowd counting method according to claim 1, characterized in that, The dual-path reconstruction module is based on motion guidance and includes a channel-space reconstruction submodule and a global attention reconstruction submodule; wherein, The channel-space reconstruction submodule is used to generate channel attention weights and spatial attention weights based on the dynamic motion features; the static appearance features are multiplied element-wise with the channel attention weights and spatial attention weights to obtain differential guidance features; The global attention reconstruction submodule uses the static appearance feature as the query and the dynamic motion feature as the key and value to perform global attention calculation, and then merges and refines the calculated output feature with the static appearance feature to obtain motion guidance feature. The differential guidance feature and the motion guidance feature are concatenated along the channel dimension to obtain the fused enhancement feature.

4. The video crowd counting method according to claim 1, characterized in that, The global temporal context aggregation module includes two parallel three-dimensional convolutional branches with different temporal dilation rates, which respectively extract short-range local dynamics and long-range global temporal correlations from the continuous static appearance features; the temporal features extracted by the two three-dimensional convolutional branches are concatenated along the channel dimension and a global aggregation operation is performed to obtain the global spatiotemporal context features; The global spatiotemporal context features and the fusion enhancement features are fused together using element-level addition to obtain the final features.

5. The video crowd counting method according to claim 4, characterized in that, The two parallel 3D convolutional branches with different temporal dilation rates include the same kernel size; one of the 3D convolutional branches has a temporal receptive field of 3, and the other has a receptive field of 5.

6. The video crowd counting method according to claim 1, characterized in that, The crowd density map of the current frame is output based on the final features, including: The final feature and the dynamic motion feature are input into the pre-constructed multi-path fusion network for deep integration, and then fed into the progressive regression head; the progressive regression head uses a three-layer network architecture with decreasing channels to refine and generate the population density map layer by layer.

7. The video crowd counting method according to claim 1, characterized in that, The video frame sequence and its corresponding motion guide map are processed by a pre-constructed feature coding network for feature extraction. The feature coding network consists of multiple cascaded feature extraction stages, and the weights are shared when extracting features from the current frame and the motion guide map.

8. The video crowd counting method according to claim 7, characterized in that, The feature encoding network is a visual Transformer model or a convolutional neural network model with the global average pooling layer and fully connected layers removed; wherein, the visual Transformer model includes the ShiftViT-T model; and the convolutional neural network model includes the ResNet model, the VGG model, and the ConvNext model.

9. A video crowd counting device, characterized in that, include: The data acquisition module is configured to acquire the video frame sequence to be processed. The motion guide graph generation module is configured to generate a motion guide graph for each frame in the video frame sequence, based on the current frame and its temporally adjacent frames. The feature extraction module is configured to: extract features from multiple consecutive frames in the video frame sequence and their corresponding motion guidance maps to obtain the static appearance features and dynamic motion features of each frame; The feature fusion module is configured to fuse the static appearance features and dynamic motion features through a pre-built multi-path fusion network to obtain the final features. The people counting module is configured to: output the crowd density map of the current frame based on the final features, and calculate the estimated number of people in the current frame based on the crowd density map; The pre-built multipath fusion network includes: The dual-path reconstruction module is used to achieve the collaborative fusion and enhancement of the static appearance features and dynamic motion features from feature recalibration and spatial context association, so as to obtain fused and enhanced features; The global temporal context aggregation module is used to capture the temporal dependencies in the video frame sequence based on the continuous static appearance features, obtain global spatiotemporal context features, and fuse the fusion enhancement features with the global spatiotemporal context features to obtain the final features.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the video crowd counting method according to any one of claims 1 to 8.

Images