Video crowd counting method, device, terminal equipment and storage medium

Abstract

The application discloses a video crowd counting method, device and equipment and a storage medium, which comprises the following steps: inputting image features of a to-be-counted image sequence corresponding to a to-be-counted image into a decoder of a target deep neural network model; extracting first spatial features in the to-be-counted image features through a local spatial self-attention module; extracting first time features in the to-be-counted image features through a global time self-attention module; generating a first crowd density map based on the first spatial features and the first time features through the target deep neural network model, and determining a target crowd density map corresponding to the to-be-counted image sequence based on the first crowd density map; and adding the target crowd density map corresponding to the to-be-counted image sequence pixel by pixel through the target deep neural network model to obtain a crowd counting result corresponding to the to-be-counted image sequence. The application realizes the space-time correlation between image sequences in the crowd counting algorithm and improves the counting accuracy of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a video crowd counting method, apparatus, terminal device, and computer-readable storage medium. Background Technology

[0002] With the vigorous development of society, economy and culture, the rapid advancement of urbanization and the dramatic growth of the global population, the flow and large-scale gathering of people are becoming increasingly frequent, and the development of intelligent video surveillance technology is receiving more and more attention and importance.

[0003] Currently, crowd counting in real-world video scenarios faces numerous challenges due to the diversity of camera shooting angles, the complexity and vastness of scenes, and the uncertainty of crowd distribution. These challenges include drastic scale changes caused by perspective effects, interference from complex backgrounds, uneven scene lighting distribution, and severe occlusion between crowds. Beyond these scene-level challenges, the accuracy and efficiency of crowd counting are also affected by the algorithm itself. In actual surveillance scenarios, crowd counting algorithms are often input to consecutive video frame sequences within the same scene. However, most current counting methods still use datasets composed of static images as evaluation benchmarks, ignoring the temporal and spatial correlations between frame sequences. Furthermore, to improve model accuracy, crowd counting algorithms must introduce more network modules and complex computational units, leading to higher demands on storage space and computing power, making it difficult to apply and deploy these algorithms in real-world scenarios.

[0004] In summary, how to reflect the spatiotemporal correlation between image sequences in crowd counting algorithms and improve the counting accuracy of the algorithms with low computational complexity has become a technical problem that urgently needs to be solved in the field of deep learning technology. Summary of the Invention

[0005] The main objective of this invention is to provide a video crowd counting method, apparatus, terminal device, and computer-readable storage medium. The aim is to reflect the spatiotemporal correlation between image sequences in the crowd counting algorithm and improve the counting accuracy with low computational complexity.

[0006] To achieve the above objectives, the present invention provides a video crowd counting method, the video crowd counting method comprising:

[0007] The image features corresponding to the image sequence to be counted are input into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module;

[0008] The first spatial feature is extracted from the features of the image to be counted by the local spatial self-attention module.

[0009] The first temporal feature is extracted from the features of the image to be counted by the global temporal self-attention module.

[0010] The target deep neural network model generates a first crowd density map based on the first spatial features and the first temporal features, and determines the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map.

[0011] The target deep neural network model is used to add the target crowd density map corresponding to the image sequence to be counted pixel by pixel to obtain the crowd counting result corresponding to the image sequence to be counted.

[0012] Optionally, the step of determining the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map includes:

[0013] After decrementing the current count value by one by the target deep neural network model, it is detected whether the count value is equal to zero. Before the step of inputting the features of the image to be counted corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the count value is set to a preset number of loops.

[0014] If so, the loop ends, and the first crowd density map is determined to be the target crowd density map corresponding to the image sequence to be counted;

[0015] If not, the first crowd density map is used as the feature of the image to be counted, and the process returns to the step of extracting the first spatial feature from the feature of the image to be counted through the local spatial self-attention module.

[0016] Optionally, the step of extracting the first spatial feature from the features of the image to be counted through the local spatial self-attention module includes:

[0017] The local spatial self-attention module divides the image features to be counted into blocks to obtain the image features of each block.

[0018] The local spatial self-attention module performs multi-head self-attention calculation on the image features of each block to obtain the second spatial feature in the image features to be counted.

[0019] The second spatial feature is normalized and linearly rectified by the local spatial self-attention module to obtain the first spatial feature in the image features to be counted.

[0020] Optionally, the step of extracting the first temporal feature from the features of the image to be counted through the global temporal self-attention module includes:

[0021] The global temporal self-attention module performs multi-head self-attention calculation on the features of the image to be counted to obtain the second temporal feature in the features of the image to be counted.

[0022] The second temporal feature is normalized and linearly rectified by the global temporal self-attention module to obtain the first temporal feature in the image features to be counted.

[0023] Optionally, before the step of inputting the image features corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the method further includes:

[0024] The sequence of images to be counted is input into the encoder of the target deep neural network model, and the encoder determines the features of the images to be counted corresponding to the sequence of images to be counted.

[0025] Optionally, the method further includes:

[0026] An image sequence training set is established based on the first image sequence and the actual crowd density map of the first image sequence;

[0027] The target deep neural network model is obtained by training the pre-constructed initial deep neural network model based on the image sequence training set.

[0028] Optionally, the step of training a pre-constructed initial deep neural network model based on the image sequence training set to obtain a target deep neural network model includes:

[0029] The first image sequence is input into a pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence;

[0030] Determine the loss function corresponding to the initial deep neural network model, and compare the target crowd density map and the actual crowd density map corresponding to the first image sequence based on the loss function to obtain the comparison result;

[0031] Based on the gradient descent optimizer and the comparison results, the parameters of the initial deep neural network model are optimized through backpropagation to obtain the target deep neural network model.

[0032] Furthermore, to achieve the above objectives, the present invention also provides a video crowd counting device, the video crowd counting device comprising:

[0033] The decoder module inputs the image features corresponding to the image sequence to be counted into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module;

[0034] The first spatial feature module extracts the first spatial features from the features of the image to be counted through the local spatial self-attention module;

[0035] The first temporal feature module extracts the first temporal features from the features of the image to be counted through the global temporal self-attention module;

[0036] The target crowd density map module generates a first crowd density map based on the first spatial features and the first temporal features through the target deep neural network model, and determines the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map;

[0037] The crowd counting result module adds the target crowd density map corresponding to the image sequence to be counted pixel by pixel using the target deep neural network model to obtain the crowd counting result corresponding to the image sequence to be counted.

[0038] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a video crowd counting program stored in the memory and executable on the processor, wherein the video crowd counting program of the terminal device, when executed by the processor, implements the steps of the video crowd counting method as described above.

[0039] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a video crowd counting program, which, when executed by a processor, implements the steps of the video crowd counting method described above.

[0040] This invention proposes a video crowd counting method, apparatus, terminal device, and computer-readable storage medium. The method involves inputting the features of the images to be counted corresponding to a sequence of images to be counted into a decoder of a target deep neural network model. The decoder includes a local spatial self-attention module and a global temporal self-attention module. The local spatial self-attention module extracts a first spatial feature from the features of the images to be counted. The global temporal self-attention module extracts a first temporal feature from the features of the images to be counted. The target deep neural network model generates a first crowd density map based on the first spatial and first temporal features, and determines a target crowd density map corresponding to the sequence of images to be counted based on the first crowd density map. The target deep neural network model then performs pixel-by-pixel summation on the target crowd density map corresponding to the sequence of images to be counted to obtain the crowd counting result for the sequence of images to be counted.

[0041] This invention addresses the challenges of spatiotemporal information modeling and multi-scale feature learning by introducing a self-attention mechanism that consumes fewer resources and is less dependent on external information. To mitigate the issues of high model complexity and parameter count, a feature decoder based on a fine-grained spatiotemporal self-attention mechanism is designed. This allows the representation of spatiotemporal correlation between image sequences in the crowd counting algorithm to be improved while maintaining low computational complexity. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0043] Figure 2 This is a flowchart illustrating the steps of the first embodiment of the video crowd counting method of the present invention;

[0044] Figure 3 This is a schematic diagram of the deep neural network model involved in an embodiment of the video crowd counting method of the present invention;

[0045] Figure 4 This is a schematic diagram of a loop-based fine-grained self-attention process involved in an embodiment of the video crowd counting method of the present invention;

[0046] Figure 5 This is a schematic diagram showing a quantitative comparison of the counting performance of various crowd counting methods involved in an embodiment of the video crowd counting method of the present invention;

[0047] Figure 6 This is a qualitative comparison diagram of the crowd density maps output by various crowd counting methods involved in an embodiment of the video crowd counting method of the present invention;

[0048] Figure 7 This is a schematic diagram of a local spatial enhancement module based on a self-attention mechanism, which is involved in an embodiment of the video crowd counting method of the present invention.

[0049] Figure 8This is a schematic diagram of a global temporal enhancement module based on a self-attention mechanism, which is part of an embodiment of the video crowd counting method of the present invention.

[0050] Figure 9 This is a schematic diagram of the functional modules of an embodiment of the video crowd counting device of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiment of the present invention.

[0054] The terminal device in this embodiment of the invention can be a terminal device applied in the field of deep learning technology. Specifically, the terminal device can be a smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0055] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a video crowd counting program.

[0058] exist Figure 1 In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client and communicate data with it; and processor 1001 can be used to call the video crowd counting program stored in memory 1005 and perform the following operations:

[0059] The image features corresponding to the image sequence to be counted are input into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module;

[0060] The first spatial feature is extracted from the features of the image to be counted by the local spatial self-attention module.

[0061] The first temporal feature is extracted from the features of the image to be counted by the global temporal self-attention module.

[0062] The target deep neural network model generates a first crowd density map based on the first spatial features and the first temporal features, and determines the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map.

[0063] The target deep neural network model is used to add the target crowd density map corresponding to the image sequence to be counted pixel by pixel to obtain the crowd counting result corresponding to the image sequence to be counted.

[0064] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005 and perform the following operations:

[0065] After decrementing the current count value by one by the target deep neural network model, it is detected whether the count value is equal to zero. Before the step of inputting the features of the image to be counted corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the count value is set to a preset number of loops.

[0066] If so, the loop ends, and the first crowd density map is determined to be the target crowd density map corresponding to the image sequence to be counted;

[0067] If not, the first crowd density map is used as the feature of the image to be counted, and the process returns to the step of extracting the first spatial feature from the feature of the image to be counted through the local spatial self-attention module.

[0068] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005 and perform the following operations:

[0069] The local spatial self-attention module divides the image features to be counted into blocks to obtain the image features of each block.

[0070] The local spatial self-attention module performs multi-head self-attention calculation on the image features of each block to obtain the second spatial feature in the image features to be counted.

[0071] The second spatial feature is normalized and linearly rectified by the local spatial self-attention module to obtain the first spatial feature in the image features to be counted.

[0072] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005 and perform the following operations:

[0073] The global temporal self-attention module performs multi-head self-attention calculation on the features of the image to be counted to obtain the second temporal feature in the features of the image to be counted.

[0074] The second temporal feature is normalized and linearly rectified by the global temporal self-attention module to obtain the first temporal feature in the image features to be counted.

[0075] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005, and before the step of inputting the features of the images to be counted corresponding to the sequence of images to be counted into the decoder of the target deep neural network model, the following operations are also performed:

[0076] The sequence of images to be counted is input into the encoder of the target deep neural network model, and the encoder determines the features of the images to be counted corresponding to the sequence of images to be counted.

[0077] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005, and perform the following operations:

[0078] An image sequence training set is established based on the first image sequence and the actual crowd density map of the first image sequence;

[0079] The target deep neural network model is obtained by training the pre-constructed initial deep neural network model based on the image sequence training set.

[0080] Optionally, the processor 1001 can also be used to call the video crowd counting program stored in the memory 1005 and perform the following operations:

[0081] The first image sequence is input into a pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence;

[0082] Determine the loss function corresponding to the initial deep neural network model, and compare the target crowd density map and the actual crowd density map corresponding to the first image sequence based on the loss function to obtain the comparison result;

[0083] Based on the gradient descent optimizer and the comparison results, the parameters of the initial deep neural network model are optimized through backpropagation to obtain the target deep neural network model.

[0084] Based on the aforementioned terminal devices, various embodiments of the video crowd counting method of the present invention are proposed.

[0085] Currently, crowd counting in real-world video scenarios faces numerous challenges due to the diversity of camera shooting angles, the complexity and vastness of scenes, and the uncertainty of crowd distribution. These challenges include drastic scale changes caused by perspective effects, interference from complex backgrounds, uneven scene lighting distribution, and severe occlusion between crowds. Beyond these scene-level challenges, the accuracy and efficiency of crowd counting are also affected by the algorithm itself. In actual surveillance scenarios, crowd counting algorithms are often input to consecutive video frame sequences within the same scene. However, most current counting methods still use datasets composed of static images as evaluation benchmarks, ignoring the temporal and spatial correlations between frame sequences. Furthermore, to improve model accuracy, crowd counting algorithms must introduce more network modules and complex computational units, leading to higher demands on storage space and computing power, making it difficult to apply and deploy these algorithms in real-world scenarios.

[0086] To address the aforementioned issues, this invention proposes a video crowd counting method. This method inputs the features of the images to be counted, corresponding to the sequence of images to be counted, into the decoder of a target deep neural network model. The decoder includes a local spatial self-attention module and a global temporal self-attention module. The local spatial self-attention module extracts the first spatial feature from the features of the images to be counted, and the global temporal self-attention module extracts the first temporal feature. Then, a first crowd density map is generated based on the first spatial and first temporal features, and a target crowd density map corresponding to the sequence of images to be counted is determined based on this first crowd density map. Finally, the target crowd density map is summed pixel-by-pixel to obtain the crowd counting result for the sequence of images to be counted. Thus, this invention addresses the difficulties of spatiotemporal information modeling and multi-scale feature learning by introducing a self-attention mechanism that consumes fewer resources and has lower dependence on external information. To address the problem of excessive model complexity and parameter count, a feature decoder based on a fine-grained spatiotemporal self-attention mechanism is designed. This achieves the representation of spatiotemporal correlation between image sequences in the crowd counting algorithm and improves the counting accuracy with low computational complexity.

[0087] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of the video crowd counting method of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the video crowd counting method of the present invention may, of course, execute the steps shown or described in a different order than that shown here.

[0088] In a first embodiment of the video crowd counting method of the present invention, the video crowd counting method of the present invention includes:

[0089] Step S10: Input the image features corresponding to the image sequence to be counted into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module;

[0090] In this embodiment, the terminal device inputs the image features corresponding to the image sequence to be counted into the decoder of the pre-trained target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module.

[0091] It's important to note that the attention mechanism (self-attention) in Transformers is a powerful machine learning technique used to extract information from an input sequence and enhance the representation of specific parts. The attention mechanism calculates the importance of each input position to a specific context and assigns a corresponding weight to each position. These weights can be used to calculate a weighted average of the input sequence, resulting in a context-sensitive representation. Self-attention, on the other hand, is a specific form of attention mechanism that calculates the relative importance of each position to other positions only within the same sequence; therefore, it is also known as internal attention.

[0092] Furthermore, in a feasible embodiment, prior to step S10 above, the video crowd counting method of the present invention may further include:

[0093] Step S101: Input the image sequence to be counted into the encoder of the target deep neural network model, and determine the image features to be counted corresponding to the image sequence through the encoder.

[0094] In this embodiment, the terminal device inputs the image sequence to be counted into the encoder of the pre-trained target deep neural network model, and then extracts the image features corresponding to the image sequence to be counted through the encoder.

[0095] For example, the main function of an encoder is to transform the input image into abstract, high-dimensional features. Here, we follow the mainstream work in the field of crowd counting, using the first ten convolutional layers of the VGG-16 pre-trained model as the encoder to obtain rich spatiotemporal feature information, i.e., the features of the image to be counted, without excessively compressing the feature size (one-eighth of the input image size).

[0096] Step S20: Extract the first spatial feature from the features of the image to be counted through the local spatial self-attention module;

[0097] In this embodiment, the terminal device extracts spatial features (hereinafter referred to as the first spatial feature for distinction) from the features of the image to be counted through the local spatial self-attention module of the decoder.

[0098] For example, the local spatial self-attention module of the decoder extracts unique information in the spatial dimension of the features of the image to be counted, namely the first spatial feature, based on the fine-grained spatial self-attention mechanism. This can also be understood as filtering out the irrelevant parts of the features of the image to be counted and mapping the originally coarse, blurry, and difficult-to-distinguish low-level features to a new feature space, thereby achieving stronger distinguishability in the spatial dimension.

[0099] Step S30: Extract the first temporal feature from the features of the image to be counted through the global temporal self-attention module;

[0100] In this embodiment, the terminal device extracts the temporal features (hereinafter referred to as the first temporal features for distinction) from the features of the image to be counted through the global temporal self-attention module of the decoder.

[0101] For example, the global spatial self-attention module of the decoder extracts unique information in the time dimension from the features of the image to be counted, namely the first temporal features, based on a fine-grained temporal self-attention mechanism.

[0102] It should be noted that, in the time domain, considering the limited time step of the input image sequence, a global temporal self-attention mechanism is adopted.

[0103] Step S40: Generate a first crowd density map based on the first spatial features and the first temporal features using the target deep neural network model, and determine the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map;

[0104] In this embodiment, the terminal device generates a crowd density map (hereinafter referred to as the first crowd density map for distinction) based on the first spatial features and the first temporal features through the target deep neural network model, and determines the crowd density map corresponding to the image sequence to be counted (hereinafter referred to as the target crowd density map for distinction) based on the first crowd density map.

[0105] For example, after the image sequence to be counted is input into the target deep neural network model, the decoder in the target deep neural network model extracts the first spatial feature and the first temporal feature from the image features to be counted corresponding to the image sequence to be counted, retains the first spatial feature and the first temporal feature from the image features to be counted, filters out the spatiotemporal features from the image features to be counted except for the first spatial feature and the first temporal feature, and then determines the target population density map corresponding to the image sequence to be counted based on the first population density map.

[0106] Furthermore, in a feasible embodiment, step S40 above, the step of "determining the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map using the target deep neural network model" may include:

[0107] Step S401: After decrementing the current count value by one by the target deep neural network model, it is detected whether the count value is equal to zero. Before the step of inputting the features of the image to be counted corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the count value is set to a preset number of loops.

[0108] In this embodiment, the terminal device decrements the current count value by one using the target deep neural network model and then checks whether the count value is equal to zero. Prior to step S10, the count value is set to a preset number of cycles.

[0109] For example, such as Figure 4 As shown in the diagram, the cyclic fine-grained self-attention process involves the encoder outputting image features for counting. After passing through the decoder SA-Block, these features are fed back to the decoder for repeated learning. This process is repeated K times, which is the preset number of iterations mentioned above. After completing K iterations, the refined features are dimensionality-reduced through three convolutional layers, ultimately yielding the crowd distribution density map corresponding to the image sequence to be counted, i.e., the target crowd density map, and the corresponding crowd counting results. The preset number of iterations is K.

[0110] It should be noted that when stacking decoder modules (SA-Block) in the traditional way, the number of model parameters increases significantly with the number of stacking iterations. However, when using a recursive strategy, the model exhibits better performance, and changes in this strategy have no impact on the model size. Clearly, when the number of stacking iterations increases to a certain extent, the model's performance does not improve proportionally; instead, it significantly reduces computational efficiency. As a trade-off between accuracy and efficiency, K is set to K=4 in this invention.

[0111] Step S402: If yes, then end the loop and determine that the first crowd density map is the target crowd density map corresponding to the image sequence to be counted;

[0112] In this embodiment, when the target deep neural network model detects that the count value is equal to zero, the loop ends and the current first population density map is determined to be the target population density map corresponding to the image sequence to be counted.

[0113] For example, when the target deep neural network model detects that the count value is equal to zero, it means that the decoder has performed learning 4 times and the loop termination condition has been reached. Therefore, the first population density map generated in the 4th learning process is used as the target population density map.

[0114] Step S403: If not, then use the first crowd density map as the feature of the image to be counted, and return to the step of extracting the first spatial feature from the feature of the image to be counted through the local spatial self-attention module.

[0115] In this embodiment, when the target deep neural network model detects that the count value is not equal to zero, it returns to execute the above step S20.

[0116] For example, at the end of the first learning iteration of the decoder, the current count is the preset number of loops, which is 4. The count is decremented by one, resulting in a count of 3. It is then determined that the count is not equal to zero, so the next loop continues.

[0117] For example, such as Figure 3 As shown in the diagram, the deep neural network model process is as follows: First, image features are determined based on the input image sequence. Then, the image features are input to a decoder based on a self-attention mechanism. The local spatial self-attention module and the global temporal self-attention module of the decoder cyclically refine the density map until the loop ends, generating the final target population density map.

[0118] It should be noted that refinement refers to the process of repeatedly learning spatiotemporal features. The features output from the decoder are then fed back into the decoder to repeatedly strengthen the spatial and temporal representation of the features.

[0119] Step S50: The target crowd density map is summed pixel by pixel using the target deep neural network model to obtain the crowd counting result corresponding to the image sequence to be counted.

[0120] In this embodiment, the terminal device uses a target deep neural network model to add pixel-by-pixel the target crowd density map corresponding to the image sequence to be counted, thereby obtaining the crowd counting result corresponding to the image sequence to be counted.

[0121] In this embodiment, the video crowd counting method of the present invention inputs the sequence of images to be counted into the encoder of a pre-trained target deep neural network model through a terminal device, and then extracts the image features corresponding to the sequence of images to be counted through the encoder; the terminal device inputs the image features corresponding to the sequence of images to be counted into the decoder of the pre-trained target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module; the terminal device extracts the first spatial feature from the image features to be counted through the local spatial self-attention module of the decoder; the terminal device extracts the first temporal feature from the image features to be counted through the global temporal self-attention module of the decoder; the terminal device then extracts the first temporal feature from the image features to be counted based on the first spatial feature and the first temporal feature of the target deep neural network model. A first crowd density map is generated based on the time features, and a target crowd density map corresponding to the image sequence to be counted is determined based on the first crowd density map. The terminal device decrements the current count value by one using the target deep neural network model and checks whether the count value is equal to zero. Before step S10, the count value is set to a preset number of loops. When the target deep neural network model detects that the count value is equal to zero, the loop ends and the current first crowd density map is determined to be the target crowd density map corresponding to the image sequence to be counted. When the target deep neural network model detects that the count value is not equal to zero, the process returns to step S20. The terminal device adds the target crowd density map corresponding to the image sequence to be counted pixel by pixel using the target deep neural network model to obtain the crowd count result corresponding to the image sequence to be counted.

[0122] Thus, in this embodiment of the invention, the features of the images to be counted corresponding to the image sequence to be counted are input into the decoder of the target deep neural network model. The decoder includes a local spatial self-attention module and a global temporal self-attention module. The local spatial self-attention module extracts the first spatial feature from the features of the images to be counted, and the global temporal self-attention module extracts the first temporal feature. Then, a first crowd density map is generated based on the first spatial and first temporal features, and a target crowd density map corresponding to the image sequence to be counted is determined based on the first crowd density map. Finally, the target crowd density map is summed pixel-by-pixel to obtain the crowd counting result corresponding to the image sequence to be counted. In this way, the present invention addresses the difficulties of spatiotemporal information modeling and multi-scale feature learning by introducing a self-attention mechanism that consumes fewer resources and has lower dependence on external information. To address the problem of excessive model complexity and parameter quantity, a feature decoder based on a fine-grained spatiotemporal self-attention mechanism is designed. This enables the spatiotemporal correlation between image sequences to be reflected in the crowd counting algorithm, and improves the counting accuracy of the algorithm with low computational complexity.

[0123] Furthermore, based on the first embodiment of the video crowd counting method of the present invention described above, a second embodiment of the video crowd counting method of the present invention is proposed.

[0124] In this embodiment, the video crowd counting method of the present invention, step S20 above may include:

[0125] Step S201: The image features to be counted are divided into blocks by the local spatial self-attention module to obtain the image features of each block.

[0126] In this embodiment, the terminal device divides the image features to be counted into blocks through the local spatial self-attention refinement of the decoder to obtain the image features of each block.

[0127] For example, such as Figure 7 As shown, this is a schematic diagram of a local spatial enhancement module based on a self-attention mechanism. In the spatial dimension, for the input X... S ∈R H×W×C (H and W are the height and width of the input feature, and C is the dimension of the input feature), which is divided into several blocks of size p×p by the terminal device.

[0128]

[0129]

[0130] Step S202: Perform multi-head self-attention calculation on the image features of each block using the local spatial self-attention module to obtain the second spatial features in the image features to be counted.

[0131] In this embodiment, the terminal device performs multi-head self-attention calculation on the image features of each block through the local spatial self-attention module of the decoder to obtain the spatial features (hereinafter referred to as the second spatial features for distinction) in the image features to be counted.

[0132] For example, Local Spatial Self-Attention, also known as the Local Spatial Booster (RSB), is defined as follows:

[0133] The unit of measurement is the size of the image features to be counted after the multi-head self-attention calculation of the image features of each block is completed.

[0134] Step S203: The second spatial feature is normalized and linearly rectified by the local spatial self-attention module to obtain the first spatial feature in the image features to be counted.

[0135] In this embodiment, the terminal device performs normalization and linear rectification processing on the second spatial features through the local spatial self-attention module of the decoder to obtain the first spatial features in the image to be counted.

[0136] For example, the decoder is mainly divided into two stages: spatial domain modeling and temporal domain modeling.

[0137] Spatial domain modeling can be defined as

[0138]

[0139] Among them, X S The input spatial characteristics are represented by RSB(·), which represents the local spatial enhancement module, LN represents layer normalization, and ReLU represents the rectified linear unit. and It is the weight matrix of the two fully connected layers that immediately follow the local space enhancement module, where r is the scaling factor in nonlinear dimensionality reduction.

[0140] It should be noted that the bottleneck network Transform, which follows the local space augmentation module, S (·) Further learning of spatial feature representations accelerated network convergence.

[0141] Furthermore, in a feasible embodiment, step S30 above may include:

[0142] Step S301: Perform multi-head self-attention calculation on the features of the image to be counted through the global temporal self-attention module to obtain the second temporal feature in the features of the image to be counted;

[0143] In this embodiment, the terminal device performs multi-head self-attention calculation on the image features to be counted through the global temporal self-attention module of the decoder to obtain the temporal features in the image features to be counted (hereinafter referred to as the second temporal feature for distinction).

[0144] For example, such as Figure 8 As shown, this is a schematic diagram of a global temporal enhancement module based on a self-attention mechanism. In the time domain, for input X... T ∈R T×H×W×C (T is the time step of the input sequence, H and W are the height and width of the input features, and C is the dimension of the input features), we use The operation transforms it into X′ T ∈R HW×T×C ,Right now Therefore, Global Temporal Self-Attention, also known as Global Temporal Booster (GTB), is defined as follows:

[0145]

[0146] Step S302: The second temporal feature is normalized and linearly rectified by the global temporal self-attention module to obtain the first temporal feature in the image features to be counted.

[0147] In this embodiment, the terminal device performs normalization and linear rectification processing on the second temporal features through the global temporal self-attention module of the decoder to obtain the first temporal features in the image features to be counted.

[0148] For example, time-domain modeling can be defined as

[0149]

[0150] Among them, X T This represents the input time feature sequence, and GTB(·) represents the global time augmentation module. and It is the weight matrix of the two fully connected layers that immediately follow the local space enhancement module.

[0151] For example, such as Figure 4 As shown in the diagram, the cyclic fine-grained self-attention process involves inputting a continuous sequence of image frames into a target deep neural network model. The encoder of the target deep neural network model extracts image features from the image sequence, and these features are then input into the decoder. The decoder's local spatial enhancement module extracts spatial features from the image features, and its global temporal enhancement module extracts temporal features from the image features. The spatial and temporal features are then normalized and linearly rectified to obtain a first crowd density map. This first crowd density map is then fed back to the decoder for repeated learning. This process is repeated K times, and the target crowd density map corresponding to the image sequence is output.

[0152] In this embodiment, the video crowd counting method of the present invention uses a terminal device to segment the features of the image to be counted into blocks through the local spatial self-attention refinement of the decoder, obtaining image features of each block; the terminal device uses the local spatial self-attention module of the decoder to perform multi-head self-attention calculation on each block of image features, obtaining the second spatial feature in the image features to be counted; the terminal device uses the local spatial self-attention module of the decoder to perform normalization and linear rectification on the second spatial feature, obtaining the first spatial feature in the image to be counted; the terminal device uses the global temporal self-attention module of the decoder to perform multi-head self-attention calculation on the image features to be counted, obtaining the second temporal feature in the image features to be counted; the terminal device uses the global temporal self-attention module of the decoder to perform normalization and linear rectification on the second temporal feature, obtaining the first temporal feature in the image features to be counted.

[0153] Thus, based on the bottleneck network Transform, which consists of two fully connected layers immediately following the self-attention module... S (·) and Transform T The (·) method further enhances the model's ability to represent spatial and temporal features using only a small number of parameters. Layer normalization, acting as a regularizer, simplifies the parameter optimization process. As the network deepens, residual connections used for feature fusion reduce training difficulty and accelerate model convergence.

[0154] Furthermore, based on the first and / or second embodiments of the video crowd counting method of the present invention described above, a third embodiment of the video crowd counting method of the present invention is proposed.

[0155] In this embodiment, the video crowd counting method of the present invention may further include:

[0156] Step A10: Establish an image sequence training set based on the first image sequence and the actual crowd density map of the first image sequence;

[0157] Step A20: Train the pre-constructed initial deep neural network model based on the image sequence training set to obtain the target deep neural network model.

[0158] In this embodiment, the terminal device establishes an image sequence training set based on a pre-collected image sequence (hereinafter referred to as the first image sequence for distinction) and the actual crowd density map of the first image sequence. Then, the device trains the pre-constructed initial deep neural network model based on the image sequence training set to obtain the trained target deep neural network model.

[0159] Furthermore, in a feasible embodiment, step A20 above may include:

[0160] Step A201: Input the first image sequence into the pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence;

[0161] In this embodiment, the terminal device inputs the first image sequence into a pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence.

[0162] For example, the encoder of the initial deep neural network model extracts spatiotemporal features from the first image sequence to obtain image features in the first image sequence. Then, the decoder in the initial deep neural network model extracts spatial and temporal features from the image features. After the decoder learns the spatiotemporal features K times, it outputs the target crowd density map corresponding to the first image sequence.

[0163] Step A202: Determine the loss function corresponding to the initial deep neural network model, and compare the target crowd density map and the actual crowd density map corresponding to the first image sequence based on the loss function to obtain the comparison result;

[0164] In this embodiment, the terminal device determines the loss function corresponding to the initial deep neural network model, and then compares the target crowd density map and the actual crowd density map corresponding to the first image sequence through the loss function to obtain the comparison result.

[0165] For example, the loss function corresponding to the initial deep neural network model is defined as:

[0166]

[0167] Where N is the number of samples in the training set, I i D represents the i-th input image. EST (I i ;Θ) is the output of a network with parameter Θ, D GT (I i ) is the density map corresponding to the actual situation.

[0168] Step A203: Based on the gradient descent optimizer and the comparison results, the parameters of the initial deep neural network model are optimized through backpropagation to obtain the target deep neural network model.

[0169] In this embodiment, based on the gradient descent optimizer and the above comparison results, the parameters of the presented deep neural network model are optimized through backpropagation to generate the optimized target deep neural network model.

[0170] For example, after completing the second iteration, the fine-tuned model features are fed into a regression network consisting of several convolutional layers to achieve feature dimensionality reduction and density map generation. Finally, the density map estimated by the model is compared with the true density map using a loss function, and the network parameters are trained and optimized using a gradient descent-based optimizer through backpropagation.

[0171] For example, such as Figure 5 As shown in the diagram, the counting performance of various population counting methods is quantitatively compared. The smaller the MAE (Mean Absolute Error) and RMSE (Root Mean Square Error), the better the counting performance of the corresponding method. It can be seen that the RFSNet method of this invention achieves the best performance. Figure 6 The diagram illustrates a qualitative comparison of crowd density maps output by various crowd counting methods. For each image, each column represents the input image, the true density map, and the predicted density maps of the proposed method RFSNet, image-based CSRNet, and video-based STDNet, respectively. The counting results are marked in the lower right corner of each density map, and the regions with the greatest differences in the density maps are highlighted. Clearly, the proposed model is capable of accurately estimating the density map while simultaneously making accurate counts. Particularly for the same regions between adjacent frames, RFSNet's predictions are more stable and consistent. Compared to other image- or video-based counting methods, RFSNet provides more accurate estimations of both the overall count and the details of the density map.

[0172] In this embodiment, the video crowd counting method of the present invention establishes an image sequence training set based on a pre-collected first image sequence and the actual crowd density map of the first image sequence using a terminal device. Then, it trains a pre-constructed initial deep neural network model based on the image sequence training set to obtain a trained target deep neural network model. The terminal device inputs the first image sequence into the pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence. The terminal device determines the loss function corresponding to the initial deep neural network model, and then compares the target crowd density map and the actual crowd density map corresponding to the first image sequence using the loss function to obtain the comparison result. Based on the gradient descent optimizer and the above comparison result, the parameters of the output deep neural network model are optimized through backpropagation to generate an optimized target deep neural network model.

[0173] Thus, during the training of the neural network model, this invention constrains the features of its final output density map and continuously updates the model parameters through gradient descent to reduce the distance between the predicted population density map and the density map in the actual situation, thereby improving the accuracy of model prediction.

[0174] In addition, embodiments of the present invention also provide a video crowd counting device.

[0175] Please refer to Figure 9 , Figure 9 This is a functional module diagram of an embodiment of the video crowd counting device of the present invention, as shown below. Figure 9 As shown, the video crowd counting device of the present invention includes:

[0176] Decoder module 10 is used to input the features of the images to be counted corresponding to the image sequence to the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module;

[0177] The first spatial feature module 20 is used to extract the first spatial feature from the features of the image to be counted through the local spatial self-attention module;

[0178] The first temporal feature module 30 is used to extract the first temporal features from the features of the image to be counted through the global temporal self-attention module;

[0179] The target crowd density map module 40 is used to generate a first crowd density map based on the first spatial features and the first temporal features through the target deep neural network model, and to determine the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map.

[0180] The crowd counting result module 50 is used to perform pixel-by-pixel summation on the target crowd density map corresponding to the image sequence to be counted through the target deep neural network model to obtain the crowd counting result corresponding to the image sequence to be counted.

[0181] Optionally, the target population density map module 40 includes:

[0182] The counting value detection unit is used to detect whether the current counting value is equal to zero after the target deep neural network model decrements the current counting value by one. The counting value is set to a preset number of loops before the step of inputting the features of the image to be counted corresponding to the image sequence to be counted into the decoder of the target deep neural network model.

[0183] The loop termination unit is used to terminate the loop if the condition is met, and to determine that the first crowd density map is the target crowd density map corresponding to the image sequence to be counted.

[0184] The loop continues, and if not, the first crowd density map is used as the feature of the image to be counted, and the process returns to the step of extracting the first spatial feature from the features of the image to be counted through the local spatial self-attention module.

[0185] Optionally, the first spatial feature module 20 includes:

[0186] The block processing unit is used to divide the image features to be counted into blocks through the local spatial self-attention module to obtain the image features of each block.

[0187] The second spatial feature unit is used to perform multi-head self-attention calculation on the image features of each block through the local spatial self-attention module to obtain the second spatial feature in the image features to be counted.

[0188] The first spatial feature unit is used to perform normalization and linear rectification processing on the second spatial feature through the local spatial self-attention module to obtain the first spatial feature in the image features to be counted.

[0189] Optionally, the first-time feature module 30 further includes:

[0190] The second temporal feature unit is used to perform multi-head self-attention calculation on the image features to be counted through the global temporal self-attention module to obtain the second temporal feature in the image features to be counted.

[0191] The first temporal feature unit is used to normalize and linearly rectify the second temporal feature through the global temporal self-attention module to obtain the first temporal feature in the image features to be counted.

[0192] Optionally, the video crowd counting device of the present invention further includes:

[0193] The encoder module is used to input the image sequence to be counted into the encoder of the target deep neural network model, and to determine the image features corresponding to the image sequence to be counted through the encoder.

[0194] Optionally, the video crowd counting device of the present invention further includes:

[0195] The training set module is used to establish an image sequence training set based on the first image sequence and the actual crowd density map of the first image sequence;

[0196] The model training module is used to train the pre-constructed initial deep neural network model based on the image sequence training set to obtain the target deep neural network model.

[0197] Optionally, the model training module includes:

[0198] The target crowd density map unit is used to input the first image sequence into a pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence.

[0199] The loss function unit is used to determine the loss function corresponding to the initial deep neural network model, and to compare the target crowd density map and the actual crowd density map corresponding to the first image sequence based on the loss function to obtain the comparison result;

[0200] The parameter optimization unit is used to optimize the parameters of the initial deep neural network model based on the gradient descent optimizer and the comparison results, through backpropagation, to obtain the target deep neural network model.

[0201] The present invention also provides a computer storage medium storing a video crowd counting program, which, when executed by a processor, implements the steps of the video crowd counting program method as described in any of the above embodiments.

[0202] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the video crowd counting program method of the present invention described above, and will not be repeated here.

[0203] The present invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the video crowd counting method of the present invention as described in any of the above embodiments, which will not be repeated here.

[0204] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0205] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (such as TWS earphones, etc.) to execute the methods described in the various embodiments of the present invention.

[0207] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method of video crowd counting, characterized by, The video crowd counting method includes: The image features corresponding to the image sequence to be counted are input into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module; The first spatial feature is extracted from the features of the image to be counted by the local spatial self-attention module. The first temporal feature is extracted from the features of the image to be counted by the global temporal self-attention module. The target deep neural network model generates a first crowd density map based on the first spatial features and the first temporal features, and determines the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map. The target deep neural network model is used to add the target crowd density map corresponding to the image sequence to be counted pixel by pixel to obtain the crowd counting result corresponding to the image sequence to be counted. The step of extracting the first temporal feature from the features of the image to be counted through the global temporal self-attention module includes: The global temporal self-attention module performs multi-head self-attention calculation on the features of the image to be counted to obtain the second temporal feature in the features of the image to be counted. The second temporal feature is normalized and linearly rectified by the global temporal self-attention module to obtain the first temporal feature in the image features to be counted.

2. The video crowd counting method of claim 1, wherein, The step of determining the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map includes: After decrementing the current count value by one by the target deep neural network model, it is detected whether the count value is equal to zero. Before the step of inputting the features of the image to be counted corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the count value is set to a preset number of loops. If so, the loop ends, and the first crowd density map is determined to be the target crowd density map corresponding to the image sequence to be counted; If not, the first crowd density map is used as the feature of the image to be counted, and the process returns to the step of extracting the first spatial feature from the feature of the image to be counted through the local spatial self-attention module.

3. The video crowd counting method of claim 1, wherein, The step of extracting the first spatial feature from the features of the image to be counted through the local spatial self-attention module includes: The local spatial self-attention module divides the image features to be counted into blocks to obtain the image features of each block. The local spatial self-attention module performs multi-head self-attention calculation on the image features of each block to obtain the second spatial feature in the image features to be counted. The second spatial feature is normalized and linearly rectified by the local spatial self-attention module to obtain the first spatial feature in the image features to be counted.

4. The video crowd counting method of claim 1, wherein, Before the step of inputting the image features corresponding to the image sequence to be counted into the decoder of the target deep neural network model, the method further includes: The sequence of images to be counted is input into the encoder of the target deep neural network model, and the encoder determines the features of the images to be counted corresponding to the sequence of images to be counted.

5. The video people counting method according to any one of claims 1 to 4, wherein, The method further includes: An image sequence training set is established based on the first image sequence and the actual crowd density map of the first image sequence; The target deep neural network model is obtained by training the pre-constructed initial deep neural network model based on the image sequence training set.

6. The video crowd counting method of claim 5, wherein, The step of training a pre-constructed initial deep neural network model based on the image sequence training set to obtain a target deep neural network model includes: The first image sequence is input into a pre-constructed initial deep neural network model to obtain the target crowd density map corresponding to the first image sequence; Determine the loss function corresponding to the initial deep neural network model, and compare the target crowd density map and the actual crowd density map corresponding to the first image sequence based on the loss function to obtain the comparison result; Based on the gradient descent optimizer and the comparison results, the parameters of the initial deep neural network model are optimized through backpropagation to obtain the target deep neural network model.

7. A video crowd counting device, characterized by, The video crowd counting device includes: The decoder module inputs the features of the images to be counted corresponding to the sequence of images to be counted into the decoder of the target deep neural network model, wherein the decoder includes a local spatial self-attention module and a global temporal self-attention module; The first spatial feature module extracts the first spatial features from the features of the image to be counted through the local spatial self-attention module; The first temporal feature module extracts a first temporal feature from the image features to be counted using the global temporal self-attention module; performs multi-head self-attention calculation on the image features to be counted using the global temporal self-attention module to obtain a second temporal feature from the image features to be counted; and performs normalization and linear rectification processing on the second temporal feature using the global temporal self-attention module to obtain the first temporal feature from the image features to be counted. The target crowd density map module generates a first crowd density map based on the first spatial features and the first temporal features through the target deep neural network model, and determines the target crowd density map corresponding to the image sequence to be counted based on the first crowd density map; The crowd counting result module adds the target crowd density map corresponding to the image sequence to be counted pixel by pixel using the target deep neural network model to obtain the crowd counting result corresponding to the image sequence to be counted.

8. A terminal device, comprising: The terminal device includes: a memory, a processor, and a video crowd counting program stored in the memory and executable on the processor, wherein the video crowd counting program, when executed by the processor, implements the steps of the video crowd counting method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a video crowd counting program, which, when executed by a processor, implements the steps of the video crowd counting method as described in any one of claims 1 to 6.

Images