A multi-scale adaptive crowd counting method fusing temporal characteristics
By using adaptive illumination correction and an improved Transformer network, the problem of illumination and scale differences in crowd counting in large public places was solved, achieving higher counting accuracy and continuity.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2025-07-31
- Publication Date
- 2026-06-26
Smart Images

Figure CN121095859B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of crowd counting, and in particular to a multi-scale adaptive crowd counting method that integrates temporal characteristics. Background Technology
[0002] In recent years, deep learning methods have become the mainstream technique in crowd counting. This technique typically learns the mapping relationship between the input image and a crowd density map. The density map is a two-dimensional matrix of the same size as the original image, where each pixel value represents the crowd density at that location. By integrating the entire density map, the total number of people can be obtained. Compared to traditional object detection methods, this approach can better address the problems of dense crowds and severe occlusion.
[0003] However, the monitoring environment in large public places is extremely complex, posing a severe challenge to existing crowd counting technologies. These challenges mainly stem from the inherent characteristics of video images: First, to cover a wide area, surveillance cameras typically use high-angle shots, which not only results in generally smaller target sizes and blurred details in the image, but also creates significant scale differences due to the perspective effect of "nearer objects appear larger and farther objects smaller," demanding extremely high scale adaptability from the model. Second, complex ambient lighting, such as the interplay of indoor and outdoor light sources or localized strong light, easily causes severe uneven brightness issues within a single frame, with both overexposure and underexposure. Finally, crowds themselves have dynamic temporal characteristics, and their movement and changes often result in noise such as motion blur and momentary occlusion in single-frame images. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a multi-scale adaptive crowd counting method that integrates temporal characteristics. This method can adaptively handle the problem of large differences in illumination, mine multi-scale human features, utilize global time information and video temporal information, enhance the adaptability to complex environments, and improve the accuracy of crowd counting.
[0005] To achieve the above objectives, the technical solution provided by this invention is: a multi-scale adaptive crowd counting method that integrates temporal characteristics, comprising the following steps:
[0006] S1: Acquire video and preprocess the video, that is, extract an image sequence containing T images from the video according to the time sequence;
[0007] S2: Use adaptive instance normalization to perform adaptive illumination correction on the image sequence containing T images to obtain the image sequence after adaptive illumination correction;
[0008] S3: The image sequence after adaptive illumination correction is fed into the improved Swin-Transformer network, which outputs a multi-scale feature map set. The improvements to the Swin-Transformer network include: adding global temporal location encoding on the basis of location encoding. This global temporal location encoding embeds weekend and weekday information, holiday information, hour and minute information, enabling the network to obtain crowd density probability cues and improving the network's adaptability to different scenes; adding a linear layer after each Swin-Transformer Block to transform the outputs of different Swin-Transformer Blocks into the same dimension; obtaining the multi-scale feature map of each image in the image sequence by summing and averaging the outputs of multiple linear layers; merging the multi-scale feature maps of all images together to form a multi-scale feature map set, which is the output of the improved Swin-Transformer network.
[0009] S4: The multi-scale feature map set is fed into the improved Vision Transformer network, outputting a multi-scale feature map that integrates temporal characteristics. The improvements to the Vision Transformer network include: inverting the embedding vectors input to the Vision Transformer network to change the input of the multi-scale feature map set from a single feature map to multiple feature vectors at the same location within the multi-scale feature map set, thereby integrating the temporal characteristics of the multi-scale feature map set and mining dynamic information about people; adding a global time-location encoding to the location encoding, which embeds weekend / weekday, holiday, hour, and minute information, to further enhance crowd density probability cues based on the global time-location encoding added to the improved Swin-Transformer network, improving the network's adaptability to different scenarios. The improved Vision Transformer network outputs a multi-scale feature map that integrates temporal characteristics.
[0010] S5: A fully connected layer is used to regress the multi-scale feature map that incorporates temporal characteristics to obtain the crowd density map;
[0011] S6: Sum the pixel values of the crowd density map, and use the sum as the final crowd count.
[0012] Furthermore, the specific steps of step S2 are as follows:
[0013] S21: For an image sequence V = {I1, I2, ..., I...} containing T images, i ,...,I TThe global illumination information for each image in the image sequence is obtained through global average pooling, and then the channel-level scaling factor γ and bias factor β are predicted through two fully connected layers.
[0014] γ i =FC(GAP(I) i ))
[0015] β i =FC(GAP(I) i ))
[0016] In the formula, I i Represents the i-th image in the image sequence, γ i and β i Let represent the scaling factor and bias factor of the i-th image, respectively; FC() represents a fully connected layer; and GAP() represents global average pooling.
[0017] S22: Using scaling factor γ i and bias factor β i For the input image I i Adaptive instance normalization is performed to obtain the adaptive illumination-corrected image I`. i :
[0018] I i `=AdaIN(I i ,γ i ,β i )=γ i *(I i -μ(I i )) / δ(I i )+β i
[0019] In the formula, I` i Let AdaIN() represent the i-th image after adaptive illumination correction, and let μ(I) represent adaptive instance normalization. i ) and δ(I i ) respectively represent I i The mean and standard deviation;
[0020] Image sequence V = {I1, I2, ..., I i ,...,I T After adaptive instance normalization, the illumination-adaptive calibrated image sequence V` = {I`1, I`2, ..., I` is obtained. i ,...,I` T}
[0021] Furthermore, in step S3, the improved Swin-Transformer network is described in detail below:
[0022] Building upon the positional encoding of the Swin-Transformer network, a global temporal positional encoding is added to improve the network. This global temporal positional encoding embeds weekend / weekday information, holiday information, and hour and minute information, each represented by different values. These values are concatenated into a vector, which is then projected onto the same dimension as the input features of the Swin-Transformer network through a linear layer, thus obtaining the global temporal positional encoding.
[0023]
[0024] In the formula, Denotes the real number field, d ST The input features of the Swin-Transformer network are represented by the dimension of Linear(), which represents a linear layer, Holiday represents holiday information, Week represents weekend and weekday information, Hour represents hour information, and Minute represents minute information.
[0025] To further improve the Swin-Transformer network, a linear layer is added after each Swin-Transformer Block to map the feature maps of different scales output by each Swin-Transformer Block to the same dimension, resulting in an illumination-adaptive corrected image I'. i After passing through the Swin-TransformerBlock and linear layers, we obtain a multi-scale feature map set F. i ={P i 1 ,P i 2 ,...,P i K}, P i K Represents the i-th image with adaptive illumination correction, I` i The feature map obtained after passing through the Kth Swin-Transformer Block and a linear layer; the multi-scale feature map set F i ={P i 1 ,P i 2 ,...,P i K Calculate the mean to obtain the fused multi-scale feature map of the i-th image.
[0026]
[0027] IViTB Q ()
[0028]
[0029]
[0030] In the formula, STB K () represents the Kth Swin-Transformer Block, This represents the output of the i-th image in the k-th Swin-Transformer Block. This represents the output of the i-th image in the (K-1)-th Swin-TransformerBlock, where LE() represents linear embedding, PP() represents image block division, and PE... i This represents the positional encoding of the i-th image, and Mean() represents the mean operation.
[0031] For an image sequence V` = {I`1, I`2, ..., I` with adaptive illumination correction i ,...,I` T After the improvement of the Swin-Transformer network, a set of T fused multi-scale feature maps is obtained.
[0032] Furthermore, in step S4, the improved Vision Transformer network is described in detail as follows: By inverting the embedding vectors input to the Vision Transformer network, the improved Vision Transformer network is obtained. Specifically, this is achieved by fusing a multi-scale feature map set. The input is inverted, and the original feature map is replaced with multiple feature vectors from the same location in a multi-scale feature map set to fuse the temporal characteristics of the multi-scale feature map set and extract dynamic information about the person.
[0033] To further improve the Vision Transformer network, a global temporal location encoding is added to the positional encoding of the Vision Transformer network. The global temporal location encoding embeds information on weekends and weekdays, holidays, hours and minutes, which further enhances the crowd density probability cues obtained by the network. Different values are used to represent information on weekends and weekdays, holidays, hours and minutes of the day. These values are concatenated into a vector, and then a linear layer is used to project the concatenated vector onto the same dimension as the input features of the Vision Transformer network.
[0034] Feature map set After applying the improved Vision Transformer network, a multi-scale feature map F integrating temporal characteristics is obtained. fused :
[0035]
[0036]
[0037] In the formula, IViT() represents the improved Vision Transformer network; ViT() represents the Vision Transformer network, which consists of Q Vision Transformer Blocks; Invert() represents inverting the input vector. This represents the global time and location encoding of the improved Vision Transformer network. Holiday represents holiday information, Week represents weekend / weekday information, Hour represents hour information, Minute represents minute information, and ViTB represents the global time and location encoding. Q () represents the Q-th Vision Transformer Block. This represents the output of the Qth Vision Transformer Block, and Linear() represents a linear layer.
[0038] Furthermore, in step S5, a fully connected layer is used to fuse the multi-scale feature map F of temporal characteristics. fused Regression yields the population density map D. pred :
[0039] D pred =FC(F fused )
[0040] In the formula, FC() represents a fully connected layer.
[0041] Furthermore, in step S6, the population density map D is... pred The pixel values are summed, and the sum is used as the final crowd count value C.
[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0043] 1. The adaptive instance normalization process used in this invention can map image features under different lighting conditions into a unified and standardized feature space, which enhances the network's robustness to changes in lighting, thereby enabling accurate feature extraction even in overexposed or dimly lit areas.
[0044] 2. The global time-location encoding of the present invention enables the network to obtain crowd density probability information, thereby improving the counting accuracy under different crowd densities.
[0045] 3. The improved Swin-Transformer network of this invention realizes dynamic fusion of multi-scale features, solves the scale difference problem, and improves the counting accuracy in wide-angle and long-distance shooting scenarios.
[0046] 4. The improved Vision Transformer network of this invention models the temporal relationships within image sequences, utilizing contextual information to correct instantaneous errors in individual images, resulting in output count values that are smoother, more continuous, and more reliable over time. Furthermore, the network can learn the motion patterns of real "people," effectively distinguishing static humanoid interference objects in the background and reducing false alarm rates. Attached Figure Description
[0047] Figure 1 This is a schematic diagram of the logic flow of the present invention.
[0048] Figure 2 This is a schematic diagram illustrating an example of the neural network structure of the present invention. Detailed Implementation
[0049] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0050] like Figure 1 and Figure 2 As shown, this embodiment discloses a multi-scale adaptive crowd counting method that integrates temporal characteristics. It introduces adaptive instance normalization and improves the Swin-Transformer and Vision Transformer networks to solve crowd counting problems in complex environments. The method includes the following steps:
[0051] Step 1: Acquire video and preprocess it. This involves extracting an image sequence containing T images from the video in chronological order. Label the position of the head in the first image of the sequence as a tag, record the number of heads as the actual crowd count, and generate a realistic crowd density map using the head positions.
[0052] Step 2: For an image sequence V = {I1, I2, ..., I...} containing T images... i ,...,I T The global illumination information for each image in the image sequence is obtained through global average pooling, and then the channel-level scaling factor γ and bias factor β are predicted through two fully connected layers.
[0053] γ i=FC(GAP(I) i ))
[0054] β i =FC(GAP(I) i ))
[0055] In the formula, I i Represents the i-th image in the image sequence, γ i and β i Let represent the scaling factor and bias factor of the i-th image, respectively. Let FC() represent a fully connected layer and GAP() represent global average pooling.
[0056] Use scaling factor γ i and bias factor β i For the input image I i Adaptive instance normalization is performed to obtain the adaptive illumination-corrected image I`. i :
[0057] I` i =AdaIN(I i ,γ i ,β i )=γ i *(I i -μ(I i )) / δ(I i )+β i
[0058] In the formula, I` i Let AdaIN() represent the i-th image after adaptive illumination correction, and let μ(I) represent adaptive instance normalization. i ) and δ(I i ) respectively represent I i The mean and standard deviation;
[0059] Image sequence V = {I1, I2, ..., I i ,...,I T After adaptive instance normalization, the illumination-adaptive calibrated image sequence V` = {I`1, I`2, ..., I` is obtained. i ,...,I` T Adaptive instance normalization can adaptively adjust the brightness of each input image at the pixel level to eliminate the effects of uneven lighting such as overexposure and underexposure.
[0060] Step 3: Building upon the location encoding of the Swin-Transformer network, a global temporal location encoding is added to improve the network. This global temporal location encoding embeds weekend / weekday, holiday, hour, and minute information, enabling the network to obtain crowd density probability cues and enhancing its adaptability to different scenarios. Different values are used to represent weekend / weekday, holiday, hour, and minute information, which are then concatenated into a vector. This vector is then projected onto the same dimension as the input features of the Swin-Transformer network through a linear layer to obtain the global temporal location encoding.
[0061]
[0062] In the formula, Denotes the real number field, d ST The dimension of the input features of the Swin-Transformer network is represented by `Linear()`, where `Holiday` represents holiday information, `Week` represents weekend / weekday information, `Hour` represents hourly information, and `Minute` represents minute information. Since cameras installed in public places mostly use wide-angle lenses, the images captured show people of varying sizes and exhibit multi-scale characteristics. Therefore, to further improve the Swin-Transformer network, a linear layer is added after each Swin-Transformer Block to map the feature maps of different scales output by each Swin-Transformer Block to the same dimension. The image I' is then adaptively corrected for illumination. i After passing through the Swin-Transformer Block and the linear layer, we obtain a multi-scale feature map set F. i ={P i 1 ,P i 2 ,...,P i K}, P i K Represents the i-th image with adaptive illumination correction, I` i The feature map obtained after passing through the Kth Swin-Transformer Block and a linear layer. For the multi-scale feature map set F... i ={P i 1 ,P i 2 ,...,P i K Calculate the mean to obtain the fused multi-scale feature map of the i-th image.
[0063]
[0064] IViTB Q ()
[0065]
[0066] In the formula, STB K () represents the Kth Swin-Transformer Block, This represents the output of the i-th image in the k-th Swin-Transformer Block. This represents the output of the i-th image in the (K-1)-th Swin-TransformerBlock, where LE() represents linear embedding, PP() represents patch partitioning, and PE... i This represents the position code of the i-th image. This represents the global temporal location encoding of the improved Swin-Transformer network. Linear() represents a linear layer, and Mean() represents the mean operation.
[0067] For an image sequence V` = {I`1, I`2, ..., I` with adaptive illumination correction i ,...,I` T After the improvement of the Swin-Transformer network, a set of T fused multi-scale feature maps is obtained.
[0068] Step 4: Single images are easily blurred due to human movement, and many public places have billboards and statues containing people, which can easily lead to problems such as failure to recognize or misjudgment. Therefore, we can improve the recognition accuracy under conditions of image blur and interference by mining the temporal characteristics of multiple images. To achieve this, we need to improve the Vision Transformer network. By inverting the embedding vectors input to the Vision Transformer network, we obtain the improved Vision Transformer network. Specifically, this is done by fusing a set of feature maps from multiple scales. The input is inverted, changing the original feature map input to multiple feature vectors at the same location in a multi-scale feature map set, in order to fuse the temporal characteristics of the multi-scale feature map set and extract dynamic information about the person.
[0069] To further improve the Vision Transformer network, a global temporal location encoding is added to the existing location encoding. This global temporal location encoding embeds weekend / weekday, holiday, hour, and minute information, further enhancing the network's ability to provide crowd density probability cues. Different values are used to represent weekend / weekday, holiday, and hour / minute information, which are then concatenated into a vector. This vector is then projected onto the same dimension as the input features of the Vision Transformer network through a linear layer.
[0070] Feature map set After applying the improved Vision Transformer network, a multi-scale feature map F integrating temporal characteristics is obtained. fused :
[0071]
[0072] In the formula, IViT() represents the improved Vision Transformer network; ViT() represents the Vision Transformer network, which consists of Q Vision Transformer Blocks; Invert() represents inverting the input vector. This represents the global time and location encoding of the improved Vision Transformer network. Holiday represents holiday information, Week represents weekend / weekday information, Hour represents hour information, Minute represents minute information, and ViTB represents the global time and location encoding. Q () represents the Q-th Vision Transformer Block. This represents the output of the Qth Vision Transformer Block, and Linear() represents a linear layer.
[0073] Step 5: Use a fully connected layer to fuse the multi-scale feature map F of temporal characteristics. fused Regression yields the population density map D. pred :
[0074] D pred =FC(F fused )
[0075] In the formula, FC() represents a fully connected layer.
[0076] Step Six: Analyze the population density map D pred The pixel values are summed, and the sum is used as the final crowd count value C.
[0077] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A multi-scale adaptive crowd counting method that integrates temporal characteristics, characterized in that, Includes the following steps: S1: Acquire video and preprocess the video, that is, extract an image sequence containing T images from the video according to the time sequence; S2: Use adaptive instance normalization to perform adaptive illumination correction on the image sequence containing T images to obtain the image sequence after adaptive illumination correction; S3: The image sequence after adaptive illumination correction is fed into the improved Swin-Transformer network, which outputs a multi-scale feature map set. The improvements to the Swin-Transformer network include: adding global temporal location encoding on the basis of location encoding. This global temporal location encoding embeds weekend and weekday information, holiday information, hour and minute information, enabling the network to obtain crowd density probability cues and improving the network's adaptability to different scenes; adding a linear layer after each Swin-Transformer Block to transform the outputs of different Swin-Transformer Blocks into the same dimension; obtaining the multi-scale feature map of each image in the image sequence by summing and averaging the outputs of multiple linear layers; merging the multi-scale feature maps of all images together to form a multi-scale feature map set, which is the output of the improved Swin-Transformer network. S4: The multi-scale feature map set is fed into the improved Vision Transformer network, outputting a multi-scale feature map that integrates temporal characteristics. The improvements to the Vision Transformer network include: inverting the embedding vectors input to the Vision Transformer network to change the input of the multi-scale feature map set from a single feature map to multiple feature vectors at the same location within the multi-scale feature map set, thereby integrating the temporal characteristics of the multi-scale feature map set and mining dynamic information about people; adding a global time-location encoding to the location encoding, which embeds weekend / weekday, holiday, hour, and minute information, to further enhance crowd density probability cues based on the global time-location encoding added to the improved Swin-Transformer network, improving the network's adaptability to different scenarios. The improved Vision Transformer network outputs a multi-scale feature map that integrates temporal characteristics. S5: A fully connected layer is used to regress the multi-scale feature map that incorporates temporal characteristics to obtain the crowd density map; S6: Sum the pixel values of the crowd density map, and use the sum as the final crowd count.
2. The multi-scale adaptive crowd counting method integrating temporal characteristics according to claim 1, characterized in that, The specific steps for step S2 are as follows: S21: For an image sequence V = {I1, I2, ..., I...} containing T images, i ,...,I T The global illumination information for each image in the image sequence is obtained through global average pooling, and then the channel-level scaling factor γ and bias factor β are predicted through two fully connected layers. γ i =FC(GAP(I i )) β i =FC(GAP(I i )) In the formula, I i Represents the i-th image in the image sequence, γ i and β i Let represent the scaling factor and bias factor of the i-th image, respectively; FC() represents a fully connected layer; and GAP() represents global average pooling. S22: Using scaling factor γ i and bias factor β i For the input image I i Adaptive instance normalization is performed to obtain the adaptive illumination-corrected image I`. i : I` i =AdaIN(I i ,c i ,b i )=γ i *(I i -μ(I i )) / δ(I i )+b i In the formula, I` i Let AdaIN() represent the i-th image after adaptive illumination correction, and let μ(I) represent adaptive instance normalization. i ) and δ(I i ) respectively represent I i The mean and standard deviation; Image sequence V = {I1, I2, ..., I i ,...,I T After adaptive instance normalization, the illumination-adaptive calibrated image sequence V` = {I`1, I`2, ..., I` is obtained. i ,...,I` T } 3. The multi-scale adaptive crowd counting method integrating temporal characteristics according to claim 2, characterized in that, In step S3, the improved Swin-Transformer network is described in detail below: Building upon the positional encoding of the Swin-Transformer network, a global temporal positional encoding is added to improve the network. This global temporal positional encoding embeds weekend / weekday information, holiday information, and hour and minute information, each represented by different values. These values are concatenated into a vector, which is then projected onto the same dimension as the input features of the Swin-Transformer network through a linear layer, thus obtaining the global temporal positional encoding. In the formula, Denotes the real number field, d ST The input features of the Swin-Transformer network are represented by the dimension of Linear(), which represents a linear layer, Holiday represents holiday information, Week represents weekend and weekday information, Hour represents hour information, and Minute represents minute information. To further improve the Swin-Transformer network, a linear layer is added after each Swin-Transformer Block to map the feature maps of different scales output by each Swin-Transformer Block to the same dimension, resulting in an illumination-adaptive corrected image I. i After passing through the Swin-TransformerBlock and linear layers, a multi-scale feature map set is obtained. P i K Represents the i-th image with adaptive illumination correction, I` i The feature map obtained after passing through the Kth Swin-Transformer Block and a linear layer; a multi-scale feature map set. Calculate the mean to obtain the fused multi-scale feature map of the i-th image. IViTB Q () In the formula, STB K () represents the Kth Swin-Transformer Block, This represents the output of the i-th image in the k-th Swin-Transformer Block. This represents the output of the i-th image in the (K-1)-th Swin-Transformer Block, where LE() represents linear embedding, PP() represents image block division, and PE() represents the output of the i-th image in the (K-1)-th Swin-Transformer Block. i This represents the positional encoding of the i-th image, and Mean() represents the mean operation. For an image sequence V` = {I`1, I`2, ..., I` with adaptive illumination correction i ,...,I` T After the improvement of the Swin-Transformer network, a set of T fused multi-scale feature maps is obtained.
4. The multi-scale adaptive crowd counting method integrating temporal characteristics according to claim 3, characterized in that, In step S4, the improved Vision Transformer network is described in detail below: By inverting the embedding vectors input to the Vision Transformer network, the improved Vision Transformer network is obtained. Specifically, this is achieved by fusing a multi-scale feature map set. The input is inverted, and the original feature map is replaced with multiple feature vectors from the same location in a multi-scale feature map set to fuse the temporal characteristics of the multi-scale feature map set and extract dynamic information about the person. To further improve the Vision Transformer network, a global temporal location encoding is added to the positional encoding of the Vision Transformer network. The global temporal location encoding embeds information on weekends and weekdays, holidays, hours and minutes, which further enhances the crowd density probability cues obtained by the network. Different values are used to represent information on weekends and weekdays, holidays, hours and minutes of the day. These values are concatenated into a vector, and then a linear layer is used to project the concatenated vector onto the same dimension as the input features of the Vision Transformer network. Feature map set After applying the improved Vision Transformer network, a multi-scale feature map F integrating temporal characteristics is obtained. fused : In the formula, IViT() represents the improved Vision Transformer network; ViT() represents the Vision Transformer network, which consists of Q Vision Transformer Blocks; Invert() represents inverting the input vector. This represents the global time and location encoding of the improved Vision Transformer network. Holiday represents holiday information, Week represents weekend / weekday information, Hour represents hour information, Minute represents minute information, and ViTB represents the global time and location encoding. Q () represents the Q-th VisionTransformer Block. This represents the output of the Qth Vision Transformer Block, and Linear() represents a linear layer.
5. The multi-scale adaptive crowd counting method incorporating temporal characteristics according to claim 4, characterized in that, In step S5, a fully connected layer is used to fuse the multi-scale feature map F of temporal characteristics. fused Regression yields the population density map D. pred : D pred =FC(F fused ) In the formula, FC() represents a fully connected layer.
6. The multi-scale adaptive crowd counting method incorporating temporal characteristics according to claim 5, characterized in that, In step S6, the population density map D is processed. pred The pixel values are summed, and the sum is used as the final crowd count value C.