Self-supervised crowd counting method based on color and texture features

Abstract

This invention discloses a self-supervised crowd counting method based on color and texture features, comprising three stages: In the self-supervised pre-training stage, firstly, pseudo-labels for crowd counting are generated by aggregating pixels with similar color or texture features from a large number of unlabeled crowd images. Then, a crowd counting model based on an improved U-Net is constructed, and self-supervised pre-training is performed using the unlabeled crowd images and their pseudo-labels to obtain the pre-trained model. In the supervised fine-tuning stage, the pre-trained model is fine-tuned using a small number of labeled crowd images to obtain the target model. In the prediction stage, the crowd image to be counted is input into the target model, and the crowd counting result for that image is output. This invention uses pseudo-labels for crowd counting in self-supervised pre-training, making the pre-training process closely related to the subsequent crowd counting task. When fusing shallow and deep features, spatial and channel attention are used to enhance features, improving the model's counting accuracy.

Description

Technical Field

[0001] This invention relates to the field of crowd counting technology, and in particular to a self-supervised crowd counting method based on color and texture features. Background Technology

[0002] Crowd counting is an important research task in the field of computer vision. By monitoring the density and distribution of crowds, it is possible to promptly detect safety risks such as overcrowding and stampedes in scenarios such as large-scale events, public transportation hubs, and urban evacuation routes. This provides early warning information to relevant departments, improves public safety, and protects the lives and property of the public.

[0003] Currently, with the rapid development of deep learning technology, convolutional neural networks (CNNs) are widely used in crowd counting, driving the rapid development of this field. Although CNN-based crowd counting methods have achieved significant results, deep CNN learning typically requires a large number of labeled image samples. Due to the high cost of manual annotation, it is often difficult to obtain a sufficient number of labeled image samples in practical applications. To address this issue, self-supervised learning techniques that can fully utilize unlabeled image samples have become a research hotspot.

[0004] Current self-supervised learning methods can be divided into two main categories: image reconstruction-based self-supervised learning methods and contrastive learning-based self-supervised learning methods. However, during self-supervised pre-training, both types of methods focus more on the similarity between image samples after different processing methods, resulting in a low correlation between the self-supervised pre-training process and the subsequent crowd counting task, thus affecting the crowd counting performance. Summary of the Invention

[0005] The problem this invention aims to solve is to provide a self-supervised crowd counting method based on color and texture features, addressing the shortcomings of existing technologies. By generating pseudo-labels for crowd counting for self-supervised pre-training, the pre-training process is closely related to the subsequent crowd counting task. When fusing shallow and deep features, spatial and channel attention are used to enhance features and improve the counting accuracy of the model.

[0006] This invention adopts the following technical solution: a self-supervised crowd counting method based on color and texture features, comprising six steps in three stages:

[0007] Phase 1: Self-supervised pre-training phase

[0008] Step 1: Obtain an unlabeled crowd image dataset and perform data preprocessing to use it as a self-supervised pre-training dataset;

[0009] Step 2: Construct a crowd counting model based on an improved U-Net;

[0010] Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the crowd counting model based on the improved U-Net to obtain the pre-trained model;

[0011] Phase 2: Supervised Fine-tuning Phase

[0012] Step 4: Obtain the labeled crowd image dataset and perform data preprocessing, dividing the dataset into training, validation, and test sets according to the proportions;

[0013] Step 5: Fine-tune the pre-trained model using the training and validation sets obtained in Step 4 to obtain the target model;

[0014] Phase 3, Prediction Phase

[0015] Step 6: Input the test set image obtained in Step 4 as the crowd image to be counted into the target model obtained in Step 5, and obtain the crowd counting result of the image.

[0016] Further, in step 1, obtaining the unlabeled crowd image dataset and performing data preprocessing includes the following sub-steps:

[0017] Step 1-1: Download or create your own unlabeled crowd image dataset from the internet;

[0018] Step 1-2: For each unlabeled image in the dataset, generate a pseudo-label map of crowd counts;

[0019] Steps 1-3: Resize the images and their pseudo-labels in the dataset;

[0020] Steps 1-4: Use the Gaussian kernel function to generate a pseudo-population density map from the pseudo-label map.

[0021] Further, in steps 1-2, for each unlabeled image in the dataset, a pseudo-label map for crowd counting is generated, including the following sub-steps:

[0022] Step 1-2-1: Let It is a three-dimensional pixel value matrix of an unlabeled crowd image, with a size of , Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, It is a preset integer. ;

[0023] Step 1-2-2: Generate a size of A three-dimensional matrix A, any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ;

[0024] Steps 1-2-3: Generate two sizes. Given two-dimensional matrices B and C, generate a matrix of size . Given two-dimensional matrices D, B, and C, any element... and correspond pixels on ,in, This indicates rounding to the nearest integer. , Initialize all elements in matrix B to False, and any element in matrix C... All elements in matrix D are initialized to 0.

[0025] Step 1-2-4: Select an element in matrix B whose value is False. ,make , Represented by pixels Centered Local Binary Pattern (LBP) features of the region , , ,in, Calculated based on matrix C, It is a pre-set odd number. and This represents the minimum and maximum values ​​of the pixel's x-axis index. and This represents the minimum and maximum values ​​of the pixel's y-axis index; d is a temporary variable.

[0026] Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The LBP characteristics of the region are ,from Surrounding distance equals Select two types of pixels from the available pixels:

[0027] One type is pixels with similar colors, referring to pixels whose red, green, and blue color channels are located in the same range. The pixels;

[0028] Another type is pixels with similar textures, requiring that the pixel be centered on the desired pixel. Local binary pattern characteristics of the region and The correlation is greater than the preset threshold;

[0029] Change the element value of matrix B corresponding to the selected pixel to True, and update it according to the x-axis and y-axis indices of the selected pixel. , , and ,in, , , LBP features are calculated based on matrix C;

[0030] Step 1-2-6: If no pixel was selected in step 1-2-5, proceed to step 1-2-7; otherwise, let... Return to steps 1-2-5;

[0031] Step 1-2-7: If , , and To meet the preset requirements for head size for crowd counting, If matrix B contains no element with a value of False, then matrix D is the pseudo-label map of the crowd count for the image img; otherwise, return to step 1-2-4.

[0032] Furthermore, in step 2, constructing a crowd counting model based on the improved U-Net includes the following sub-steps.

[0033] Step 2-1: Replace the shrinkage path of U-Net with an encoder consisting of one convolutional module and four downsampled convolutional modules connected in sequence; the convolutional module outputs feature map F1, and the first downsampled convolutional module, the second downsampled convolutional module, the third downsampled convolutional module, and the fourth downsampled convolutional module output feature maps F2, F3, F4, and F5 in sequence;

[0034] Step 2-2: Replace the U-Net's expansion path with a decoder consisting of one multi-attention upsampling feature fusion module and three multi-attention downsampling feature fusion modules connected in sequence; the multi-attention upsampling feature fusion module receives feature maps F4 and F5 and outputs feature map F6; the first multi-attention downsampling feature fusion module receives feature maps F3 and F6 and outputs feature map F7; the second multi-attention downsampling feature fusion module receives feature maps F2 and F7 and outputs feature map F8; the third multi-attention downsampling feature fusion module receives feature maps F1 and F8 and outputs feature map F9.

[0035] Steps 2-3: The final U-Net... The convolution is replaced by a density map generation module. This module receives the feature map F9, outputs a crowd density map, and then performs an integral operation on the crowd density map to obtain the number of people.

[0036] Further, in step 3, the crowd counting model based on the improved U-Net is trained using the self-supervised pre-training dataset obtained in step 1 to obtain a pre-trained model, including the following sub-steps:

[0037] Step 3-1: Set the initial parameters for model training;

[0038] Step 3-2: Use during training Loss and The loss is calculated using a loss function composed of norm losses; the gradient is calculated based on the loss, and the network model parameters are updated using the Adam optimizer;

[0039] Step 3-3: Evaluate the model using mean absolute error (MAE) and mean squared error (MSE);

[0040] Steps 3-4: Repeat the training process from steps 3-2 to 3-3. Evaluate the network model after each training round until the preset total number of training rounds is completed. Save the optimal network model as the pre-trained model.

[0041] Further, in step 4, the labeled crowd image dataset is obtained and preprocessed, and the dataset is divided into training, validation, and test sets proportionally, including the following sub-steps:

[0042] Step 4-1: Download or create a labeled crowd image dataset from an open-source dataset website;

[0043] Step 4-2: Resize the images and their labels in the dataset;

[0044] Step 4-3: Divide the scaled images and labels into training, validation, and test sets;

[0045] Step 4-4: Use the Gaussian kernel function to generate a real population density map from the labeled map.

[0046] Furthermore, in step 5, the pre-trained model is fine-tuned using the training set and validation set obtained in step 4 to obtain the target model, including the following sub-steps:

[0047] Step 5-1: Set the initial parameters for model training;

[0048] Step 5-2: Use during training Loss and The loss is calculated using a loss function composed of norm losses; the gradient is calculated based on the loss, and the network model parameters are updated using the Adam optimizer;

[0049] Step 5-3: Evaluate the model using the mean absolute error (MAE) and mean squared error (MSE) of the validation set;

[0050] Step 5-4: Repeat the training process from Step 5-2 to Step 5-3. After each training round, use the validation set to evaluate the network model until the preset total number of training rounds is completed. Save the best network model as the target model.

[0051] Further, in step 6, the test set image obtained in step 4 is input as the crowd image to be counted into the target model obtained in step 5 to obtain the crowd counting result of the image, including the following sub-steps:

[0052] Step 6-1: Import the target model saved in Step 5, input the test set images obtained in Step 4 as the crowd images to be counted into the target model, and output the final crowd density map.

[0053] Step 6-2: Calculate MAE and MSE values ​​and save the test results.

[0054] The present invention also provides: an electronic device, comprising:

[0055] One or more processors;

[0056] A storage device on which one or more programs are stored;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the self-supervised crowd counting methods described above.

[0058] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0059] 1. For a large number of unlabeled crowd images, this invention generates pseudo-labels for crowd counting by aggregating pixels with similar color or texture features. These pseudo-labels are used for self-supervised pre-training of a crowd counting model based on an improved U-Net, making the self-supervised pre-training process closely related to the subsequent crowd counting task and improving the counting accuracy of the model.

[0060] 2. When fusing shallow and deep features, this invention utilizes spatial attention and channel attention to enhance features, further improving the counting accuracy of the model. Attached Figure Description

[0061] Figure 1 This is a flowchart of the self-supervised crowd counting method based on color and texture features of the present invention;

[0062] Figure 2 Pixels in embodiments of the present invention Surrounding distance equals A diagram illustrating the pixels;

[0063] Figure 3 This is a schematic diagram of the crowd counting model structure based on the improved U-Net of this invention;

[0064] Figure 4 This is a schematic diagram of the convolution module structure of the present invention;

[0065] Figure 5 This is a schematic diagram of the downsampling convolution module structure of the present invention;

[0066] Figure 6 This is a schematic diagram of the multi-attention upsampling feature fusion module structure of the present invention;

[0067] Figure 7 This is a schematic diagram of the multi-attention downsampling feature fusion module structure of the present invention;

[0068] Figure 8 This is a schematic diagram of the density map generation module of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0070] In one embodiment of the present invention, a self-supervised crowd counting method based on color and texture features is provided, the specific process of which is as follows: Figure 1 As shown, the steps include:

[0071] Step 1: Obtain an unlabeled crowd image dataset and perform data preprocessing to serve as a self-supervised pre-training dataset.

[0072] Furthermore, step 1 specifically includes the following steps:

[0073] Step 1-1: Download or create an unlabeled crowd image dataset from the internet; the unlabeled crowd image dataset used in this embodiment contains 10,000 unlabeled crowd images collected from the internet;

[0074] Step 1-2: For each unlabeled image in the dataset, generate a pseudo-label map of the crowd image;

[0075] Furthermore, steps 1-2 specifically include the following steps:

[0076] Step 1-2-1: Let It is a 3D pixel value matrix of an unmarked urban scene image, with a size of ,in, Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, For a preset integer, ;

[0077] Preferably, this embodiment is set as follows: .

[0078] Step 1-2-2: Generate a size of A three-dimensional matrix A, any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ;

[0079] Steps 1-2-3: Generate two sizes. Given two-dimensional matrices B and C, generate a matrix of size . Given two-dimensional matrices D, B, and C, any element... and Corresponding pixels on img ,in, This indicates rounding to the nearest integer. , Initialize all elements in matrix B to False, and any element in matrix C... All elements in matrix D are initialized to 0.

[0080] Step 1-2-4: Select an element in matrix B whose value is False. ,make , Represented by pixels Centered Local Binary Pattern (LBP) features of the region , , ,in, Calculated based on matrix C, It is a pre-set odd number. and This represents the minimum and maximum values ​​of the pixel's x-axis index. and This represents the minimum and maximum values ​​of the pixel's y-axis index; d is a temporary variable.

[0081] Preferably, this embodiment is set as follows: .

[0082] Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The LBP characteristics of the region are ,from Surrounding distance equals Select two types of pixels from the available pixels:

[0083] One type is pixels with similar colors, referring to pixels whose red, green, and blue color channels are located in the same range. The pixels;

[0084] Another type is pixels with similar textures, requiring that the pixel be centered on the desired pixel. LBP characteristics of the region and The correlation is greater than the preset threshold;

[0085] Change the element value of matrix B corresponding to the selected pixel to True, and update it according to the x-axis and y-axis indices of the selected pixel. , , and ,in, , , The LBP features are calculated based on matrix C.

[0086] In this embodiment, pixels Surrounding distance equals pixels such as Figure 2 As shown, the dark gray area is Surrounding distance equals The pixels; the light gray area is Surrounding distance equals The pixels; The same principle applies to other situations.

[0087] Step 1-2-6: If no pixel was selected in step 1-2-5, proceed to step 1-2-7; otherwise, let... Return to steps 1-2-5;

[0088] Step 1-2-7: If , , and To meet the preset requirements for head size for crowd counting, ;

[0089] If matrix B contains no element with a value of False, then matrix D is the pseudo-label map of the crowd count for the image img; otherwise, return to steps 1-2-4.

[0090] Steps 1-3: Resize the images in the dataset and their pseudo-labels so that the longest side of the image does not exceed 2048 pixels.

[0091] Steps 1-4: Generate a pseudo-crowd density map from the pseudo-labeled map using a Gaussian kernel function; parameters of the Gaussian kernel function. Different sizes are used on different datasets; this embodiment sets... .

[0092] Step 2: Construct a crowd counting model based on the improved U-Net.

[0093] Furthermore, step 2 specifically includes the following steps:

[0094] Step 2-1: As Figure 3As shown, the shrinkage path of U-Net is replaced with an encoder consisting of one convolutional module and four downsampling convolutional modules connected in sequence; the convolutional module outputs feature map F1, and the first downsampling convolutional module, the second downsampling convolutional module, the third downsampling convolutional module, and the fourth downsampling convolutional module output feature maps F2, F3, F4, and F5 in sequence.

[0095] The structure of the convolutional module is as follows Figure 4 As shown, it contains two groups: The system consists of convolutional layers, batch normalization layers, and ReLU activation function layers. The padding of the convolutional layers is set to 1, and the number of channels is set to 64.

[0096] The structure of the downsampling convolution module is as follows: Figure 5 As shown, it contains one Maximum pooling layer and two sets The system consists of convolutional layers, batch normalization layers, and ReLU activation function layers. The stride of the max pooling layer is set to 2, and the padding of the convolutional layers is set to 1. The number of channels in the convolutional layers of the first, second, third, and fourth downsampling convolutional modules are set to 128, 256, 512, and 512, respectively.

[0097] Step 2-2: As Figure 3 As shown, the expansion path of U-Net is replaced with a decoder consisting of one multi-attention upsampling feature fusion module and three multi-attention downsampling feature fusion modules connected in sequence:

[0098] The multi-attention upsampling feature fusion module receives feature maps F4 and F5 and outputs feature map F6.

[0099] The first multi-attention downsampling feature fusion module receives feature maps F3 and F6 and outputs feature map F7.

[0100] The second multi-attention downsampling feature fusion module receives feature maps F2 and F7 and outputs feature map F8;

[0101] The third multi-attention downsampling feature fusion module receives feature maps F1 and F8 and outputs feature map F9.

[0102] The structure of the multi-attention upsampling feature fusion module is as follows: Figure 6 As shown, the input shallow and deep feature maps are multiplied by their respective spatial attention values. The deep feature map is then upsampled. The shallow and deep feature maps are then concatenated along the channels. The concatenated feature map is multiplied by its channel attention value, and then passed sequentially through... Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers yield the output feature map of the multi-attention upsampling feature fusion module. Among these, the spatial attention submodule... Convolution padding is set to 3, and the number of channels is set to 1; in the channel attention submodule... The number of channels in the convolution is the same as the number of channels in the input feature map; the final... The number of channels in the convolution is the same as the number of channels in the input shallow feature map. Convolution padding is set to 1, and the number of channels is the same as the number of channels in the input shallow feature map.

[0103] The structure of the multi-attention downsampling feature fusion module is as follows: Figure 7 As shown, the input shallow and deep feature maps are multiplied by their respective spatial attention values. The shallow feature map is then downsampled, and the shallow and deep feature maps are concatenated along the channels. The concatenated feature map is then multiplied by its channel attention value, and then passed sequentially through... Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers yield the output feature map of the multi-attention downsampling feature fusion module. Among these, the spatial attention submodule... Convolution padding is set to 3, and the number of channels is set to 1; in the channel attention submodule... The number of channels in the convolution is the same as the number of channels in the input feature map; the final... The number of channels in the convolution is the same as the number of channels in the input shallow feature map. Convolution padding is set to 1, and the number of channels is the same as the number of channels in the input shallow feature map.

[0104] Steps 2-3: The final U-Net... The convolution is replaced by a density map generation module. This module receives the feature map F9, outputs a crowd density map, and then performs an integral operation on the crowd density map to obtain the crowd size. The density map generation module is as follows: Figure 8 As shown, the input feature map F9 passes through sequentially Convolutional layers, batch normalization layers, ReLU activation function layers, Convolutional layers, batch normalization layers, and ReLU activation function layers output a crowd density map. Convolution padding is set to 1, and the number of channels is set to 64. The number of convolution channels is set to 1.

[0105] Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the crowd counting model based on the improved U-Net to obtain the pre-trained model.

[0106] Furthermore, step 3 specifically includes the following steps:

[0107] Step 3-1: Set the initial parameters for model training as follows:

[0108] Initial learning rate: 0.00001;

[0109] Optimizer: Adam optimizer;

[0110] Weight decay: 0.0001;

[0111] Batch size: 4;

[0112] Training epochs: 500;

[0113] Data augmentation: The input image is randomly cropped into 512×512 image blocks and horizontally flipped with a probability of 0.5. The RGB channels of the color image are swapped with a probability of 0.2. Gaussian noise with a mean of 0 and a standard deviation of 5 is added with a probability of 0.1. For datasets containing grayscale images, the color images are randomly converted to grayscale images with a probability of 0.1.

[0114] Step 3-2: Use during training Loss and The loss is calculated using a loss function derived from norm loss combinations. The formula for the loss function is as follows:

[0115]

[0116] in, This is the number of images to input. Image The difference in population size between the predicted density map and the actual density map.

[0117]

[0118]

[0119] in, It is a set of learnable network weights. Image The predicted density map picture The true density map;

[0120] Calculate the gradient based on the loss and use the Adam optimizer to update the network model parameters.

[0121] Step 3-3: Evaluate the model using Mean Absolute Error (MAE) and Mean Squared Error (MSE). The calculation methods for MAE and MSE are as follows:

[0122]

[0123]

[0124] in, It is a predicted count. These are the actual counts. The lower the values ​​of MAE and MSE, the better the counting performance of the model.

[0125] Steps 3-4: Repeat the training process from steps 3-2 to 3-3. After each training round, evaluate the network model until the preset total number of training rounds of 500 rounds is completed. Save the optimal network model as the pre-trained model according to the optimal MAE value.

[0126] Step 4: Obtain the labeled crowd image dataset and perform data preprocessing, dividing the dataset into training, validation, and test sets according to the proportions.

[0127] Furthermore, step 4 specifically includes the following steps:

[0128] Step 4-1: Download Part A of the ShanghaiTech dataset from the open-source dataset website;

[0129] Step 4-2: Resize the 300 training images and 182 test images and their labeled images in the downloaded dataset so that the longest side of the images does not exceed 2048 pixels, and group the images with similar numbers of people into one category according to the labeled images. The number of categories can be customized. In this invention, the number of categories is set to C=10.

[0130] Step 4-3: Place the scaled training images and their labeled images into the training set, and place the scaled test images and their labeled images into both the validation set and the test set;

[0131] Step 4-4: Generate a crowd density map from the marker map using a Gaussian kernel function; parameters of the Gaussian kernel function. Using different sizes on different datasets, preferably, in this embodiment, the size is set as follows: .

[0132] Step 5: Use the training set and validation set obtained in Step 4 to fine-tune the pre-trained model to obtain the target model.

[0133] Furthermore, step 5 specifically includes the following steps:

[0134] Step 5-1: Set the initial parameters for model training as follows:

[0135] Initial learning rate: 0.00001;

[0136] Optimizer: Adam optimizer;

[0137] Weight decay: 0.0001;

[0138] Batch size: 4;

[0139] Training epochs: 500;

[0140] Data augmentation: Randomly crop the input image to... The image patches are horizontally flipped with a probability of 0.5, the RGB channels of the color image are swapped with a probability of 0.2, and Gaussian noise with a mean of 0 and a standard deviation of 5 is added with a probability of 0.1. For datasets containing grayscale images, the color images are randomly converted to grayscale images with a probability of 0.1.

[0141] Step 5-2: Use during training Loss and loss function composed of norm loss Calculate the loss; calculate the gradient based on the loss, and update the network model parameters using the Adam optimizer;

[0142] Step 5-3: Evaluate the model using the mean absolute error (MAE) and mean squared error (MSE) of the validation set;

[0143] Step 5-4: Repeat the training process from Step 5-2 to Step 5-3. After each training round, use the validation set to evaluate the network model until the preset total number of training rounds of 500 rounds is completed. Save the optimal network model as the target model according to the optimal MAE value.

[0144] Step 6: Input the test set image obtained in Step 4 as the crowd image to be counted into the target model obtained in Step 5, and obtain the crowd counting result of the image.

[0145] Furthermore, step 6 specifically includes the following steps:

[0146] Step 6-1: Import the target model saved in Step 5, input the test set images obtained in Step 4 as the crowd images to be counted into the target model, and output the final crowd density map.

[0147] Step 6-2: Calculate MAE and MSE values ​​and save the test results.

[0148] In this embodiment of the invention, an electronic device is also provided, comprising: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors implement the self-supervised crowd counting method of the foregoing embodiments.

[0149] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A self-supervised crowd counting method based on color and texture features, characterized in that, It includes the following three stages: Phase 1: Self-supervised pre-training phase Step 1: Obtain an unlabeled crowd image dataset and perform data preprocessing to use it as a self-supervised pre-training dataset; Step 2: Construct a crowd counting model based on an improved U-Net; Step 3: Use the self-supervised pre-training dataset obtained in Step 1 to train the crowd counting model based on the improved U-Net to obtain the pre-trained model; Phase 2: Supervised Fine-tuning Phase Step 4: Obtain the labeled crowd image dataset and perform data preprocessing. Divide the labeled crowd image dataset into training set, validation set and test set according to the proportions. Step 5: Fine-tune the pre-trained model using the training and validation sets obtained in Step 4 to obtain the target model; Phase 3, Prediction Phase Step 6: Use the test set images obtained in Step 4 as the crowd images to be counted, input them into the target model obtained in Step 5, and obtain the crowd counting results for the images.

2. The self-supervised population counting method according to claim 1, characterized in that, Step 1 involves acquiring an unlabeled crowd image dataset and performing data preprocessing, including the following sub-steps: Step 1-1: Download or create your own unlabeled crowd image dataset from the internet; Step 1-2: For each unlabeled image in the dataset, generate a pseudo-label map of crowd counts; Steps 1-3: Resize the images and their pseudo-labels in the dataset; Steps 1-4: Use the Gaussian kernel function to generate a pseudo-population density map from the pseudo-label map.

3. The self-supervised population counting method according to claim 2, characterized in that, In steps 1-2, for each unlabeled image in the dataset, a pseudo-label map for crowd counting is generated, including the following sub-steps: Step 1-2-1: Represent the three-dimensional pixel value matrix of the unlabeled crowd image as follows: Size is , Indicates the height of the image. The width of the image is represented by 3, and the number 3 represents the three color channels: red, green, and blue. Pixel values ​​from 0 to 255 are divided into c intervals. Take the median value of each interval as the representative pixel value for that interval, and define the pixel value interval as follows: The representative pixel value is ,in, It is a preset integer. ; Step 1-2-2: Generate a size of A three-dimensional matrix A, where any element in matrix A correspond pixel values ,like The range of pixel values ​​is ,make ,in, , , ; Steps 1-2-3: Generate two sizes. Given two-dimensional matrices B and C, generate a matrix of size . Given two-dimensional matrices D, B, and C, any element... and correspond pixels on ,in, This indicates rounding to the nearest integer. , Initialize all element values ​​in matrix B to False, and represent any element in matrix C as... All elements in matrix D are initialized to 0. Step 1-2-4: Select an element in matrix B whose value is False. ,make , Represented by pixels Centered Local binary pattern characteristics of the region , , ,in, Calculated based on matrix C, It is a pre-set odd number. and This represents the minimum and maximum values ​​of the pixel's x-axis index. and This represents the minimum and maximum values ​​of the pixel's y-axis index; d is a temporary variable. Step 1-2-5: If pixel points The pixel value ranges for the red, green, and blue color channels are respectively Centered on this pixel The local binary pattern characteristics of the region are ,from Surrounding distance equals Select pixels with similar color and texture from the selected pixels, change the element values ​​of matrix B corresponding to the selected pixels to True, and update the matrix B based on the x-axis and y-axis indices of the selected pixels. , , and ,in, , , The local binary pattern features are calculated based on matrix C; Step 1-2-6: If no pixel was selected in step 1-2-5, proceed to step 1-2-7; otherwise, let... Return to steps 1-2-5; Step 1-2-7: If , , and To meet the preset requirements for head size for crowd counting, If matrix B contains no element with a value of False, then matrix D is the pseudo-label map of the crowd count for the image img; otherwise, return to step 1-2-4.

4. The self-supervised population counting method according to claim 3, characterized in that, The pixels with similar colors refer to pixels whose pixel values ​​in the red, green, and blue color channels are respectively located in the following ranges: The pixels; The pixels with similar textures refer to those centered around that pixel. Local binary pattern characteristics of the region and The pixels whose correlation is greater than a preset threshold.

5. The self-supervised population counting method according to claim 3, characterized in that, Step 2 involves constructing a crowd counting model based on the improved U-Net, which includes the following sub-steps: Step 2-1: Replace the shrinkage path of U-Net with an encoder consisting of one convolutional module and four downsampling convolutional modules connected in sequence; the convolutional module outputs feature map F1, and the four downsampling convolutional modules output feature maps F2, F3, F4, and F5 in sequence; Step 2-2: Replace the expansion path of U-Net with a decoder consisting of a multi-attention upsampling feature fusion module and three multi-attention downsampling feature fusion modules connected in sequence; the multi-attention upsampling feature fusion module receives feature maps F4 and F5 and outputs feature map F6; The first multi-attention downsampling feature fusion module receives feature maps F3 and F6 and outputs feature map F7. The second multi-attention downsampling feature fusion module receives feature maps F2 and F7 and outputs feature map F8; the third multi-attention downsampling feature fusion module receives feature maps F1 and F8 and outputs feature map F9. Steps 2-3: The final U-Net... The convolution is replaced by a density map generation module. This module receives the feature map F9, outputs a crowd density map, and then performs an integral operation on the crowd density map to obtain the number of people.

6. The self-supervised population counting method according to claim 5, characterized in that, In step 3, the crowd counting model based on the improved U-Net is trained using the self-supervised pre-training dataset obtained in step 1 to obtain the pre-trained model, including the following sub-steps: Step 3-1: Set the initial parameters for model training; Step 3-2: Use during training Loss and The loss is calculated using a loss function composed of norm losses; the gradient is calculated based on the loss, and the network model parameters are updated using the Adam optimizer; Step 3-3: Evaluate the model using mean absolute error and mean squared error; Step 3-4: Repeat the training process from Step 3-2 to Step 3-3. Evaluate the network model after each training round until the preset total number of training rounds is completed. Save the best network model as the pre-trained model.

7. The self-supervised population counting method according to claim 1, characterized in that, Step 4 involves acquiring a labeled crowd image dataset and preprocessing it, then dividing the dataset into a training set, a validation set, and a test set proportionally. This includes the following sub-steps: Step 4-1: Download or create a labeled crowd image dataset from an open-source dataset website; Step 4-2: Resize the images and their labels in the dataset; Step 4-3: Divide the scaled images and labels into training, validation, and test sets; Step 4-4: Use the Gaussian kernel function to generate a real population density map from the labeled map.

8. The self-supervised population counting method according to claim 1, characterized in that, In step 5, the pre-trained model is fine-tuned using the training set and validation set obtained in step 4 to obtain the target model, including the following sub-steps: Step 5-1: Set the initial parameters for model training; Step 5-2: Use during training Loss and The loss is calculated using a loss function composed of norm losses; the gradient is calculated based on the loss, and the network model parameters are updated using the Adam optimizer; Step 5-3: Evaluate the model using the mean absolute error and mean squared error of the validation set; Step 5-4: Repeat the training process from Step 5-2 to Step 5-3. After each training round, use the validation set to evaluate the network model until the preset total number of training rounds is completed. Save the best network model as the target model.

9. The self-supervised population counting method according to claim 1, characterized in that, In step 6, the test set image obtained in step 4 is input as the crowd image to be counted into the target model obtained in step 5 to obtain the crowd counting result of the image, including the following sub-steps: Step 6-1: Import the target model saved in Step 5, input the test set images obtained in Step 4 as the crowd images to be counted into the target model, and output the final crowd density map. Step 6-2: Calculate the mean absolute error and mean square error, and save the test results.

10. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the self-supervised crowd counting method as described in any one of claims 1 to 9.

Images