A crowd counting method based on perspective self-adaption in complex scene
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANCHANG UNIV
- Filing Date
- 2023-02-15
- Publication Date
- 2026-07-24
Smart Images
Figure CN116189087B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of signal processing technology, and specifically to a crowd counting method for complex scenes based on viewpoint adaptation. Background Technology
[0002] Over a long period of development, the field of crowd counting has seen many novel inventions. Mainstream inventions use density map-based methods to estimate crowd size. Besides fully supervised learning methods, semi-supervised, weakly supervised, and transformer-based learning methods have also been proposed and have gradually become hot topics in recent years. However, existing methods do not consider the problem of perspective changes in complex natural scenes; they cannot eliminate the influence of height differences, nor can they solve the errors caused by data labeling.
[0003] Therefore, it is necessary to propose a perspective-adaptive population counting method by improving the data labeling approach. Summary of the Invention
[0004] The purpose of this invention is to provide a crowd counting method for complex scenes based on perspective adaptation, in order to address the shortcomings of the prior art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a crowd counting method for complex scenes based on viewpoint adaptation, the counting method comprising the following steps:
[0006] S1: The NOOMP framework fits the natural world through few-shot learning methods;
[0007] S2: The NOOMP framework uses meta-learning training to adapt the perspective to a multi-head parallel network as the backbone of the NOOMP framework.
[0008] S3: View-adaptive multi-head parallel network for estimating population density maps;
[0009] S4: The view-adaptive multi-head parallel network is trained in multiple different scenarios and the sub-losses are summarized.
[0010] Preferably, in step S1, the few-shot learning method includes meta-training and meta-testing, and further includes the following steps:
[0011] S1.1: Train a pre-trained model during the training phase;
[0012] S1.2: Fine-tune the model using the labeled image in the target scene.
[0013] Preferably, the meta-training is used to train the model M. pre For a series of parameters L n ={l1,l2,l3,.....,ln} To learn, each training task In, the parameter set L n The update is performed using mini-batch gradient descent. The loss function and optimization function for the n tasks are shown in equations (1) and (2):
[0014]
[0015]
[0016] In each task, the sub-loss is calculated as follows: In n tasks, the sub-losses are accumulated. For an image in a task, y (i) This represents the actual ground density map, and the estimated density map is represented as follows:
[0017] Preferably, the meta-test is used to test the pre-trained model M. pre After fine-tuning in the target scene, the loss function in the new scene is calculated as shown in equation (3):
[0018]
[0019] For the target task Scene and There are significant differences in characteristics between them.
[0020] Preferably, in step S3, estimating the population density map includes the following steps:
[0021] S3.1: Label two points for a person in the image, with variance σ. i Estimate head size;
[0022] S3.2: For passerby p n A point Located at the top of its head, another point Located at the center of its head, at a distance of d n The weighted Euclidean distance equation (5) is used to calculate:
[0023]
[0024] via distance d n To calculate the variance σ n Its formula is equation (6):
[0025] σ n =(d n +a)*λ(6)
[0026] The constant 'a' represents the distance 'd'. n The deviation is λ, which is a hyperparameter, and λ = 2.5.
[0027] Preferably, step S2 includes the following steps:
[0028] S2.1: The initial image I is sent to the view-adaptive multi-head parallel network with a size of 1024*768. I is a copy of I1, I2, and I3, and the three inputs correspond to three different parallel branches.
[0029] S2.2: I1 is input into a shallow network containing four convolutional layers, and I2 and I3 are input into a deeper network containing 7 and 10 convolutional layers, respectively;
[0030] S2.3: f h1 (), f h2 () and f h3 () represents the front-end network, expressed as equation (7):
[0031]
[0032] In the above formula, function O represents the convolutional layer of each branch, and W... n Representing different convolution operations, R represents the ReLU activation function, M represents the Max_pooling operation, and a 3*3 kernel is used.
[0033] Preferably, step S2 further includes the following steps:
[0034] S2.4: After the front branch, three feature maps are obtained. and Features contain information from different dimensions;
[0035] Information from different dimensions includes color and texture information in shallow networks and semantic information in deep networks;
[0036] S2.5: Convolutional layers using 1x1 kernels, for and Add two convolutional layers with a kernel size of 1*1;
[0037] S2.6: Enhanced features obtained using adaptive average pooling operation and
[0038] S2.7: Use the concat() operation to collect features and
[0039] Preferably, it also includes
[0040] S5: The NOOMP framework is evaluated using mean absolute error and root mean square error, expressed as equations (8) and (9):
[0041]
[0042]
[0043] N represents the total number of test images, for each test image x i , parameter y i Represents the actual ground conditions, parameters This indicates the estimation result.
[0044] The technical effects and advantages provided by the present invention in the above technical solution are as follows:
[0045] 1. In this invention, since common scenes in daily life often involve height differences, crowd counting needs to be performed under complex perspectives. Therefore, this invention proposes a novel labeling method, Absolute-geometry Gaussian Generation. This method only requires adding one point to each person in the image to achieve good accuracy;
[0046] 2. The NOOMP framework of this invention employs meta-learning and few-shot methods to train the counting model. This enables the invention to effectively achieve viewpoint adaptation while also solving the problem of excessively high labeling costs;
[0047] 3. This invention proposes a novel multi-head parallel network (MPNet) for NOOMP to extract crowd features. MPNet is a hybrid structure composed of shallow and deep networks. This network effectively preserves the features of both shallow and deep networks, enabling MPNet to achieve good performance in NOOMP.
[0048] 4. This invention collects a new dataset for NOOMP. This dataset is named the Multiple Height Differences in Mall (MHDM) dataset. It contains images from shopping malls and supermarkets. These images have different viewpoints and height differences. Experiments based on MHDM and other benchmarks show that the NOOMP framework has good accuracy and can effectively solve the problem of viewpoint changes. Attached Figure Description
[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0050] Figure 1This is a general framework diagram of the present invention.
[0051] Figure 2 This is a structural diagram of a multi-head parallel network.
[0052] Figure 3 This is a front-end network activation map and activation heatmap for crowd scenarios.
[0053] Figure 4 Example image of a shopping mall multiple height difference (MHDM) dataset.
[0054] Figure 5 This is a density map estimated by the present invention and CANNet.
[0055] Figure 6 This is a schematic diagram of the structure of MPNet(a) and MPNet(b) in this invention. Detailed Implementation
[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] Example 1
[0058] Please see Figure 1 As shown in this embodiment, a crowd counting method for complex scenes based on viewpoint adaptation is described. The counting method includes the following steps:
[0059] The NOOMP framework employs a few-shot learning approach to fit the real natural world, using a meta-learning training method to achieve the proposed viewpoint adaptation effect. A multi-head parallel network (MPNet) serves as the backbone of NOOMP, used to estimate crowd density maps. MPNet is trained in multiple different scenarios and the sub-losses are then aggregated.
[0060] 1) Adaptation based on few-shot learning perspective
[0061] The goal of few-shot learning is to train a pre-trained model during the training phase and then fine-tune the model using fewer labeled images in the target scene. This learning approach is suitable for high-cost labeling tasks. The entire few-shot learning process consists of two phases: the first phase is "meta-training," and the second phase is "meta-testing."
[0062] In the "meta-training" phase, it will be compared with the trained model M. pre For a series of parameters Ln =l1,l2,l3,.....,l n The convolutional neural network learns parameters that reflect the feature mapping of the population. After training with MPNet, meta-optimization optimizes the loss function. This optimization is performed on each training task. (These tasks have different perspectives) In the parameter set L n All updates are performed using mini-batch gradient descent. The loss functions and optimization functions for the n tasks are shown in equations (1) and (2).
[0063]
[0064]
[0065]
[0066] In each task, the sub-loss is calculated as follows: Furthermore, these sub-losses will be accumulated across n tasks. For an image in a task, y (i) This represents the actual ground density map. The estimated density map is represented as follows: In the training process of a few-shot learning model, there are many loops corresponding to different tasks. The model's parameters are continuously updated until the network converges.
[0067] In the "meta-test" phase, the pre-trained model M will be... pre Fine-tune it within the target scenario. For the target task. Scene and There are significant feature gaps between them. Through further training, the pre-trained model M... pre It can perform well in the target scenario. The loss function calculation in the new scenario is shown in equation (3).
[0068]
[0069] After the "meta-testing" phase, the crowd counting model achieves viewpoint adaptation. The pre-trained model M is then obtained. pre After that, the model can easily adapt to new scenes from different perspectives.
[0070] 2) Density map of absolute geometric Gaussian kernel
[0071] Density map-based crowd counting plays a crucial role in current crowd counting research. Since training a CNN (Convolutional Neural Network) is necessary to estimate the crowd density map from the input image, the quality of a given density in the training data significantly impacts the performance of the counting method, MCNN (Multi-column Convolutional Neural Network). Wan et al. found that better ground truth density maps lead to better training results, and different density maps have varying effects. They also discovered that using different density maps as ground truth for training counters can produce different performance results.
[0072] Existing methods attempt to solve crowd counting problems using parallel perspective. For example, consider crowds on a ground surface or a plane. In such scenarios, the perspective follows a pattern: objects appear larger when closer and smaller when farther away. Since the distances around each person are uniformly distributed, two main methods are used to calculate crowd counts based on the ground perspective: adaptive geometric functions and fixed geometric functions.
[0073] Traditional density map generation methods: Given an image, if a person's head is located at pixel x i In the middle, the distance to its k-th nearest neighbor pixel is These distances can be calculated using the pixels of the head location. The average distance formula is shown in (4):
[0074]
[0075] Absolute geometric Gaussian density map generation methods: Adaptive geometric functions seem to work well. However, natural landscapes always have height differences, such as stairs, towers, and tall buildings. These height differences can cause significant differences in height between adjacent people in an image. Imagine two people in a shopping mall, one downstairs and the other upstairs. Due to perspective issues, these two people can appear very close in size in a single photograph. However, in reality, the person upstairs (assuming the image was taken from upstairs) is larger than the person downstairs.
[0076] Therefore, to address the shortcomings of adaptive and fixed geometric functions, this invention proposes an absolute geometric Gaussian function to generate density maps for crowd counting as ground reality. The key to this method is labeling two points for each person in the image. The variance σ... i It was used to estimate the size of people's heads. Because existing datasets lack this information, it's difficult to know the true head size in different viewpoints. This parameter becomes crucial as scenes become more complex. Therefore, this work directly labels the absolute geometric distance of each person. For a passerby p... n A point Located at the top of its head, another point Located at the center of its head. Distance d n The weighted Euclidean distance equation (5) can be used to calculate it.
[0077]
[0078] via distance d n To calculate the variance σ n Its formula is equation (6).
[0079] σ n =(d n +a)*λ (6)
[0080] This constant is represented by the distance d. n The deviation is used to fine-tune d. n Furthermore, to address the issue of scale inconsistency across different scenarios, a hyperparameter λ is introduced. The hyperparameter λ can avoid the problem of σ... n The value is too small. In experiments, this invention found that, empirically speaking, λ = 2.5 yields the best results. Therefore, the density plot is formed by a variance of σ. n It is generated by an absolute geometric Gaussian kernel, and its variance is determined by two points on the human head.
[0081] Example 2
[0082] In this embodiment, the analysis mainly focuses on the training set and MPNet details:
[0083] 1) The proposed NOOMP training set
[0084] The pre-trained model M trained in the "meta-training" phase pre Fine-tuning can be performed on target scenes from different perspectives. "Meta-training" is based on a new scene θ. n Yes. This new scene is not included in the training set. This stage further refines model M. pre A new model M is obtained through training. tar Model M tar New scenario θ n The model M has a higher content than the model M pre Better performance. In the "meta-training" and "meta-testing" phases, training is performed using only a small number of labeled images from the scene. For example, scene θ. n Of the labeled images M, only labeled images K << M are used to train the model. In existing experiments, K ranges from 1 to 5, typically 1 and 5, corresponding to "One-shot" and "Five-shot" respectively. Due to the proposed image labeling method, each person in the image requires two points to represent their position and size. This increases the cost of labeling. Therefore, it is appropriate for this invention to employ a few-shot learning approach to train the counting model. Only a few images in the target scene require labels, which is acceptable.
[0085] During the training and testing phases, the dataset D is divided into two parts—the "meta-training" D. train And the D of "meta-test" test In the first phase, a dataset containing k tasks was used. Model M is trained in two parts. pre of and testing (k1+k2=k). D train-test This allows us to test the accuracy and generalization ability of the learned model. In the second stage, a dataset containing c tasks is used. It can also be divided into two parts --- training model M tar of and for testing D test-train Used in the target scene θ n Model M tar Make minor adjustments.
[0086] 2) The proposed MPNet structure
[0087] The proposed architecture of the Multi-Head Parallel Network (MPNet) is as follows: Figure 2 As shown, MPNet is designed for learning frameworks with few shots. Compared to existing crowd counting models such as CANNet (environmentally sensitive crowd counting) or CSRNet (crowded scene recognition network, which introduces dilated convolutional kernels into the crowd counting network to expand the receptive field and achieve better accuracy), MPNet's structure is not complex. It mainly consists of shallow and deep networks and a large number of convolutional layers.
[0088] The initial image I is sent to MPNet, with a size of 1024*768. I is a copy of I1, I2, and I3. These three inputs correspond to three different parallel branches. I1 is input into a shallow network containing four convolutional layers. I2 and I3 are input into a deeper network containing 7 and 10 convolutional layers, respectively. This method refers to the front-end network as the "head," and MPNet has a multi-head structure. h1 (), f h2 () and f h3 () represents the aforementioned front-end network, which can be represented by equation (7).
[0089]
[0090] In equation (7), the function O() represents the convolutional layer of each branch. n These represent different convolution operations. R represents the ReLU activation function. M represents the Max_pooling operation. A 3x3 kernel is used in these processes.
[0091] After the front-end branch, three feature maps can be obtained. and These features contain information across different dimensions, such as color and texture information in shallow networks and semantic information in deep networks. Existing methods primarily utilize the VGG16 model to extract raw features from the crowd and employ other feature enhancement methods to obtain better feature representations. Deep networks exhibit good representational capabilities in image classification and detection. However, the goal of few-shot learning is to first train a pre-trained model. Research has found that if the model is designed to be too deep, the pre-trained parameters become too fixed in the upstream scene, resulting in poor performance when fine-tuning in the downstream scene.
[0092] Figure 3 The images show the activation maps and heatmaps of the front-end network in a crowd scene. It can be observed that the shallow network exhibits shape features of the crowd and surrounding buildings, represented by the black portion of line 1. Higher-density networks constitute the activated portions (the red portions in rows 2 and 3), representing the semantic information of the crowd. As the network deepens, the features of the shallow network gradually disappear during training. This loss of texture information is particularly severe with the two-stage training method. Therefore, compared to existing counting networks, this study designs MPNet as a relatively shallow network structure. Experimental results also demonstrate the effectiveness of this structure.
[0093] To obtain the final density map, features will be further processed. and because The layer contains only 4 convolutional layers, so another convolutional layer with a 1x1 kernel is used. For and Two convolutional layers with a kernel size of 1x1 were then added. Adaptive average pooling was then used to obtain enhanced features. and Finally, use the concat() operation to collect features. and This operation combines features from both shallow and deep layers.
[0094] 3) Layer details of MPNet
[0095] This section showcases the layer details of MPNet, demonstrating the structure of convolutions.
[0096] THE LAYER STRUCTURE OF THE PROPOSED MPNET
[0097]
[0098] Table 1
[0099] Table 1 shows the elements of MPNet. MPNet contains 13 convolutional layers. The final convolutional layer is fed into an adaptive average pooling layer and an upsampling layer to obtain a density map of the same size as the input image.
[0100] Example 3
[0101] The NOOMP framework for view-adaptive crowd counting proposed in this invention was implemented in a Linux Ubuntu 16.04, Python 3.6.13, and PyTorch 1.7.0 software environment. The hardware environment consisted of an Intel Core i9-10900K 3.70GHz processor, 64GB ROM, and a Tesla V100 32GB CPU.
[0102] 1) Evaluation criteria
[0103] To estimate the accuracy of counting models, two main evaluation metrics are used: mean absolute error (MAE) and root mean square error (RMSE). These two metrics are widely used in many tasks and can be defined by equations (8) and (9).
[0104]
[0105]
[0106] N represents the total number of test images. For test image x i , parameter y i Represents the actual ground conditions, parameters This indicates the estimation result.
[0107] 2) Evaluate the dataset
[0108] To test the performance of the proposed viewpoint-adaptive counting method, this application collected a new dataset, the Multiple Height Difference in Mall (MHDM) dataset. This dataset contains 523 images of 13 different viewpoint scenes selected from the DISCO (audiovisual crowd counting) dataset. These images were taken in a large shopping mall, where significant perspective challenges exist.
[0109] Figure 4 Some examples of MHDM are shown. To expand the scale of MHDM and take into account the richness of perspective, this work rotates the original image 180 degrees along the x-axis. For example... Figure 4As shown, the rotated image is a mirror image of the original image. This operation expands the original dataset to 1046 images, containing scenes from 26 different viewpoints. Furthermore, this study uses the proposed absolute geometric Gaussian kernel method to label these images. Two points are used to represent the position and size of the person. Simultaneously, this work also retains a pint-sized labeling method to generate an adaptive geometry method for experimental comparison. The high-density model with absolute geometric labeling is denoted as "MHDM-ABS", and the high-density model with adaptive geometric labeling is denoted as "MHDM-ADA". In addition, this study randomly selected 200 images from "MHDM-ABS" and "MHDM-ADA", denoted as "MHDM-ABS-small" and "MHDM-ADA-small", respectively. A smaller, randomly selected dataset was also used to test the models.
[0110] To test the performance of the NOOMP framework, two other benchmarks were also tested: ShanghaiTech and WorldExpo'10 (two datasets). ShanghaiTech contains many different scene images, but most were collected using parallel perspective. WorldExpo'10 contains 103 different scenes and was used to test scene adaptation performance in practice. Because WorldExpo'10 contains rich perspective information, this dataset was also used to test the performance of the NOOMP framework.
[0111] 3) Performance of the proposed absolute geometry kernel
[0112] To demonstrate the effectiveness of the proposed absolute geometric Gaussian generation method, this work tested CANNet ((Environmentally Sensitive Population Counting)) on four different MHDM datasets, namely "MHDM-ADA-small", "MHDM-ABS-small", "MHDM-ADA", and "MHDM-ABS".
[0113] THE PERFORMANCES OF CANNET ON FOUR DIFFERENTMHDM DATASETS
[0114]
[0115] Table 2
[0116] Table 2 presents the experimental results. It can be seen from the table that CANNet performs best on the "MHDM-ABS" dataset. On the test set, the MAE value is 4.46 and the RMSE value is 6.83. CANNet also outperforms "MHDM-ADA-small" on "MHDM-ABS-small". Although the training images are the same, different ground conditions have different effects. Experiments were also conducted using a fixed geometric kernel method with different kernel sizes. Different labeling methods do indeed affect the accuracy of the counting model. The experiments further demonstrate that the proposed absolute geometric kernel is suitable for scenes with complex viewpoint changes.
[0117] THE PERFORMANCES OF CANNET ON MHDM-ABS WITH DIFFERENT VALUE OF λ
[0118]
[0119] Table 3
[0120] This invention also considers the influence of different values of λ in the absolute geometric kernel method formula (6). Table 3 shows the parameters. For fairness, this work also uses the public network CANNet to test the MHDM-ABS dataset. As can be seen from the table, the CANNet model performs best when λ = 2.5. As the value of λ increases or decreases, the accuracy of the model follows a normal distribution. Therefore, in the process of generating ground reality maps, this paper uses an absolute geometric kernel Gaussian function with λ = 2.5.
[0121] 4) Model performance under MHDM and multiple benchmarks
[0122] To test the model's viewpoint adaptability, this experiment also tested the NOOMP framework on the MHDM dataset ("MHDM-ABS") with an absolute geometric kernel. MHDM-ABS contains 26 scenes with different viewpoints. This paper selected 23 scenes for model training and 3 scenes for model testing. To compare the training results, CSRNet (Crowded Scene Recognition Network) and CANNet were selected as the other two backbones. The NOOMP framework with CSRNet is denoted as "CSRNet+", and the NOOMP framework with CANNet is denoted as "CANNet+". In the training step, two sets of parameters were set, namely K=1 (one-shot) and K=5 (five-shot), that is, one or five images were used for training in one scene.
[0123] THE "META" MODEL'SPERFORMANCES ON MHDM-ABS DATASET
[0124]
[0125] Table 4
[0126] THE PERFORMANCES OF SOTASON SHANGHAITECH AND MHDM DATASETS
[0127]
[0128] Table 5
[0129] The experimental results are shown in Table 4. The table shows that the proposed NOOMP framework performs well in both single and five training iterations. In single training, the mean MAE and RMSE scores for NOOMP are 20.58 and 23.27, respectively. For "CSRNet+", the MAE and RMSE scores in single training are 48.00 and 55.11, respectively. For "CANNet+", the MAE and RMSE scores in single training are 37.59 and 41.47, respectively. Therefore, NOOMP achieves excellent results in viewpoint-adaptive crowd counting for few-shot images. It also demonstrates that the proposed MPNet is a shallow network with good performance under a few-shot learning framework. After careful analysis, we can easily conclude that shallow networks have stronger trainability in downstream tasks. When training MPNet in upstream scenes, the NOOMP base network still exhibits strong learning capabilities. Compared to more complex network structures such as CSRNet and CANNet (which employ deep extended convolutional kernels and context-aware structures to extract crowd features), shallow networks have better two-stage learning capabilities.
[0130] To demonstrate the above analysis, this work also tested MPNet against other state-of-the-art (SOTA) models (CSRNet, PSDDN (apoint-supervised deep detection network), SPN (Scale Preserving Network), DENet (a network consisting of a detection network (DNet) and an encoder-decoder estimation network (ENet), LSC-CNN (dense face detection framework), and CANNet (environmentally sensitive crowd counting)). Testing was performed on the ShanghaiTechpart_A, ShanghaiTechpart_B, and MHDM-ABS datasets. Due to the high cost of absolute geometric labeling methods, we used the original ShanghaiTech dataset.
[0131] Table 5 shows the performance of the test results. Compared with CSRNet, DENet, PSDDN, and LSC-CNN, MPNet achieves better MAE accuracy. However, it can also be observed that CANNet outperforms MPNet on the ShanghaiTech dataset. It is important to note that the MHDM-ABS dataset is directly divided into two parts: 700 images for training and 346 images for testing. The training method is fully supervised, without using the NOOMP framework. In this case, CANNet outperforms MPNet. CANNet's MAE value is 4.7, while MPNet's MAE value is 4.9. The experimental results seem to contradict those in Table 4. Instead, the results demonstrate that the proposed shallow network MPNet is suitable for few-shot learning of perspective adaptation tasks using a small number of labeled images. The advantage of NOOMP is that it can solve the adaptation problem using weakly supervised strategies. On the other hand, although the proposed MPNet structure is not complex, its results on the ShanghaiTech public dataset are close to the state-of-the-art (SOTA) results and even outperform some counters.
[0132] THE DIFFERENT MODEL'S PERFORMANCES ON WORLOECPO'S 10DATASET.
[0133]
[0134] Table 6
[0135] In addition to our self-collected MHDM dataset, we also tested the proposed NOOMP framework on the public benchmark WorldExpo'10. Due to the high cost of ground truth labeling, we only used the original dataset to test the model. WorldExpo'10 contains 103 different scenes; 98 scenes were used for training the model, and 5 scenes were used for testing. These scenes also have different viewpoints, suitable for viewpoint adaptability testing. Table 6 shows the performance of NOOMP, "CSRNet+", and "CANNet+" in five different scenes. On average, the NOOMP framework achieved an MAE of 5.54 and an RMSE of 6.72 on the single-sample dataset. The test results are superior to those of "CSRNet+" and "CANNet+", demonstrating the superiority of the NOOMP framework.
[0136] This experiment also enabled the visualization of density map results from the NOOMP framework. Figure 5These are the density maps estimated by us and CANNet. The first column shows the selected examples, the second column shows the actual ground conditions, the third column shows the CANNet results, and the fourth column shows our results. In each density map, the total population is given in the lower right corner of the image. Visualization shows that the values generated by NOOMP are closer to the actual ground conditions. This further demonstrates that the proposed NOOMP method outperforms other methods.
[0137] 5) Ablation test
[0138] To explore suitable structures for the MPNet proposed in this invention, two other schemes were explored. An MPNet without shallow branches and an MPNet without deep branches are compared. The former is denoted as "MPNet(a)" and the latter as "MPNet(b)".
[0139] Figure 6 Two architectures are clearly shown. "MPNet(a)" removes the deeper network, and "MPNet(b)" removes the shallower network. This is done to understand how the proposed architecture affects the accuracy of the viewpoint adaptation model. In "MPNet(a)", this feature does not contain high-level semantic information. In "MPNet(b)", this feature does not contain shallow texture information.
[0140] THE COMPARISONS BETWEEN DIFFERENT MPNETS
[0141]
[0142] Table 7
[0143] Table 7 shows that MPNet achieves the best accuracy results compared to "MPNet(a)" and "MPNet(b)". Experiments demonstrate that information extracted from both shallow and deep networks can be used to capture crowd features.
[0144] Therefore, both shallow and high-level information are essential in the NOOMP framework.
[0145] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.
[0146] It should be understood that the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this application generally indicates an "or" relationship between the preceding and following related objects, but it may also indicate an "and / or" relationship. Please refer to the context for a more accurate understanding.
[0147] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0148] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0151] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0153] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0154] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A crowd counting method for complex scenes based on viewpoint adaptation, characterized in that: The counting method includes the following steps: S1: The NOOMP framework fits the natural world through a few-shot learning method; the few-shot learning method includes meta-training and meta-testing. S2: The NOOMP framework uses meta-learning to train a multi-head parallel network as its backbone, adapting the viewpoint to the training environment. This meta-training is used to train the pre-trained model... For a series of parameters To learn, each training task In, parameter set The update is performed using mini-batch gradient descent. The loss function and optimization function for the n tasks are shown in equations (1) and (2): (1) (2) In each task, the sub-loss is calculated as follows: In n tasks, the sub-losses are accumulated. For an image in a task, This represents the actual ground density map, and the estimated density map is represented as follows: ; This represents the learning rate in gradient descent. The meta-test is used to evaluate the pre-trained model. After fine-tuning in the target scene, the loss function in the new scene is calculated as shown in equation (3): (3) For the target task Scene and There are significant differences in characteristics between them; S3: A viewpoint-adaptive multi-head parallel network is used to estimate the density map of a population; the estimation of the population density map includes the following steps: S3.1: Label two points for a person in the image, variance Estimate head size; S3.2: For passersby A point Located at the top of its head, another point Located at the center of its head, at a distance The weighted Euclidean distance equation (5) is used to calculate: (5) By distance To calculate variance Its formula is equation (6): (6) constant Represented as distance deviation, It is a hyperparameter, and ; S4: The view-adaptive multi-head parallel network is trained in multiple different scenarios and the sub-losses are summarized.
2. The crowd counting method in complex scenes based on viewpoint adaptation according to claim 1, characterized in that: Step S1 also includes the following steps: S1.1: Train a pre-trained model during the training phase; S1.2: Fine-tune the model using the labeled image in the target scene.
3. The crowd counting method in complex scenes based on viewpoint adaptation according to claim 1, characterized in that: Step S2 includes the following steps: S2.1: Initial image of size 1024*768 Send to the view-adaptive multi-head parallel network, and... Copy as , ,and These three inputs correspond to three different parallel branches; S2.2: , and The inputs are fed into a shallow network with four convolutional layers, a network with seven convolutional layers, and a deeper network with ten convolutional layers, respectively. S2.3: , and The front-end network is represented by equation (7): (7) In the above formula, the function This represents the convolutional layer of each branch. , and To represent different convolution operations, express The activation function, M represents the Max_pooling operation, uses a 3*3 kernel.
4. The crowd counting method in complex scenes based on viewpoint adaptation according to claim 3, characterized in that: Step S2 also includes the following steps: S2.4: After the front branch, three feature maps are obtained. , and Features contain information in different dimensions; Information from different dimensions includes color and texture information in shallow networks, as well as semantic information in deep networks; S2.5: For Adding a 1x1 kernel convolutional layer for and Add two 1x1 kernel convolutional layers to each; S2.6: Enhanced features obtained using adaptive average pooling operation , and ; S2.7: Use the concat() operation to collect enhanced features , and Here, concat() represents the concatenation operation.
5. A crowd counting method for complex scenes based on viewpoint adaptation according to any one of claims 1-4, characterized in that: Also includes: S5: The NOOMP framework is evaluated using mean absolute error and root mean square error, expressed as equations (8) and (9): (8) (9) N represents the total number of test images, parameter Represents the actual ground conditions, parameters This indicates the estimation result.