Cross-domain crowd counting method and system based on fine-grained feature similarity retrieval
Through the fine-grained feature similarity search method, the fine-grained features of the source domain and the target domain are extracted and matched, pseudo-labels are generated and model fine-tuned, which solves the problem of low accuracy in the migration of the population counting model between different domains, and realizes the Statistics on high accuracy rate people in real surveillance scenarios.
Patent Information
- Application Number
- CN202210555319.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-05-19
AI Technical Summary
The existing population counting model is not effective in migration between different domains, especially in dense scenarios. Direct model migration cannot effectively solve inter-domain differences, resulting in low counting accuracy.
By using the fine-grained feature similarity search method, the fine-grained feature elements of the source domain and the target domain are extracted, the database is constructed, the similarity is calculated, the pseudo-label generation and model fine-tuning is performed, and the population count prediction is performed in combination with the sliding window strategy to reduce the domain spacing. , improve migration accuracy.
It realizes high-accuracy people statistics in real monitoring scenarios, avoids labeling of monitoring images, improves the accuracy and speed of model migration, retains source domain information, and significantly improves the performance of cross-domain population counting.
Smart Images

Figure CN114842422B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain crowd counting method and system combining fine-grained feature similarity retrieval. Background Art
[0002] Crowd counting, as a topic in computer vision, has received extensive attention from academia and industry. Formally, crowd counting is a technique for determining the total number of individuals in a surveillance image or video. It is often used in crowd behavior analysis and public safety management to detect crowding or abnormal behavior in real time and prevent accidental injuries. Although some methods trained and tested in the same domain have achieved good results, they are not applicable in actual surveillance scenarios, especially in dense scenes. Therefore, studying the domain transfer problem of crowd counting is crucial for the practical application of crowd counting. However, due to the differences between different domains, direct model transfer cannot achieve the expected results.
[0003] In order to perform model migration, existing crowd counting methods find the differences between the entire images in different domains and migrate the model by narrowing the differences. However, they do not consider cutting the images in different domains into small pieces for similarity comparison. By using this fine-grained similarity to migrate the model, the accuracy of model migration is higher. Summary of the invention
[0004] The purpose of the present invention is to reduce the domain distance between the target domain and the source domain, and proposes a cross-domain crowd counting method combined with feature fine-grained retrieval, aiming to use the fine-grained similarity between different domains to perform cross-domain crowd counting.
[0005] To achieve the above purpose, the present invention designs a cross-domain crowd counting method combining fine-grained feature similarity retrieval, which is special in that the method comprises the steps of:
[0006] S1 extracts fine-grained feature elements: collects fine-grained crowd distribution for source domain and target domain images respectively, and uses a sliding window strategy to obtain fine-grained blocks to extract fine-grained feature elements for the source domain and target domain. The source domain refers to a deep learning image dataset with crowd counting labels, and the target domain refers to a real-world scene image dataset to be detected.
[0007] S2: Find the relationship between domains: Build a database of fine-grained feature elements of the source domain and the target domain, calculate the similarity between each element in the source domain and the elements in the database, and retrieve the source domain elements that contain the target domain elements and have the highest similarity with them according to the given fine-grained features of the target domain;
[0008] S3 Crowd Count Prediction: Perform density map prediction on the target domain fine-grained block, fuse the predicted value with the matching value of the source domain fine-grained block to obtain a pseudo label, normalize the long and short sides of the target domain image to multiples of 128, and then sample the image using a sliding window. The final counting result of the entire image is the sum of the counting results of all small blocks cut from the entire image.
[0009] Preferably, step S1 comprises the following steps:
[0010] S11 Fine-grained crowd distribution collection: A sliding window with a size of (128, 128) and a step size of (64, 64) is used to slide across the entire image to obtain fine-grained blocks. The set of fine-grained blocks is defined as:
[0011] S12 Fine-grained feature extraction: Use the position-sensitive backbone network HRNet and use the labeled blocks in the source domain to initialize the feature extraction model, and define the feature extraction model as M r , r represents the number of iterations, and the initialization model is recorded as M 0 ; The source domain features and target domain features are defined as: in and Represent the i-th source domain feature and the j-th target domain feature respectively;
[0012] S13. The feature extraction model is optimized using an independent structure similarity loss function and a mean square error loss, wherein the loss function is as shown in formula (4), wherein the independent structure similarity loss is as shown in formula (1), and L S As shown in formula (2), the structural loss is shown in formula (3):
[0013]
[0014] L S =1-SSIM(E,GT) (2)
[0015]
[0016] L=L MSE +ηL I-S (4)
[0017] N is the total number of people, E is the predicted image, GT is the ground-truth, μ A and σ B are the mean value of E and the standard deviation of GT, σ A·B represents the covariance of E and GT, γ1 and γ2 are set to 0.0001 and 0.0009.
[0018] Preferably, step S2 comprises the following steps:
[0019] S21 diffuses all fine-grained feature elements in the source domain and the target domain to form a database by finding a high similarity method, and finds similar blocks in the database for the fine-grained features in the target domain to obtain the final feature sequence
[0020] S22 Selection The rank1 in Its similarity score is recorded as Set the threshold λ to filter The characteristic pair of
[0021] Preferably, step S21 includes the following steps:
[0022] Construction of S211 graph: All features of the source domain and the target domain are taken as a set and defined as: Using local constraints, the graph only contains pairwise similarities and defines the affinity matrix: A = a ij ;
[0023] S212 Graph normalization: normalize the affinity matrix to S. The normalization process is shown in formula (5):
[0024]
[0025] D=∑a ij (6)
[0026] S213 divides the normalized affinity matrix S into four blocks, as shown in formula (7).
[0027]
[0028] in, S mm S md S dm S dd Respectively represent the similarity between source domain to source domain, source domain to target domain, target domain to source domain, and target domain to target domain, where d = m + n;
[0029] S214 Random Walk: Perform a random walk until it converges to a closed solution. The state of each iteration in the random walk is defined as: Each iteration strategy is shown in formula (8):
[0030] f t+1 =αSf t +(1-α)f 0 ,α∈(0,1) (8)
[0031] Among them, α is the probability of random walk to the next step, and (1-α) is the probability of restarting the walk from the initial state;
[0032] S215 obtains a closed solution: The closed solution f is obtained by random walk in step S214 * As shown in formula (9), the similarity between each element in the source domain and the element in the database is included and used as the re-ranking score, as shown in formula (10):
[0033] f * =(1-α)(1-αS)f 0 (9)
[0034]
[0035] S216 Feature Retrieval: Given the fine-grained features of the target domain, the final retrieval results are obtained through the KNN method. The retrieval results contain the target domain elements with the highest similarity to the source domain elements.
[0036] Preferably, step S3 comprises the following steps:
[0037] S31 Pseudo-label generation: For the fine-grained blocks of the target domain, density map prediction is performed, denoted as For the fine-grained blocks in the source domain, ground-truth matching is performed, denoted as And and Fusion and processing are performed to obtain the final pseudo label φ ij ;
[0038] S32 model fine-tuning: fine-grained blocks of the target domain and generate pseudo labels l ij Composition block data set P r , fine-tune the model and iterate until the model converges;
[0039] S33 From fine-grained to coarse-grained: The long and short sides of the images in the target domain test set are normalized to multiples of 128, and then the images and ground-truth are sampled using a sliding window of size (128, 128) and step size (128, 128). The final count result of the entire image is the sum of the count results of all the small blocks cut from the image.
[0040] Preferably, step S31 includes the following steps:
[0041] S311 Target domain density map prediction: The FIDT method is used to predict the target domain density map. The process is shown in formula (11):
[0042]
[0043] S312 generates pseudo labels: and Fusion is performed as shown in formula (12), where W is the fusion weight. The fusion process is shown in formula (12), and the pseudo-label local maximum amplification is performed as shown in formula (13):
[0044]
[0045]
[0046] The present invention also proposes a cross-domain crowd counting system with fine-grained feature similarity retrieval, which is used to implement the above-mentioned cross-domain crowd counting method with fine-grained feature similarity retrieval. The special feature of the system is that it includes a fine-grained distribution perception module, a distribution retrieval module and a domain adaptation module: wherein,
[0047] The fine-grained distribution perception module is used to collect the fine-density crowd distribution of source domain and target domain images respectively, obtain fine-density blocks using a sliding window strategy, and extract fine-grained feature elements of the source domain and target domain. The source domain refers to a deep learning image dataset with crowd counting labels, and the target domain refers to a real-world scene image dataset to be detected.
[0048] The distribution alignment retrieval module is used to find the domain relationship between the source domain and the target domain, perform feature retrieval between different domains, and select the fine-grained block with the highest similarity ranking in the retrieval results from the target domain to the source domain;
[0049] The domain adaptation module predicts a density map of a fine-grained block in the target domain, fuses the predicted value with the matching value of the fine-grained block in the source domain to obtain a pseudo label, normalizes the long and short sides of the target domain image to multiples of 128, samples the image using a sliding window, obtains a count result for each small block, and then adds the count results of the small blocks to obtain a count result for the entire image.
[0050] Furthermore, the fine-grained distribution perception module uses a position-sensitive backbone network HRNet, and uses labeled blocks in the source domain to initialize the feature extraction model, and uses the I-SSIM independent structural similarity loss function and the mean square error loss function for optimization.
[0051] Furthermore, after the domain adaptation module generates pseudo labels, it converts the target domain fine-grained blocks and generate pseudo labels l ij Composition block data set P r , fine-tune the model used to extract block features and iterate until the model converges.
[0052] Based on the above method, the present invention further proposes a computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.
[0053] The beneficial effects of the present invention are:
[0054] 1. The present invention considers migrating the deep learning crowd counting model to the real monitoring scene to achieve the population counting in the monitoring scene. Compared with the existing crowd counting method, it does not require labeling of crowd images.
[0055] 2. The present invention searches for labeled data and unlabeled data in real scenarios for fine-grained similarity comparison, and uses the parts of the labeled data that are similar to the unlabeled data to fine-tune the model. This is superior to the existing model migration method that only roughly narrows the gap between labeled and unlabeled coarse-grained data, and achieves a higher accuracy.
[0056] 3. The present invention uses a method of fine-tuning the last layer of the model to update the model, which has the advantages of 1) retaining the information of the source domain; 2) the fine-tuning method is faster. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 This is a structural block diagram of a cross-domain crowd counting method combined with feature fine-grained retrieval proposed by the present invention.
[0058] Figure 2 Schematic diagram of coarse-grained dissimilarity and fine-grained feature-level similarity in different domains.
[0059] Figure 3 This is a framework diagram of a cross-domain crowd counting method combined with feature fine-grained retrieval proposed in the present invention.
[0060] Figure 4 This is a pseudo-label example image. From left to right, each column is the original image, ground-truth, predicted image, source domain similarity image, and pseudo-label image.
[0061] Figure 5 Visualize the prediction graph at different iteration numbers.
[0062] Figure 6 Cross-domain evaluation curves for different thresholds and sliding window sizes.
[0063] Figure 7 This is a comparison chart of the accuracy of the present invention and previous methods. DETAILED DESCRIPTION
[0064] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. The basic process of the method is as follows: Figure 1 shown.
[0065] S1. Fine-grained distribution-aware: Use a sliding window strategy to obtain fine-grained blocks and extract features.
[0066] (1) Fine-grained crowd distribution collection: A sliding window with a size of (128, 128) and a step size of (64, 64) is used to slide across the entire image to obtain fine-grained blocks. The set of fine-grained blocks is defined as: Comparison of sliding windows of different sizes Figure 5 Shown right.
[0067] (2) Fine-grained feature extraction: We use the position-sensitive backbone network HRNet and use the source domain supervised block to initialize the model. The model is defined as M r , r represents the number of iterations, and the initialization model is recorded as M 0 The source domain features and target domain features are defined as: and They represent the i-th source domain feature and the j-th target domain feature respectively.
[0068] (3) The feature extraction model is optimized using the Independent Structural Similarity (I-SSIM) loss function and the Mean Squared Error (MSE) loss function. The loss function is shown in formula (4), where the independent structural similarity loss is shown in formula (1), and L S As shown in formula (2), the structural loss is shown in formula (3).
[0069]
[0070] L S =1-SSIM(E,GT) (2)
[0071]
[0072] L=L MSE +ηL I-S (4)
[0073] N is the total number of people, E is the predicted image, and GT is the ground-truth. A and σ B are the mean value of A and the standard deviation of B, σ A·B represents the covariance of A and B, and γ1 and γ2 are set to 0.0001 and 0.0009.
[0074] S2, finding inter-domain relations through distribution alignment retrieval module: Figure 2As shown in Figure 1, the diffusion method is used to perform feature retrieval between different domains, select rank1 from the target domain to the source domain retrieval result, and set the threshold λ to filter blocks with high similarity from rank1.
[0075] (1) Diffusion Model feature retrieval: First, all fine-grained feature elements in the source domain and the target domain are diffused to form a database by finding a high similarity method. Then, the fine-grained features in the target domain search for similar blocks in the database to obtain the final feature sequence.
[0076] Graph construction: All features of the source domain and the target domain are taken as a set and defined as: Using local constraints, the graph only contains pairwise similarities, and the affinity matrix is defined as: A = a ij .
[0077] Graph normalization: Normalize the affinity matrix to S. The normalization process is shown in formula (5).
[0078]
[0079] D=∑a ij (6)
[0080] Divide S into four blocks, as shown in formula (7).
[0081]
[0082] in, S mm S md S dm S dd They represent the similarity from source domain to source domain, source domain to target domain, target domain to source domain, and target domain to target domain, respectively, where d=m+n.
[0083] Random walk: According to the PerronFrobenius theorem, a random walk is performed until it converges to a closed solution. The state of each iteration in the random walk is defined as: The strategy for each iteration is shown in formula (8).
[0084] f t+1 =αSf t +(1-α)f 0 ,α∈(0,1) (8)
[0085] Among them, α is the probability of random walk to the next step, and (1-α) is the probability of restarting the walk from the initial state.
[0086] Get the closed solution: The closed solution f is obtained by the S214 random walk* As shown in formula (9), the similarity between each element in the source domain and the element in the database is included and used as the re-ranking score, as shown in formula (10).
[0087] f * =(1-α)(1-αS)f 0 (9)
[0088]
[0089] Feature retrieval: Given the fine-grained features of the target domain, the final retrieval results are obtained through the KNN method. The retrieval results include the target domain elements and the source domain elements with high similarity.
[0090] (2) Screening distribution: Select The rank1 in Its similarity score is recorded as Set the threshold λ to filter The characteristic pair of The result comparison curves of different thresholds λ are as follows Figure 6 Shown on the left.
[0091] Table 1 Comparison of counting performance without using retrieval method and using different retrieval methods
[0092]
[0093] As shown in Table 1, the method using diffusion retrieval achieves the highest counting performance in both sets of cross-domain experiments.
[0094] S3. Through the domain adaptation module, the source domain information is gradually transferred to the target domain: the fine-grained source domain information found by the distribution alignment module is integrated with the target domain information to fine-tune the model.
[0095] (1) Pseudo-label generation: For fine-grained blocks in the target domain, density map prediction is performed, denoted as For the fine-grained blocks in the source domain, ground-truth matching is performed, denoted as And and Fusion and processing are performed to obtain the final pseudo label φ ij , the specific steps are as follows.
[0096] Target domain density map prediction: The FIDT method is used to predict the target domain density map, and the process is shown in formula (11).
[0097]
[0098] Pseudo label: and The fusion is performed as shown in formula (12), where W is the fusion weight, which is set to 0.5. The fusion process is shown in formula (12), and the pseudo-label local maximum amplification is performed as shown in formula (13).
[0099]
[0100]
[0101] The final pseudo-label is Figure 3 shown.
[0102] (2) Model fine-tuning: fine-grained blocks of the target domain and generate pseudo labels l ij Composition block data set P r , fine-tune the model. S32 is an iterative process, which is repeated until the model converges. The generation of the prediction graph in different iterations changes as follows: Figure 4 shown.
[0103] (3) From fine-grained to coarse-grained: The long and short sides of the images in the target domain test set are normalized to multiples of 128, and then the images and ground-truth are sampled using a sliding window of size (128, 128) and step size (128, 128). The final count prediction result is the sum of the number of people predicted in all small blocks.
[0104] The effects of the method of the present invention and the Cycle GAN method are compared. Figure 7 As shown in the figure, Cycle GAN converts the synthetic scene into a realistic photo scene, and then trains a crowd counting network on the converted image. Finally, the crowd counting network is applied to the actual data set. The mean error (MAE) and mean square error (MSE) of the counting results of the two methods are calculated to measure the counting error. MAE represents the accuracy of the prediction result, and MSE measures the robustness. The calculation results are shown in Table 2. It can be seen from the table that the calculation results of the method of the present invention in 4 groups of cross-domain experiments are better than the Cycle GAN method. The present invention can migrate the model trained in the labeled source domain to the actual unlabeled application scenario, which can significantly improve the crowd number prediction performance, provide more accurate crowd density estimation for public safety management, and provide a solution for crowd counting under real monitoring.
[0105] Table 2 Comparison of counting performance of the present invention and other methods
[0106]
[0107] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
[0108] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
Claims
1. A cross-domain crowd counting method combining fine-grained feature similarity retrieval, characterized by: The method comprises the following steps: S1 extracts fine-grained feature elements: collects fine-grained crowd distribution for source domain and target domain images respectively, and uses a sliding window strategy to obtain fine-grained blocks to extract fine-grained feature elements for the source domain and target domain. The source domain refers to a deep learning image dataset with crowd counting labels, and the target domain refers to a real-world scene image dataset to be detected. S2: Find the relationship between domains: Build a database of fine-grained feature elements of the source domain and the target domain, calculate the similarity between each element in the source domain and the elements in the database, and retrieve the source domain elements that contain the target domain elements and have the highest similarity with them according to the given fine-grained features of the target domain; S3 crowd count prediction: perform density map prediction on the target domain fine-grained block, fuse the predicted value with the matching value of the source domain fine-grained block to obtain a pseudo label, normalize the long and short sides of the target domain image to multiples of 128, and then sample the image using a sliding window. Finally, the total count of each image is the sum of the counts of the small blocks cut from each image; Step S1 includes the following steps: S11 Fine-grained crowd distribution collection: A sliding window with a size of (128, 128) and a step size of (64, 64) is used to slide across the entire image to obtain fine-grained blocks. The set of fine-grained blocks is defined as: S12 Fine-grained feature extraction: Use the position-sensitive backbone network HRNet and use small blocks with labels in the source domain to initialize the feature extraction model. The feature extraction model is defined as M r , r represents the number of iterations, and the initialization model is recorded as M 0 ; The source domain features and target domain features are defined as: in and Represent the i-th source domain feature and the j-th target domain feature respectively; S13 optimizes the feature extraction model using an independent structure similarity loss function and a mean square error loss, wherein the loss function is shown in formula (4), wherein the independent structure similarity loss is shown in formula (1), and L S As shown in formula (2), the structural loss is shown in formula (3): L S =1-SSIM(E,GT) (2) L=L MSE +ηL I-S (4) N is the total number of people, E is the predicted image, GT is the ground-truth, μ E and σ GT are the mean value of E and the standard deviation of GT, σ E·GT represents the covariance of E and GT, γ1 and γ2 are set to 0.0001 and 0.0009; Step S2 includes the following steps: S21 diffuses all fine-grained feature elements in the source domain and the target domain to form a database by finding a high similarity method, and searches for similar blocks in the database for the fine-grained features in the target domain to obtain the final feature sequence S22 Selection The rank1 in Its similarity score is recorded as Set the threshold λ to filter The characteristic pair of Step S21 includes the following steps: Construction of S211 graph: All features of the source domain and the target domain are taken as a set and defined as: Using local constraints, the graph only contains pairwise similarities and defines the affinity matrix: A = a ij ; S212 Graph normalization: Normalize the affinity matrix to S. The normalization process is shown in formula (5): D=∑a ij (6) S213 divides the normalized affinity matrix S into four blocks, as shown in formula (7); in, S mm S md S dm S dd Respectively represent the similarity between source domain to source domain, source domain to target domain, target domain to source domain, and target domain to target domain, where d = m + n; S214 Random Walk: Perform a random walk until it converges to a closed solution. The state of each iteration in the random walk is defined as: Each iteration strategy is shown in formula (8): f t+1 =αSf t +(1-a)f 0 ,α∈(0,1) (8) Among them, α is the probability of random walk to the next step, and (1-α) is the probability of restarting the walk from the initial state; S215 obtains a closed solution: The closed solution f is obtained by random walk in step S214 * As shown in formula (9), the similarity between each element in the source domain and the element in the database is included and used as the re-ranking score, as shown in formula (10): f * =(1-α)(1-αS)f 0 (9) S216 Feature Retrieval: Given the fine-grained features of the target domain, the final retrieval results are obtained through the KNN method. The retrieval results include the source domain elements with the highest similarity to the target domain elements; Step S3 includes the following steps: S31 Pseudo-label generation: For the fine-grained blocks of the target domain, density map prediction is performed, denoted as For the fine-grained blocks in the source domain, ground-truth matching is performed, denoted as And and Fusion and processing are performed to obtain the final pseudo label φ ij ; S32 model fine-tuning: fine-grained blocks of the target domain and generate pseudo labels l ij Composition block data set P r , fine-tune the model and iterate until the model converges; S33 From fine-grained to coarse-grained: normalize the long and short sides of the images in the target domain test set to multiples of 128, then sample the images and ground-truth using a sliding window of size (128, 128) and step size (128, 128). The count prediction result is the sum of the number of people predicted in all small blocks. Step S31 includes the following steps: S311 Target domain density map prediction: The FIDT method is used to predict the target domain density map. The process is shown in formula (11): S312 generates pseudo labels: and Fusion is performed as shown in formula (12), where W is the fusion weight. The fusion process is shown in formula (12), and the pseudo-label local maximum amplification is performed as shown in formula (13):
2. A cross-domain crowd counting system with fine-grained feature similarity retrieval, used to implement the cross-domain crowd counting method with fine-grained feature similarity retrieval according to claim 1, characterized in that: It includes a fine-grained distribution perception module, a distribution retrieval module and a domain adaptation module: The fine-grained distribution perception module is used to collect the fine-density crowd distribution of source domain and target domain images respectively, obtain fine-density blocks using a sliding window strategy, and extract fine-grained feature elements of the source domain and target domain. The source domain refers to a deep learning image dataset with crowd counting labels, and the target domain refers to a real-world scene image dataset to be detected. The distribution alignment retrieval module is used to find the domain relationship between the source domain and the target domain, perform feature retrieval between different domains, and select the fine-grained block with the highest similarity ranking in the retrieval results from the target domain to the source domain; The domain adaptation module predicts the density map of the fine-grained block in the target domain, fuses the predicted value with the matching value of the fine-grained block in the source domain to obtain a pseudo label, normalizes the long and short sides of the target domain image to multiples of 128, and then samples the image using a sliding window. Finally, the number of people in the entire image is calculated as the sum of the number of people in all small blocks divided by the image.
3. According to claim 2, a cross-domain crowd counting system combining fine-grained feature similarity retrieval is characterized by: The fine-grained distribution perception module uses a position-sensitive backbone network HRNet, and uses small blocks with labels in the source domain to initialize the feature extraction model, and uses the I-SSIM independent structural similarity loss function and the mean square error loss function for optimization.
4. According to claim 2, a cross-domain crowd counting system combining fine-grained feature similarity retrieval is characterized by: After the domain adaptation module generates pseudo labels, the target domain fine-grained blocks and generate pseudo labels l ij Composition block data set P r , fine-tune the models used in the distribution perception module and iterate until the model converges.
5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to claim 1 is implemented.
Citation Information
Patent Citations
Method for detecting and counting crowd distribution in video based on deep learning
CN113255430A
Methods and systems for wireless crowd counting
US20160315682A1
