Semi-supervised crowd counting method and system based on point model frequency domain screening

Abstract

The invention provides a semi-supervised crowd counting method and system based on point model frequency domain screening, relates to the field of crowd counting, and is applied to a semi-supervised crowd counting device based on point model frequency domain screening. The device comprises a point coordinate and confidence prediction module, a Patch confidence prediction module, a Patch confidence generation module and an unsupervised loss and frequency domain loss joint module. A reliable point coordinate and confidence coefficient prediction module and a Patch confidence coefficient prediction module are trained firstly, then an unsupervised loss and frequency domain loss joint module is combined, updated model parameters are transmitted to a teacher model by using a student model to obtain an updated teacher model, a label-free set is predicted through the updated teacher model, and the unsupervised loss and frequency domain loss joint module is used for predicting the label-free set. Therefore, semi-supervised population counting is realized, the accuracy of the prediction model is improved by updating the model parameters of the teacher model, and the accuracy of the prediction result of the updated teacher model is improved by optimizing the training set of the teacher model.

Description

Technical Field

[0001] The present invention relates to the technical field of crowd counting, and in particular to a semi-supervised crowd counting method and system based on point model frequency domain screening. Background Art

[0002] Crowd counting aims to estimate the number of people in a possibly crowded image. It has received extensive attention in recent years due to its wide applications in crowd monitoring, congestion estimation, and public safety.

[0003] Current problems with semi-supervised crowd counting: 1) In semi-supervised learning, pseudo labels for unlabeled data are generated by the model. Due to the limited performance of the model in the initial stage, these pseudo labels may be inaccurate or even have significant deviations. If these low-quality pseudo labels are used directly for training, the model may learn wrong information, affecting the final performance. 2) Due to differences in crowd distribution, camera angles, and lighting conditions, the generalization ability of the model in application scenarios other than training scenarios may be limited. This lack of cross-scenario generalization ability limits the effectiveness of the model in practical applications. 3) Semi-supervised learning models may exhibit poor robustness when faced with unseen scenes. In addition, due to the complexity of the model, it becomes more difficult to understand its decision-making process, which poses a challenge to its deployment in key application scenarios. Summary of the invention

[0004] Based on this, the purpose of the present invention is to provide a semi-supervised crowd counting method and system based on point model frequency domain screening, which is used to solve at least one technical problem in the prior art.

[0005] On one hand, the present invention provides a semi-supervised crowd counting method based on point model frequency domain screening, which is applied to a semi-supervised crowd counting device based on point model frequency domain screening. The device includes a point coordinate and confidence prediction module, a patch confidence prediction module, a patch confidence generation module, and an unsupervised loss and frequency domain loss joint module. The method includes: Obtain an RGB image of a crowd scene, and annotate the crowd positions of the RGB image to construct a data set for training a crowd counting model, wherein the data set includes a labeled set and an unlabeled set, wherein the labeled set includes a training set and a validation set, and generate head coordinate point information corresponding to each training image in the training set to obtain a GT image, wherein the GT image is a point image and the GT image is a true label image; The training image P1 in the labeled set is sent to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class; The predicted head point coordinates Pre point And the head point location reliability score Pre class The pseudo-real data set is passed to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and the confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training; The unlabeled image is sent to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage of training to obtain the predicted head point coordinates corresponding to the unlabeled image. Pre’ point , Head point location reliability score Pre’ class and reliable Patch confidence; filtering unreliable or uncertain patches in the image without label set according to the obtained reliable Patch confidence, wherein Patch is a block area; filtering the reliable Patch confidence and the head point position confidence score Pre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage of training; The training images in the unlabeled set, as well as the student model and the teacher model, are passed into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss that integrates the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

[0006] The above-mentioned semi-supervised crowd counting method based on frequency domain screening of point models first trains more reliable point coordinate and confidence prediction modules and Patch confidence prediction modules, and then combines the unsupervised loss and frequency domain loss joint module, and uses the student model to transmit the updated model parameters to the teacher model to obtain the updated teacher model, and then predicts the unlabeled set through the updated teacher model, thereby realizing semi-supervised crowd counting. The model parameters of the teacher model are updated to improve the accuracy of the prediction model. On the other hand, the training set of the teacher model is optimized to improve the accuracy of the prediction results of the updated teacher model.

[0007] In addition, the semi-supervised crowd counting method based on point model frequency domain screening according to the present invention may also have the following additional technical features: Further, the predicted head position coordinates Pre point And the head point location reliability score Pre class The data is transmitted to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The step of obtaining the Patch confidence dictionary according to the pseudo-real data set includes: According to the hyperparameter δ, the head point position confidence score is filtered out. Pre class The coordinates that are smaller than the hyperparameter δ are filtered out, and the points that exceed the image size horizontally and vertically are filtered out to generate a point map P2 based on all the remaining head coordinate points; Divide the point graph P2, the training image P1 in the labeled set, and the real point graph corresponding to the training image P1 in the labeled set into a plurality of patches of the same size, wherein a patch is a block area, and calculate the total distance between the point graph P2 and the real point graph corresponding to the training image P1 in the labeled set based on the Sinkhorn algorithm for the corresponding patch; The total distance is divided by the number of maximum point sets Num to obtain the average distance difference, and the average distance difference is used as the Patch confidence Pred t (S); the Patch confidence Pred t (S) is stored in the Patch confidence dictionary in one-to-one correspondence with the image position number; The image position number is composed of the number of the training image P1 in the labeled set in the data set and the sequential number of the patch in the same image; Among them, the calculation formula for the maximum number of point sets Num is: ; In the formula, Indicates the number of points contained in Patch in the real point map corresponding to the training image P1 in the labeled set; Indicates the number of points contained in the corresponding Patch in the point graph P2.

[0008] Furthermore, the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and together with the point coordinates and the confidence prediction module, a student model is constructed and the Patch confidence prediction module is trained so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training. The step also includes: The training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module to obtain the prediction confidence Pred corresponding to the Patch confidence dictionary. t (P); According to the Patch confidence Pred t (S) and the prediction confidence Pred t (P), calculate the Patch confidence loss l Patch To update the Patch confidence prediction module.

[0009] Further, according to the Patch confidence Pred t (S) and the prediction confidence Pred t (P), calculate the Patch confidence loss l Patch The steps include: The confidence level of the patch Pred t (S) is linearly scaled, and then the scaled Patch confidence Pred t (S) and the prediction confidence Pred t (P) Performing dimension increase to obtain all possible element pairs to generate a regional confidence difference matrix and calculate the correlation scores between all element pairs; Substitute the regional confidence difference matrix and the correlation score into the LambdaLoss formula to calculate the Patch confidence loss l Patch ; Among them, Patch confidence loss l Patch The calculation formula is: ; in, ; ; ; Where: l Patch Represents the Patch confidence loss l Patch , gt pred(S) Indicates that Pred t (S) normalization; Pred t (S) Regional confidence difference matrix; Pred t (S) Normalized matrix; Pred t (S) Normalized transposed matrix; Pred t (P) Regional confidence margin matrix; Represents the prediction confidence Pred t The transposed matrix of (P); N represents the number of patches; Among them, the confidence difference matrix is ​​used to calculate the difference in correlation scores of each item between matrices to calculate the confidence loss.

[0010] Further, unreliable or uncertain patches in the unlabeled image are screened according to the obtained reliable patch confidence, where a patch is a block area; the reliable patch confidence and the head point position confidence score are screened. Pre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage training steps include: The images in the unlabeled set P unlabel Input into the student model to get the predicted head position coordinates Pre’ point , Head point location reliability score Pre’ class and reliable patch confidence; For images in the unlabeled set P unlabel And the predicted head point coordinates generated by the student model Pre’point Generate forecast point map P dot Perform occlusion on Patch, where Patch is a block area; For images without label set P unlabel and predicted point plot P dot Perform image enhancement to obtain the occluded image P’ unlabel And the point map after occlusion P’ dot , where the occluded point map P’ dot As a real point map; The image after occlusion P’ unlabel Send it to the teacher model for training to get the predicted point map P dot_teacher , the teacher model predicts the point map P dot_teacher And the point map after occlusion P’ dot Compare to get the prediction gap between the student model and the teacher model, and use the prediction gap as a measure to get the unsupervised loss l unlabel .

[0011] Further, when screening the reliable patch confidence and the head point position confidence score Pre’ class In order to obtain a new data set, the point coordinate and confidence prediction module is retrained according to the new data set to construct a teacher model and then implement the second stage training step. The screening method of reliable patch confidence includes: All pixels in the Patch are screened according to a screening threshold to obtain Patches that meet the screening threshold and Patches that do not meet the screening threshold, wherein the Patch is a block area; the screening threshold is gradually increased with the number of training rounds; All pixels in the patch that do not meet the screening threshold are set to a specific value; P dot It is a binary image, and the corresponding patches are all set to 0; images without label sets P unlabel The corresponding patches are all set to fixed pixel values; Combine the two hyperparameters to filter out unreliable points in the patch that meet the screening threshold, where the screening object includes the head point location reliability score Pre’ class Points that are too low and predicted head point coordinates Pre’point The range does not correspond to the point of reality.

[0012] Furthermore, the unsupervised total loss L Including the total loss L of the first stage training 1总 And the total loss L of the second stage training 2总 , the total loss of the first stage training is L 1总 Including counting loss l count And Patch confidence loss l Patch ; The total loss of the second stage training is L 2总 Including the total loss L of the first stage training 1总 And the unsupervised loss l unlabel ; Among them, the counting loss l count The calculation method is: Use L1 loss function to calculate the predicted head point coordinates Pre point The error with the GT image, and the count loss is obtained through the weight coefficient l count .

[0013] On the other hand, the present invention provides a semi-supervised crowd counting system based on point model frequency domain screening, which is applied to a semi-supervised crowd counting device based on point model frequency domain screening, the device comprising a point coordinate and confidence prediction module, a patch confidence prediction module, a patch confidence generation module, and an unsupervised loss and frequency domain loss joint module, the system comprising: An acquisition module is used to acquire an RGB image of a crowd scene, and mark the crowd positions of the RGB image to construct a data set used for training a crowd counting model, wherein the data set includes a labeled set and an unlabeled set, wherein the labeled set includes a training set and a validation set, and generates head coordinate point information corresponding to each training image in the training set to obtain a GT image, wherein the GT image is a point image and the GT image is a true label image; The first stage training module is used to send the training image P1 in the labeled set to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class ; The predicted head point coordinates Pre point And the head point location reliability score Pre classThe pseudo-real data set is passed to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and the confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training; The second stage training module is used to send the unlabeled set images to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage training to obtain the predicted head point coordinates corresponding to the unlabeled set images. Pre’ point , Head point location reliability score Pre’ class and reliable Patch confidence; filtering unreliable or uncertain patches in the image without label set according to the obtained reliable Patch confidence, wherein Patch is a block area; filtering the reliable Patch confidence and the head point position confidence score Pre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage of training; The counting module is used to pass the training images in the unlabeled set, the student model and the teacher model into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss integrated with the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

[0014] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the semi-supervised crowd counting method based on point model frequency domain screening as described above.

[0015] On the other hand, the present invention also provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the semi-supervised crowd counting method based on frequency domain screening of point models as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of a semi-supervised crowd counting method based on point model frequency domain screening in an embodiment of the present invention; Figure 2 Schematic diagram of the architectural relationship between the Patch confidence prediction module and the Patch confidence generation module in an embodiment of the present invention; Figure 3 This is a schematic diagram of the second stage training process in an embodiment of the present invention; The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0019] A stable and accurate crowd counter usually relies on sufficient labeled images where each head center is annotated. The annotation process can be labor-intensive and time-consuming. Therefore, semi-supervised crowd counting attempts to reduce the annotation cost while achieving good counting accuracy.

[0020] The semi-supervised crowd counting method and system based on point model frequency domain screening provided in the present application first trains a relatively reliable point coordinate and confidence prediction module and a Patch confidence prediction module, and then combines the unsupervised loss and frequency domain loss joint module, uses the student model to transmit the updated model parameters to the teacher model to obtain the updated teacher model, and then predicts the unlabeled set through the updated teacher model, thereby realizing semi-supervised crowd counting, and improving the accuracy of the prediction model by updating the model parameters of the teacher model, and on the other hand, improving the accuracy of the prediction results of the updated teacher model by optimizing the training set of the teacher model.

[0021] In order to facilitate understanding of the present invention, several embodiments of the present invention are provided below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0022] Embodiment 1 See also Figure 1 , shown is a semi-supervised crowd counting method based on point model frequency domain screening in the first embodiment of the present invention, which is applied to a semi-supervised crowd counting device based on point model frequency domain screening, the device includes a point coordinate and confidence prediction module, a Patch confidence prediction module, a Patch confidence generation module, and an unsupervised loss and frequency domain loss joint module, and the method includes steps S101 to S104: S101. Obtain an RGB image of a crowd scene, and annotate the crowd positions in the RGB image to construct a data set for training a crowd counting model. The data set includes a labeled set and an unlabeled set. The labeled set includes a training set and a validation set. Generate head coordinate point information corresponding to each training image in the training set to obtain a GT image. The GT image is a point image, and the GT image is a true label image.

[0023] The dataset is divided into labeled and unlabeled sets, and divided into three data volumes of 5%, 10%, and 40% according to the task. The percentage represents the proportion of the labeled set in the original dataset. The prediction set (i.e., the unlabeled set) contains only RGB images of crowd scenes, without the coordinate point information of the head; the labeled set is further divided into training and validation sets in proportion. The training set, prediction set, and unlabeled set use random scaling, cropping, horizontal flipping, and other strategies to enhance the data set to improve the generalization ability of the model.

[0024] S102: Send the training image P1 in the label set to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class ; Predict the head point coordinates Pre point and the head point location reliability score Pre class It is passed into the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the label set, the real label map corresponding to the training image P1 in the label set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training.

[0025] Send the labeled training image P1 into the VGG16 network to obtain the image features Fs , and pass the C3, C4 and C5 layer features of the model to the Feature Pyramid Networks (FPN), and merge the up-sampled features, down-sampled features and features of the same layer to obtain the fourth layer features, and pass the fourth layer features to the regression module and classification module in the two parallel branches for coordinate regression and classification. The two parallel branches include regression branches and classification branches. The regression module corresponds to the regression branch, and the classification module corresponds to the classification branch. For the regression branch, due to the inherent translation invariance of the convolutional layer, it uses the offset of the predicted point coordinates. Specifically, each pixel corresponds to the size of the input image. s × s There is a set of K reference points in each patch, which are densely arranged on the patch. Patch is a block area. This branch is to predict the offset △(x, y), and then add the original coordinates to get the predicted head point coordinates. Pre point ; For the classification branch, it outputs the head point location confidence score with Softmax normalization Pre class , the reliability score of the head point position Pre class Based on the predicted head position coordinates Pre point The corresponding confidence scores generated one-to-one.

[0026] Since semi-supervised crowd counting generally allows the model to directly generate pseudo labels or directly generate corresponding confidences at corresponding points, the model will inevitably learn some useless data or knowledge during the training phase. At the same time, when training the model, especially the unsupervised reasoning pseudo-labeling process, it will inevitably consume a lot of computing power and time. Therefore, it is hoped that a confidence generation network similar to a regression network can be trained during the training phase to roughly filter out useless data. Based on this, in this application, a Patch confidence prediction module and a Patch confidence generation module are proposed, such as Figure 2 shown.

[0027] From the above, we can get the head point position confidence score with Softmax normalization through the point coordinates and confidence prediction module for a training image P1 Pre class And predict the head point coordinates Pre point For the head point location reliability score Pre class , according to the hyperparameter δ, the head point position confidence score is filtered out Pre classThe coordinates in the image that are smaller than the hyperparameter δ are filtered out, and the points that exceed the image size horizontally and vertically are respectively filtered out to generate a point map P2 according to all the remaining head coordinate points; the point map P2, the training image P1 in the labeled set, and the real point map corresponding to the training image P1 in the labeled set are divided into multiple patches of the same size, where a patch is a block area, and the total distance between the point map P2 and the real point map corresponding to the training image P1 in the labeled set is calculated based on the Sinkhorn algorithm for the corresponding patch; the total distance is divided by the number of the maximum point set Num to obtain the average distance difference, and the average distance difference is used as the patch confidence Pred t (S); Patch confidence Pred t (S) is stored in the Patch confidence dictionary in one-to-one correspondence with the image position number; The image position number is composed of the number of the training image P1 in the labeled set in the data set and the sequential number of the patch in the same image; Among them, the calculation formula for the maximum number of point sets Num is: ; In the formula, Indicates the number of points contained in Patch in the real point map corresponding to the training image P1 in the labeled set; Indicates the number of points contained in the corresponding Patch in the point graph P2.

[0028] Furthermore, in the first stage of training, the training image P1 in the label set, the real label map corresponding to the training image P1 in the label set, and the Patch confidence dictionary are input into the Patch confidence prediction module to obtain the prediction confidence Pred corresponding to the Patch confidence dictionary. t (P); According to the Patch confidence Pred t (S) and prediction confidence Pred t (P), calculate the Patch confidence loss l Patch To update the Patch confidence prediction module.

[0029] Specifically, according to the Patch confidence Pred t (S) and prediction confidence Pred t (P), calculate the Patch confidence loss l Patch The steps include: Patch confidence Pred t (S) is linearly scaled, and then the scaled Patch confidence Pred t (S) and prediction confidence Predt (P) Performing dimension increase to obtain all possible element pairs to generate a regional confidence difference matrix and calculate the correlation scores between all element pairs; Substitute the regional confidence difference matrix and the correlation score into the LambdaLoss formula to calculate the Patch confidence loss l Patch ; Among them, Patch confidence loss l Patch The calculation formula is: ; in, ; ; ; Where: l Patch Represents the Patch confidence loss l Patch , gt pred(S) Indicates that Pred t (S) normalization; Pred t (S) Regional confidence difference matrix; Pred t (S) Normalized matrix; Pred t (S) Normalized transposed matrix; Pred t (P) Regional confidence margin matrix; Represents the prediction confidence Pred t The transposed matrix of (P); N represents the number of patches; Among them, the confidence difference matrix is ​​used to calculate the difference in correlation scores of each item between matrices to calculate the confidence loss.

[0030] S103: Send the image of the unlabeled set to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage of training to obtain the predicted head point coordinates corresponding to the image of the unlabeled set. Pre’ point , Head point location reliability score Pre’ classand reliable Patch confidence; filter unreliable or uncertain patches in the unlabeled image according to the obtained reliable Patch confidence, where Patch is a block area; filter reliable Patch confidence and head point position confidence score Pre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to build a teacher model and then implement the second stage of training.

[0031] For semi-supervised learning, the most important part is the semi-supervised reasoning part. This application is different from the usual semi-supervised crowd counting. This application not only has double reset confidence to ensure the high credibility of the predicted crowd points, but also finds through research that the information of density maps or point maps exists everywhere in the spatial domain, but in the frequency domain, most of the information is concentrated in a very compact range. Based on this, this application proposes a joint module of unsupervised loss and frequency domain loss. In the first stage of training, the present application has trained a relatively reliable point coordinate and confidence prediction module and a Patch confidence prediction module. In the second stage, the present application uses the student model and the teacher model to transfer the learned knowledge to update the teacher model, and uses the teacher model to predict the unlabeled data set of the present application. For Patch screening, the screening threshold of the present application is gradually strengthened with the number of training rounds. According to the screening threshold, all pixels in the Patch are screened to obtain Patches that meet the screening threshold and Patches that do not meet the screening threshold, where the Patch is a block area; all pixels in the Patch that does not meet the screening threshold are set to a specific value; the two hyperparameters are combined to screen unreliable points in the Patch that meets the screening threshold, where the screening objects include the head point position confidence score Pre’ class Points that are too low and predicted head point coordinates Pre’ point The range does not match the actual point; about setting all pixels in the patch that do not meet the screening threshold to a specific value, specifically, due to the predicted point map P dot It is a binary image, and the corresponding patches are all set to 0; images without label sets P unlabel The corresponding patches are all set to fixed pixel values. Therefore, the crowd point coordinates determined under the double reset confidence standard of this application are used as pseudo labels, such as Figure 3 shown.

[0032] Specifically, the present application uses the above-mentioned Patch confidence screening method to screen unreliable or uncertain patches in the unlabeled image according to the obtained reliable Patch confidence, where Patch is a block area; screens the reliable Patch confidence and the head point position confidence scorePre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to build a teacher model and then implement the second stage of training, including the following steps: The images in the unlabeled set P unlabel Input into the student model to get the predicted head position coordinates Pre’ point , Head point location reliability score Pre’ class and reliable Patch confidence; for images without label sets P unlabel And the predicted head point coordinates generated by the student model Pre’ point Generate forecast point map P dot Perform occlusion on Patch, where Patch is a block area; for images without label sets P unlabel and predicted point plot P dot Perform image enhancement to obtain the occluded image P’ unlabel And the point map after occlusion P’ dot , where the occluded point map P’ dot As the real point map; the occluded image P’ unlabel Send it to the teacher model for training to get the predicted point map P dot_teacher , the teacher model predicts the point map P dot_teacher And the point map after occlusion P’ dot Compare to get the prediction gap between the student model and the teacher model, and use the prediction gap as a measure to get the unsupervised loss l unlabel .

[0033] S104: The training images in the unlabeled set, as well as the student model and the teacher model, are passed into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss that integrates the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

[0034] In this example, the unsupervised total loss LIncluding the total loss L of the first stage training 1总 And the total loss L of the second stage training 2总 , the total loss of the first stage training is L 1总 Including counting loss l count And Patch confidence loss l Patch ; The total loss of the second stage training is L 2总 Including the total loss L of the first stage training 1总 And the unsupervised loss l unlabel ; Among them, the counting loss l count The calculation method is: Use L1 loss function to calculate the predicted head point coordinates Pre point The error with the GT image, and the count loss is obtained through the weight coefficient l count .

[0035] In this embodiment, the semi-supervised crowd counting method based on point model frequency domain screening also includes: Calculate the sum of the differences between the characteristic functions of the predicted density map and the true density map in the entire frequency domain to convert the teacher model prediction point map P dot_teacher And the point map after occlusion P ’ dot Feature distribution after conversion to frequency domain space; specifically: Construct the spatial domain sampling coordinate template and the frequency domain frequency coordinate system plane, and perform inner product through matrix multiplication to obtain the phase angle corresponding to each frequency point; through the obtained phase angle, use the Euler formula to obtain the real and imaginary templates in the frequency domain, and then compare them with the teacher model prediction point map P dot_teacher And the point map after occlusion P ’ dot Multiply to get the predicted frequency domain feature map of frequency domain distribution f teacher And the real frequency domain feature map f dot ; By mapping the density map coordinates in the spatial domain to the frequency domain and using the feature function for supervised learning, the model can better capture the global and local information in the density map to improve the accuracy and generalization ability of crowd counting. The specific implementation is as follows: The spatial domain represents the distribution of the signal in space, such as the distribution of pixels in an image. The frequency domain represents the frequency components of the signal, that is, the influence of each frequency component in the image. Since a signal can be represented as a combination of sine waves and cosine waves of different frequencies (i.e., frequency domain). Specifically, assuming there is a two-dimensional image, the spatial domain coordinates of a certain point pixel are (x, y) and the frequency domain coordinates are (u, v). The spatial signal of the image can be mapped to the frequency domain through Fourier transform, which is specifically expressed as: ; in, f ( x , y ) is a signal in the spatial domain, such as an image pixel value; F ( u , v ) is the frequency domain component of the image in the frequency domain.

[0036] In this embodiment, the inner product is used to obtain the projection of the pixel information in the spatial domain in the frequency domain. F ( u , v )=cos( θ )+jsin( θ ) can get the real part cos( θ ) and the imaginary part sin( θ ), and the real part cos( θ ) and the imaginary part sin( θ ) is spliced ​​in one dimension to obtain a frequency domain feature function, and then the predicted density map is P dot_teacher and the true density map P ’ dot The characteristic functions of are converted into predicted frequency domain feature maps through the above method. f teacher And the real frequency domain feature map f dot . The predicted frequency domain feature map is calculated by L2 norm f teacher Compared with the real frequency domain feature map f dot The difference between them is taken as the counting loss in the frequency domain.

[0037] ; in, m It is the frequency domain sampling step size, which sets the distance between frequency domain coordinate points and is a preset parameter.

[0038] In summary, the semi-supervised crowd counting method based on frequency domain screening of point models in the above embodiments of the present invention first trains a relatively reliable point coordinate and confidence prediction module and a Patch confidence prediction module, and then combines the unsupervised loss and frequency domain loss joint module, and uses the student model to transmit the updated model parameters to the teacher model to obtain an updated teacher model, and then predicts the unlabeled set through the updated teacher model, thereby realizing semi-supervised crowd counting, and improving the accuracy of the prediction model by updating the model parameters of the teacher model, and on the other hand, improving the accuracy of the prediction results of the updated teacher model by optimizing the training set of the teacher model.

[0039] Embodiment 2 The second embodiment of the present invention provides a semi-supervised crowd counting system based on point model frequency domain screening, which is applied to a semi-supervised crowd counting device based on point model frequency domain screening. The device includes a point coordinate and confidence prediction module, a patch confidence prediction module, a patch confidence generation module, and an unsupervised loss and frequency domain loss joint module. The system includes: An acquisition module is used to acquire an RGB image of a crowd scene, and mark the crowd positions of the RGB image to construct a data set used for training a crowd counting model, wherein the data set includes a labeled set and an unlabeled set, wherein the labeled set includes a training set and a validation set, and generates head coordinate point information corresponding to each training image in the training set to obtain a GT image, wherein the GT image is a point image and the GT image is a true label image; The first stage training module is used to send the training image P1 in the labeled set to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class ; The predicted head point coordinates Pre point And the head point location reliability score Pre class The pseudo-real data set is passed to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and the confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training; The second stage training module is used to send the unlabeled set images to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage training to obtain the predicted head point coordinates corresponding to the unlabeled set images. Pre’ point , Head point location reliability score Pre’ class and reliable Patch confidence; filtering unreliable or uncertain patches in the image without label set according to the obtained reliable Patch confidence, wherein Patch is a block area; filtering the reliable Patch confidence and the head point position confidence score Pre’ class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage of training; The counting module is used to pass the training images in the unlabeled set, the student model and the teacher model into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss integrated with the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

[0040] In summary, the semi-supervised crowd counting system based on point model frequency domain screening in the above embodiments of the present invention first trains a relatively reliable point coordinate and confidence prediction module and a Patch confidence prediction module, and then combines the unsupervised loss and frequency domain loss joint module, and uses the student model to transmit the updated model parameters to the teacher model to obtain an updated teacher model, and then predicts the unlabeled set through the updated teacher model, thereby realizing semi-supervised crowd counting, and improving the accuracy of the prediction model by updating the model parameters of the teacher model, and on the other hand, improving the accuracy of the prediction results of the updated teacher model by optimizing the training set of the teacher model.

[0041] In addition, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0042] In addition, an embodiment of the present invention further provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method in the above embodiment when executing the program.

[0043] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0044] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0045] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0046] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0047] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A semi-supervised crowd counting method based on point model frequency domain screening, characterized in that: A semi-supervised crowd counting device is applied to frequency domain screening based on a point model, the device comprises a point coordinate and confidence prediction module, a patch confidence prediction module, a patch confidence generation module, and an unsupervised loss and frequency domain loss joint module, the method comprises: Obtain an RGB image of a crowd scene, and annotate the crowd positions of the RGB image to construct a data set for training a crowd counting model, wherein the data set includes a labeled set and an unlabeled set, wherein the labeled set includes a training set and a validation set, and generate head coordinate point information corresponding to each training image in the training set to obtain a GT image, wherein the GT image is a point image and the GT image is a true label image; The training image P1 in the labeled set is sent to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class ; The predicted head point coordinates Pre point And the head point location reliability score Pre class The pseudo-real data set is passed to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and the confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training; The unlabeled image is sent to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage of training to obtain the predicted head point coordinates corresponding to the unlabeled image. Pre' point , Head point location reliability score Pre' class and reliable Patch confidence; filtering unreliable or uncertain patches in the image without label set according to the obtained reliable Patch confidence, wherein Patch is a block area; filtering the reliable Patch confidence and the head point position confidence score Pre' class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage of training; The training images in the unlabeled set, as well as the student model and the teacher model, are passed into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss that integrates the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

2. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 1 is characterized in that: The predicted head point coordinates Pre point And the head point location reliability score Pre class The data is transmitted to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The step of obtaining the Patch confidence dictionary according to the pseudo-real data set includes: According to the hyperparameter δ, the head point position confidence score is filtered out. Pre class The coordinates that are smaller than the hyperparameter δ are filtered out, and the points that exceed the image size horizontally and vertically are filtered out to generate a point map P2 based on all the remaining head coordinate points; Divide the point graph P2, the training image P1 in the labeled set, and the real point graph corresponding to the training image P1 in the labeled set into a plurality of patches of the same size, wherein a patch is a block area, and calculate the total distance between the point graph P2 and the real point graph corresponding to the training image P1 in the labeled set based on the Sinkhorn algorithm for the corresponding patch; The total distance is divided by the number of maximum point sets Num to obtain the average distance difference, and the average distance difference is used as the Patch confidence Pred t (S); the Patch confidence Pred t (S) is saved in the Patch confidence dictionary in one-to-one correspondence with the image position number; The image position number is composed of the number of the training image P1 in the labeled set in the data set and the sequential number of the patch in the same image; Among them, the calculation formula for the maximum number of point sets Num is: ； In the formula, Indicates the number of points contained in Patch in the real point map corresponding to the training image P1 in the labeled set; Indicates the number of points contained in the corresponding Patch in the point graph P2.

3. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 2 is characterized in that: The steps of inputting the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary into the Patch confidence prediction module and constructing a student model together with the point coordinates and the confidence prediction module, and training the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training also include: The training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module to obtain the prediction confidence Pred corresponding to the Patch confidence dictionary. t (P); According to the Patch confidence Pred t (S) and the prediction confidence Pred t (P), calculate the Patch confidence loss l Patch To update the Patch confidence prediction module.

4. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 3 is characterized in that: According to the Patch confidence Pred t (S) and the prediction confidence Pred t (P), calculate the Patch confidence loss l Patch The steps include: The confidence level of the patch Pred t (S) is linearly scaled, and then the scaled Patch confidence Pred t (S) and the prediction confidence Pred t (P) Performing dimension increase to obtain all possible element pairs to generate a regional confidence difference matrix and calculate the correlation scores between all element pairs; Substitute the regional confidence difference matrix and the correlation score into the LambdaLoss formula to calculate the Patch confidence loss l Patch ; Among them, Patch confidence loss l Patch The calculation formula is: ； in, ; ； ； Where: l Patch Represents the Patch confidence loss l Patch , gt pred(S) Indicates that Pred t (S) normalization; Pred t (S) regional confidence difference matrix; Pred t (S) Normalized matrix; Pred t (S) Normalized transposed matrix; Pred t (P) Regional confidence margin matrix; Represents the prediction confidence Pred t The transposed matrix of (P); N represents the number of patches; Among them, the confidence difference matrix is ​​used to calculate the difference in correlation scores of each item between matrices to calculate the confidence loss.

5. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 4 is characterized in that: Filter unreliable or uncertain patches in the unlabeled image according to the obtained reliable patch confidence, where a patch is a block area; filter the reliable patch confidence and the head point position confidence score Pre' class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage training steps include: The images in the unlabeled set P unlabel Input into the student model to get the predicted head position coordinates Pre' point , Head point location reliability score Pre' class and reliable patch confidence; For images in the unlabeled set P unlabel And the predicted head point coordinates generated by the student model Pre' point Generate forecast point map P dot Perform occlusion on Patch, where Patch is a block area; For images without label set P unlabel and predicted point plot P dot Perform image enhancement to obtain the occluded image P’ unlabel And the point map after occlusion P’ dot , where the occluded point map P’ dot As a real point map; The image after occlusion P’ unlabel Send it to the teacher model for training to get the predicted point map P dot_teacher , the teacher model predicts the point map P dot_teacher And the point map after occlusion P’ dot Compare to get the prediction gap between the student model and the teacher model, and use the prediction gap as a measure to get the unsupervised loss l unlabel .

6. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 1 is characterized in that: When screening the reliable patch confidence and the head point position confidence score Pre' class In order to obtain a new data set, the point coordinate and confidence prediction module is retrained according to the new data set to construct a teacher model and then implement the second stage training step. The screening method of reliable patch confidence includes: All pixels in the Patch are screened according to a screening threshold to obtain Patches that meet the screening threshold and Patches that do not meet the screening threshold, wherein the Patch is a block area; the screening threshold is gradually increased with the number of training rounds; All pixels in the patch that do not meet the screening threshold are set to a specific value; P dot It is a binary image, and the corresponding patches are all set to 0; images without label sets P unlabel The corresponding patches are all set to fixed pixel values; Combine the two hyperparameters to filter out unreliable points in the patch that meet the screening threshold, where the screening object includes the head point location reliability score Pre' class Points that are too low and predicted head point coordinates Pre' point The range does not correspond to the point of reality.

7. The semi-supervised crowd counting method based on point model frequency domain screening according to claim 5 is characterized in that: The unsupervised total loss L Including the total loss L of the first stage training 1总 And the total loss L of the second stage training 2总 , the total loss of the first stage training is L 1总 Including counting loss l count And Patch confidence loss l Patch ; The total loss L of the second stage training 2总 Including the total loss L of the first stage training 1总 And the unsupervised loss l unlabel ; Among them, the counting loss l count The calculation method is: Use L1 loss function to calculate the predicted head point coordinates Pre point The error with the GT image, and the count loss is obtained through the weight coefficient l count .

8. A semi-supervised crowd counting system based on point model frequency domain screening, characterized in that: A semi-supervised crowd counting device for frequency domain screening based on a point model, the device comprising a point coordinate and confidence prediction module, a patch confidence prediction module, a patch confidence generation module, and an unsupervised loss and frequency domain loss joint module, the system comprising: An acquisition module is used to acquire an RGB image of a crowd scene, and mark the crowd positions of the RGB image to construct a data set used for training a crowd counting model, wherein the data set includes a labeled set and an unlabeled set, wherein the labeled set includes a training set and a validation set, and generates head coordinate point information corresponding to each training image in the training set to obtain a GT image, wherein the GT image is a point image and the GT image is a true label image; The first stage training module is used to send the training image P1 in the labeled set to the point coordinate and confidence prediction module to obtain the predicted head point coordinates Pre point and the head point location reliability score Pre class ; The predicted head point coordinates Pre point And the head point location reliability score Pre class The pseudo-real data set is passed to the Patch confidence generation module to obtain a pseudo-real data set, and the pseudo-real data set is used to train the Patch confidence prediction module. The Patch confidence dictionary is obtained according to the pseudo-real data set; the training image P1 in the labeled set, the real label map corresponding to the training image P1 in the labeled set, and the Patch confidence dictionary are input into the Patch confidence prediction module and constructed together with the point coordinates and the confidence prediction module to obtain a student model and train the Patch confidence prediction module so that the Patch confidence prediction module has a reliable ability to generate reliable Patch confidence to complete the first stage of training; The second stage training module is used to send the unlabeled set images to the point coordinate and confidence prediction module and the Patch confidence prediction module that have completed the first stage training to obtain the predicted head point coordinates corresponding to the unlabeled set images. Pre' point , Head point location reliability score Pre' class and reliable Patch confidence; filtering unreliable or uncertain patches in the image without label set according to the obtained reliable Patch confidence, wherein Patch is a block area; filtering the reliable Patch confidence and the head point position confidence score Pre' class To obtain a new data set, retrain the point coordinate and confidence prediction modules according to the new data set to construct a teacher model and then implement the second stage of training; The counting module is used to pass the training images in the unlabeled set, the student model and the teacher model into the unsupervised loss and frequency domain loss joint module to obtain the unsupervised total loss integrated with the unsupervised loss and the frequency domain loss. L , according to the unsupervised total loss L The student model is iterated to update the student model parameters, the teacher model is updated according to the updated student model parameters to obtain an updated teacher model, and crowd counting is performed on the image to be counted according to the updated teacher model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the semi-supervised crowd counting method based on point model frequency domain screening as described in any one of claims 1 to 7 is implemented.

10. A data processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the semi-supervised crowd counting method based on point model frequency domain screening as described in any one of claims 1 to 7 is implemented.

Images