Training method, device, computer equipment and storage medium for people detection model

Abstract

The present application relates to a training method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a crowd detection model. The method comprises: obtaining unlabeled features and labeled features obtained from capturing a crowd scene from multiple perspectives; predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives; comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result; determining the unlabeled loss based on the comparison result; and training a crowd prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features. This method can alleviate the dependence on labeling and improve the accuracy of crowd prediction.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method, apparatus, computer equipment, computer-readable storage medium, and computer program product for a people detection model. Background Art

[0002] With the development of society, more and more scenarios require crowd computing to provide data support, optimize resource allocation, and promote corresponding development in areas such as security monitoring and business decision-making.

[0003] Traditional crowd detection models primarily use sliding window detectors based on global and local detection, support vector machines, and random forest models. However, these methods work well for sparse crowds but are less effective when crowd density increases or occlusion is severe. Building on these models, regression methods have been introduced, first extracting low-level features (such as shape and texture) and then predicting the number of people using a regression model. However, this method struggles to capture high-level semantic features in complex scenarios, limiting its applicability. Accuracy still needs to be improved. Summary of the Invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a people detection model that can alleviate the dependence on labeling and improve accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for training a people detection model, the method comprising:

[0006] Obtain unlabeled features and labeled features from crowd scenes captured from multiple perspectives;

[0007] Predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0008] Comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result;

[0009] determining the unmarked loss according to the comparison result;

[0010] A headcount prediction model is trained according to the unlabeled loss and the labeled loss corresponding to the labeled features.

[0011] In one embodiment, the predicting of the number of people uncertainty based on the unlabeled features of each group of viewing angles to obtain the number of people uncertainty corresponding to each group of viewing angles includes:

[0012] Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives;

[0013] Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0014] In one embodiment, comparing the uncertainty of the number of people corresponding to each set of perspectives to obtain a comparison result includes:

[0015] Comparing the uncertainty of the number of people corresponding to each group of perspectives in sequence according to the number of perspectives, and obtaining each comparison result obtained in sequence;

[0016] Determining the unlabeled loss according to the comparison result includes:

[0017] Based on each comparison result in turn, an unlabeled loss is determined.

[0018] In one embodiment, the uncertainty of the number of people corresponding to each group of viewing angles includes the uncertainty of the number of people in the i-th group of viewing angles and the uncertainty of the number of people in the j-th group of viewing angles, where i is less than j, and both i and j are positive integers;

[0019] Determining the unlabeled loss according to the comparison result includes:

[0020] If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, then adjusting the unannotated loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints;

[0021] If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

[0022] In one embodiment, adjusting the unlabeled loss according to the difference in the number of people uncertainty between the j-th group of viewing angles and the i-th group of viewing angles includes:

[0023] Performing a difference calculation based on the uncertainty of the number of people in the j-th group of viewpoints and the uncertainty of the number of people in the i-th group of viewpoints to obtain a current uncertainty difference;

[0024] Determining a penalty value that is positively correlated with the current uncertainty difference;

[0025] The current unlabeled loss is adjusted according to the penalty value to obtain an adjusted unlabeled loss.

[0026] In one embodiment, before training the headcount prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes:

[0027] Determine the nth labeled predicted number of people and the uncertainty of the labeled predicted number of people based on the labeled features of the first n perspectives;

[0028] Determining the nth auxiliary task loss based on the mean square error between the nth labeled predicted number of people and the reference number of people;

[0029] determining an nth reference uncertainty based on an absolute difference between the nth annotated predicted number of people and the reference number of people, and determining an nth annotated uncertainty loss based on a mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty;

[0030] Determining the labeled loss based on (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, n is greater than 1, n is less than or equal to (k-1), and k is the number of the view angles;

[0031] Determining the main task loss based on a mean square error between the k-th labeled predicted number of people and the reference number of people; the k-th labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives;

[0032] The training of the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features includes:

[0033] A number prediction model is trained according to the unlabeled loss, the labeled loss, and the main task loss.

[0034] In one embodiment, the method further comprises:

[0035] Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people;

[0036] The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number of people mark map; the pixel values ​​in the number of people mark map represent the reference number of people.

[0037] In a second aspect, the present application also provides a training device for a people detection model, comprising:

[0038] An acquisition module is used to obtain unlabeled features and labeled features obtained by capturing crowd scenes from multiple perspectives;

[0039] A prediction module, configured to predict the uncertainty of the number of people based on the unlabeled features of each group of perspectives, to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0040] A comparison module, configured to compare the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result;

[0041] An adjustment module, configured to determine an unlabeled loss according to the comparison result;

[0042] A training module is used to train a headcount prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features.

[0043] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of training the people detection model in any of the above embodiments are implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of training the people detection model in any of the above embodiments.

[0045] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of training the people detection model in any of the above embodiments.

[0046] The training method, apparatus, computer device, computer-readable storage medium, and computer program product for the aforementioned crowd detection model acquire unlabeled and labeled features from crowd scenes captured from multiple viewpoints, and can mitigate the effects of occlusion across multiple viewpoints, helping to ensure the accuracy of crowd recognition. Furthermore, by predicting the number of people based on the unlabeled features for each set of viewpoints, the uncertainty of the number of people corresponding to each set of viewpoints is obtained, forming a method for fusion prediction across different numbers of viewpoints. This allows for separate training of unlabeled and labeled data from different numbers of viewpoints, forming a semi-supervised training model. This significantly reduces reliance on labeled data and eliminates the need for additional labeling. This allows unlabeled features to be trained in conjunction with tasks such as population density regression, improving crowd counting performance. By comparing the uncertainty of the number of people corresponding to each set of perspectives, a comparison result of the uncertainty is obtained. This comparison result can be used to measure the reliability of different multi-perspective fusion processes and reflect whether there are errors in the corresponding prediction process, thereby forming an order of uncertainty results for each set of perspectives. The corresponding unlabeled loss is then gradually adjusted based on the comparison result, so that the unlabeled loss is determined based on the principle of size sorting. Finally, the unlabeled loss and the labeled loss corresponding to the labeled features are simultaneously applied to the training process of the number of people prediction model. In this way, multi-perspective fusion prediction is performed on different numbers of perspective features, and then the uncertainty results of the prediction results are sorted, introducing constraints for the multi-perspective crowd counting model. Semi-supervised training is performed based on the relationship between the number of perspectives and the uncertainty of the number of people. Unlabeled features that do not require additional labeling are combined with the corresponding tasks of labeled features for training, forming a semi-supervised model training method, which can improve crowd counting performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A diagram illustrating an application environment of a training method for a people detection model in one embodiment;

[0049] Figure 2 1 is a flow chart of a method for training a people detection model in one embodiment;

[0050] Figure 3 A schematic diagram of a process for ranking uncertainty in one embodiment;

[0051] Figure 4 A schematic diagram of a process for unlabeled loss adjustment in one embodiment;

[0052] Figure 5 A schematic diagram of a process for determining a marked loss in one embodiment;

[0053] Figure 6 This is a schematic diagram of the effect of an uncertainty diagram in one embodiment;

[0054] Figure 7 A schematic diagram of the network structure of a headcount prediction model in one embodiment;

[0055] Figure 8 A schematic diagram of a process for predicting the number of people in one embodiment;

[0056] Figure 9 A schematic diagram of the characteristic effect of fusion of each set of viewing angle features in one embodiment;

[0057] Figure 10 is a structural block diagram of a training device for a people detection model in one embodiment;

[0058] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0060] When counting people in crowds, traditional single-view crowd counting methods often struggle to achieve satisfactory results due to severe occlusion. To address this challenge, multi-view crowd counting can be performed. However, the acquisition and annotation of multi-view images present significant challenges, resulting in a scarcity of available multi-view crowd counting datasets. To address this data limitation, synthetic data can be collected to circumvent the annotation step, thereby compensating for the lack of real data and enabling fully supervised training. Alternatively, semi-supervised or unsupervised learning methods can be proposed on limited multi-view datasets, hoping to better adapt to real-world scenarios.

[0061] Alternatively, crowd density estimation can be performed based on a Gaussian convolution of the crowd density map. Because training is based on the difference between the predicted and true density maps, it avoids reliance on simple regression or classification, resulting in better interpretability and learnability. This method excels at addressing crowd scale issues from a single viewpoint, but it still struggles with occlusion. To address this, multi-view crowd counting methods have been proposed, mitigating the impact of occlusion by using multiple viewpoints.

[0062] If multi-view counting methods rely entirely on fully supervised training, they require a large amount of labeled multi-view scene data, while existing real-world multi-view crowd data annotation is relatively scarce. Moreover, there are currently no relevant semi-supervised methods in the field of multi-view crowd counting. In addition, in many works, the fusion of features from different stages and perspectives is considered a key method to improve the performance of the final task, but the prediction results of multiple perspective fusion are not utilized. Therefore, in order to reasonably and effectively utilize the feature information of multiple perspectives, the occlusion problem is overcome to a certain extent, and a semi-supervised strategy is adopted to reduce the dependence on labeled data, which improves the generalization ability of the method to a certain extent. In addition, a method based on uncertainty sorting is adopted to fuse the prediction results from multiple perspectives, which improves the crowd counting performance under the current method framework.

[0063] The training method of the people detection model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104 or placed on the cloud or other network servers.

[0064] Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices may include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Server 104 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In an exemplary embodiment, Figure 2 As shown in the figure, a training method for a people detection model is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate the process, including the following steps 202 to 210. Among them:

[0066] Step 202: Obtain unlabeled features and labeled features obtained by capturing a crowd scene from multiple perspectives.

[0067] Unlabeled features are features of unlabeled data, for which no corresponding labels exist. Labeled features are features extracted from labeled data, for which corresponding labels exist. Both unlabeled and labeled features are used to characterize people, such as head features, facial features, and body features.

[0068] For each viewpoint, a multi-array convolutional neural network (CNN) can be used to extract labeled or unlabeled features at different scales. Labeled or unlabeled features can be extracted using an MCNN, which uses three networks with different convolution kernel sizes to extract features from crowd images. These features are then fused through a 1×1 convolution. However, this multi-network architecture has many model parameters and high computational complexity, making real-time prediction difficult. Furthermore, multi-array networks cannot efficiently extract features from different heads and suffer from an inefficient branching structure.

[0069] CP-CNN can be used; it enhances the constraints of crowd density maps by leveraging global and local semantic information. Its network consists of three branches: one for global context, which classifies the entire image into features of varying density levels; another branch performs the same operation on local patches of the image to obtain local context features. Finally, the global and local context features are concatenated with the density map features along the channel dimension. This comprehensive consideration of global and local density information in the image enables the network to adaptively learn density-level features for any image.

[0070] Switch-CNN can be used; it also uses three subnetworks and a classification method, but differs in that it selects the appropriate subnetwork for processing image patches of different density levels, thereby improving prediction accuracy. During training, all networks are pre-trained using pre-training techniques, and then the classification network determines the input path for each image patch. This method is similar to MCNN in terms of the number of branches selected.

[0071] CSRNet can be used; it uses a single-column framework. It uses a pre-trained VGG-Net network and dilated convolutions to achieve better counting results. Dilated convolutions are better at capturing head edge information than non-dilated convolutions, thus forming head features.

[0072] IC-CNN can be used; IC-CNN uses a method of gradually refining from low resolution (LR) to high resolution (HR). This structure is clear and scalable, and is inspired by the field of image generation, where gradually generating density maps from low to high resolution is common.

[0073] In an optional embodiment, obtaining unlabeled and labeled features from a crowd scene captured from multiple viewpoints includes selecting VGG-Net as a feature extraction framework for a single-view image, performing feature extraction on the image data of the crowd scene from each viewpoint, and obtaining unlabeled and labeled features. The VGG-Net network model is chosen because, in multi-view crowd counting tasks, the same object may be located in different positions under different viewpoints, and the size of a crowd at the same position may also vary from viewpoint to viewpoint. Accordingly, VGG-Net, as a classic convolutional neural network, has a deep structure that can gradually extract increasingly rich semantic information through convolution and downsampling operations. As the network depth increases, the size of the feature maps extracted by VGG-Net gradually decreases, and higher-level semantics reflecting crowd distribution are extracted. Combining high-level semantics with low-level semantics can form more robust crowd features. This improves the robustness of both unlabeled and labeled features.

[0074] In an optional embodiment, projection features obtained by projecting unlabeled features onto labeled features are used for number of people prediction.

[0075] In the case of a fixed camera, the camera calibration parameters (intrinsic and extrinsic) are known, and the camera frames across views are synchronized. Therefore, given a set of multi-view images, a scene-level density map defined by a fixed region on the 3D scene ground plane can be predicted. Multi-view feature fusion consists of two steps: projection of single-view features onto the ground plane and fusion of the projected features.

[0076] Optionally, the single-view projection to ground plane step includes passing the feature map obtained by the previous single-view feature extraction and decoding module through a projection layer calculated based on camera calibration parameters, which is based on a spatial transformer network (STN). The feature map is then projected onto the scene ground plane. The parameters of the projection layer are determined by the changing scene and camera viewpoints, adapting to different scenes and the number of cameras, thereby being able to simultaneously process multi-view image features.

[0077] Optionally, the step of feature fusion after projection includes: projecting the extracted multiple single-view features from the image plane to a common scene plane through a spatial transformation network, and then performing feature fusion.

[0078] For example, assuming that the projection feature of view i is expressed as , the projection feature set of all camera views is . is a set of camera viewpoints, where k represents the number of cameras. The multi-camera viewpoint features will be concatenated (cat) together, and then the entire multi-view feature representation is finally obtained. The fusion feature of the first j views is ;

[0079] Among them, i {1,2,...j}. After the fused multi-view features are input into the multi-view decoder, the corresponding convolutional network layer further convolutions the features to obtain more advanced feature semantics. Here, it is assumed that the convolution feature Represents the features of the projection features after the fusion of the first i perspectives through the corresponding convolutional network layer, that is:

[0080]

[0081] Step 204 , performing a headcount uncertainty prediction based on the unlabeled features of each group of perspectives, and obtaining the headcount uncertainty corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives.

[0082] Uncertainty represents the uncertainty or level of uncertainty about the number of people in each viewpoint. This uncertainty can be related to the choice of model structure or the imprecision of parameter estimates. Optionally, uncertainty is negatively correlated with confidence. Uncertainty can be determined using a neural network model.

[0083] Alternatively, a density map can be used to assign a density value to each pixel in the feature map, representing the density of people in different areas. The total number of people is inferred from the density value of each pixel, and the sum of all the values ​​in the density map is the prediction result.

[0084] Optionally, the unlabeled features of each group of perspectives are fused separately to form fused unlabeled features; the fused unlabeled features may be features under a single perspective, and a crowd counting method under a single perspective may be used.

[0085] Step 206 : Compare the uncertainty of the number of people corresponding to each group of viewing angles to obtain a comparison result.

[0086] Alternatively, a self-supervised proxy task can be designed based on the fact that a wider scene should contain more individuals for the same area. The number of errors made on this proxy task can be considered as uncertainty. Alternatively, a proxy function based on matched patches can be used to better approximate the uncertainty of the number of people in the crowd counting task; this approach pays sufficient attention to detail while maintaining appropriate granularity.

[0087] Optionally, a confidence branch can be added to the classification prediction branch, outputting a confidence score for each input to distinguish between in-distribution and out-of-distribution samples. By learning a specific target confidence criterion directly from labeled data, more accurate confidence scores can be generated directly at test time. Alternatively, a multi-task credible pseudo-labeling learning (MTCP) framework can be constructed to guide the generation of credible density map pseudo-labels and mitigate the impact of noise.

[0088] Step 208: Determine the unlabeled loss based on the comparison result.

[0089] Optionally, in each single-view scenario, there may be a current unlabeled loss of any class, which can be adjusted by comparing the results to obtain the unlabeled loss used in step 210. For example, the unlabeled loss can be determined in a semi-supervised manner based on the spatial uncertainty of the regularized proxy task; this step focuses on utilizing crowd information in high-confidence areas through a spatial uncertainty-aware teacher-student framework. In the case where there are no corresponding annotations for features, the substitute task of the teacher model is used to estimate the spatial uncertainty map and simultaneously guide the feature learning of the main task (density regression) and the student model. This step also introduces a differential transformation layer to strengthen the spatial consistency regularization between the main task and the proxy task in the student model, thereby improving prediction accuracy and generating high-quality uncertainty maps.

[0090] Alternatively, a supervised uncertainty estimation strategy can be used to calibrate the uncertainty of the crowd counting model, and a per-block proxy function is designed to generate reliable pseudo-annotations. A learned region ranking framework is used for training using a region ranking rule in unlabeled crowd images. This region ranking rule ensures that the number of people in any block is always at least the number of people in its sub-blocks. This semi-supervised approach effectively addresses the challenges of limited existing datasets and difficult annotation.

[0091] Semi-supervised crowd counting can be addressed from a feature learning perspective. Correspondingly, a set of binary segmentation tasks related to the original density map regression task can be derived as alternative prediction targets, and shared features of the crowd can be learned from labeled and unlabeled data. Pseudo-annotations are generated by using an auxiliary target predictor trained with an unlabeled loss, and the potential constraints between these pseudo-annotations assist in training.

[0092] In an optional embodiment, comparing the headcount uncertainty corresponding to each set of perspectives to obtain a comparison result includes: comparing the headcount uncertainty comparison results between each set of perspectives in descending order of the number of perspectives to obtain each comparison result in sequence. Correspondingly, determining the unlabeled loss based on the comparison results includes: determining whether the current unlabeled loss should be adjusted based on each comparison result obtained in sequence; if so, adjusting the current unlabeled loss based on each comparison result obtained, until all comparison results are used to determine the unlabeled loss, thereby obtaining the unlabeled loss used for model training.

[0093] Therefore, after obtaining the uncertainty of the multi-view fusion prediction by inputting different numbers of camera views, the unlabeled loss can be determined by comparing and ranking the results. The core idea is to introduce constraints to the multi-view crowd counting model. Optionally, the following rules can be enforced during training: more camera views should cover more people and better handle occlusions, so the uncertainty of the fusion prediction should be lower. In other words, the greater the number of views, the lower the corresponding crowd counting uncertainty, and the two are inversely proportional.

[0094] Step 210 : Training a headcount prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features.

[0095] The labeled loss is the model loss determined based on the labeled features. Optionally, the labeled loss can be a dynamic loss function or a specific loss value.

[0096] In an optional embodiment, the number of people prediction model includes an image encoder, a multi-view decoder trained with main task loss and auxiliary task loss, and an uncertain network trained based on unlabeled ranking loss and labeled uncertainty loss; wherein, the labeled losses corresponding to the labeled features include main task loss, auxiliary task loss and uncertainty loss; and the unlabeled loss is for unlabeled data, and the three networks can be trained through the three losses of labeled features and the unlabeled loss for labeled / unlabeled data respectively.

[0097] For people prediction models, a multi-view counting framework and the CityStreet dataset can be used to fuse multiple camera views to predict 3D scene-level density maps. Multi-view multi-scale (MVMS) counting can also be performed. In this case, a CNN is used to fuse camera views to predict 2D scene density maps, and 3D feature fusion is used to handle the complex multi-view counting task. Optionally, a height-estimation-based attention module can be used to project 2D pixels onto 3D voxels to ensure consistency between the 3D prediction and the 2D ground truth.

[0098] Optionally, the crowd prediction model may include: a cross-view cross-scene (CVCS) model; dynamically selecting and fusing multiple views through an attention mechanism, while using a noisy view regularization method to handle camera calibration errors or feature errors. Among them, the cross-view cross-scene dataset contains multi-scene, multi-person, and multi-view data. Optionally, by adding inter-view correlation, the geometric and semantic information can be decomposed into a general fusion component and a view-specific affine transformation component, effectively reducing parameter redundancy. Optionally, a multi-view learning model based on a graph can be used to perform contextual reasoning through a graph interaction space and a graph communication layer (GCL) to handle information fusion within and across viewpoints.

[0099] Alternatively, although the background and foreground regions come from the same image, their appearances are quite different, and the foreground region can be found in regions of similar density in other crowd images. Based on this principle, a semi-supervised counting step guided by a density proxy can be used. This step, in turn, constructs a density proxy to align foreground region features with corresponding density subclass features, while maximizing the difference between foreground and background features to further ensure the accuracy of the crowd prediction model.

[0100] In the above-mentioned training method for a crowd detection model, unlabeled and labeled features are acquired from multiple viewpoints of a crowd scene. The effects of occlusion can be mitigated across multiple viewpoints, helping to ensure accurate crowd recognition. The method then predicts the number of people based on the unlabeled features for each viewpoint, obtaining the number of people corresponding to each viewpoint. This method then fuses predictions across different viewpoints, allowing for separate training of unlabeled and labeled data from different viewpoints. This creates a semi-supervised training model, significantly reducing reliance on labeled data and eliminating the need for additional labeling. This allows unlabeled features to be trained together with tasks such as population density regression, improving crowd counting performance. The number of people corresponding to each viewpoint is then compared to obtain an uncertainty comparison result. This comparison result measures the reliability of the different multi-view fusion processes and reflects the presence of errors in the corresponding prediction processes, thereby determining the order of uncertainty results for each viewpoint. This comparison result is then used to gradually adjust the corresponding unlabeled loss, determining the unlabeled loss based on a ranking principle. Finally, the unlabeled loss and the labeled loss corresponding to the labeled features are simultaneously applied to the training process of the crowd prediction model. Therefore, multi-view fusion prediction is performed on different numbers of view features, and then the uncertainty results of the prediction results are sorted to introduce constraints for the multi-view crowd counting model. Based on the relationship between the number of viewpoints and the uncertainty of the number of people, semi-supervised training is performed. The unlabeled features that do not require additional labeling are combined with the corresponding tasks of the labeled features for training, forming a semi-supervised model training method, which can improve the crowd counting performance.

[0101] In an exemplary embodiment, the uncertainty of the number of people is predicted based on the unlabeled features of each group of perspectives to obtain the uncertainty of the number of people corresponding to each group of perspectives, including: predicting the uncertainty of the number of people in the crowd scene based on the unlabeled features of the first i perspectives to obtain the uncertainty of the number of people in the i-th group of perspectives; predicting the uncertainty of the number of people in the crowd scene based on the unlabeled features of the first j perspectives to obtain the uncertainty of the number of people in the j-th group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0102] Optionally, the unlabeled features of the first i perspectives are concatenated to obtain the i-th concatenated feature; the i-th concatenated feature is input into the neural network model to obtain the uncertainty of the number of people in the i-th group of perspectives. Thus, the i-th concatenated feature is formed by concatenating all the features of the first i perspectives.

[0103] Alternatively, the unlabeled features of the first j perspectives are concatenated to obtain the jth concatenated feature; this jth concatenated feature is input into the neural network model to obtain the uncertainty of the number of people in the jth perspective. Thus, the jth concatenated feature is formed by concatenating all the features of the first j perspectives.

[0104] The overlap between the first i and j perspectives is equivalent to controlling the perspective variable. Since the uncertainty corresponding to more perspectives should be smaller, that is, when i is less than j, the uncertainty of the number of people in the jth group of perspectives is based on the unlabeled features predicted from more perspectives, and the uncertainty reflects the accuracy of the prediction at a certain pixel point. Therefore, under theoretical rules, the uncertainty corresponding to the jth group of perspectives should be smaller than the uncertainty corresponding to the ith group of perspectives. Correspondingly, because the perspective variable is controlled, once the uncertainty corresponding to the number of people in the jth group of perspectives is greater than the uncertainty corresponding to the number of people in the ith group of perspectives, it is easier to find that the model needs to be adjusted.

[0105] In this embodiment, each group of perspectives has overlapping unlabeled features and prediction results, and are arranged in a corresponding order. Since the uncertainty can reflect the accuracy of the prediction of a certain pixel point, the uncertainty corresponding to the number of people in each group of perspectives should have a unidirectional change law. Therefore, once the uncertainty does not conform to the unidirectional change law, it can be reflected through the comparison result corresponding to the uncertainty, thereby improving the accuracy of the unlabeled loss.

[0106] In an exemplary embodiment, Figure 3 As shown, the uncertainty of the number of people corresponding to each group of viewpoints is compared to obtain a comparison result, including step 302; correspondingly, the unlabeled loss is determined according to the comparison result, including step 304; wherein:

[0107] Step 302 : Compare the uncertainty of the number of people corresponding to each group of perspectives in sequence according to the number of perspectives, and obtain each comparison result obtained in sequence.

[0108] Optionally, the uncertainty of the number of people corresponding to each group of perspectives can be compared in sequence according to the number of perspectives contained in each group of perspectives and in the order of the number of perspectives from large to small or from small to large, to obtain each comparison result obtained in sequence.

[0109] Optionally, the uncertainty corresponding to the number of people in the i-th group of perspectives is compared with the uncertainty corresponding to the number of people in the j-th group of perspectives to obtain a comparison result between the i-th group of perspectives and the j-th group of perspectives. Thus, the comparison result between the i-th group of perspectives and the j-th group of perspectives forms each comparison result obtained in sequence.

[0110] Step 304 : determining the unlabeled loss based on each comparison result in turn.

[0111] Optionally, based on each comparison result, it is determined whether the current unlabeled loss needs to be adjusted; if adjustment is required, the adjustment is performed based on each comparison result; if adjustment is not required, it is determined whether the current unlabeled loss needs to be adjusted based on the next comparison result, until each comparison result is determined.

[0112] In this embodiment, by comparing the view angles in order, the comparison results in the corresponding iterative process can be obtained in an orderly manner, and the unlabeled loss is continuously adjusted, so that the unlabeled loss is gradually improved, and the required accuracy is controllable.

[0113] In an exemplary embodiment, Figure 4 As shown, the uncertainty of the number of people corresponding to each group of perspectives includes the uncertainty of the number of people in the i-th group of perspectives and the uncertainty of the number of people in the j-th group of perspectives, i is less than j, and i and j are both positive integers. The unannotated loss is determined based on the comparison results, including:

[0114] Step 402: If the uncertainty of the number of people in the j-th group of views is greater than the uncertainty of the number of people in the i-th group of views, then adjust the unlabeled loss according to the difference in the uncertainty of the number of people between the j-th group of views and the i-th group of views.

[0115] The uncertainty difference may be the difference between the uncertainty of the number of people in the j-th group of viewpoints and the uncertainty of the number of people in the i-th group of viewpoints; the uncertainty difference may be the difference or the absolute value of the difference.

[0116] In an optional embodiment, the unlabeled loss is adjusted according to the uncertainty difference in the number of people between the j-th group of perspectives and the i-th group of perspectives, including: performing a difference calculation based on the uncertainty in the number of people in the j-th group of perspectives and the uncertainty in the number of people in the i-th group of perspectives to obtain a current uncertainty difference; determining a penalty value that is positively correlated with the current uncertainty difference; and adjusting the current unlabeled loss according to the penalty value to obtain an adjusted unlabeled loss.

[0117] The penalty value is used to adjust the current unlabeled loss. A larger penalty value results in a larger parameter gradient for updating the network, and a larger adjustment to the current unlabeled loss. Optionally, the penalty value can be a regularization term adjusted for the uncertainty difference, or the uncertainty difference itself. The penalty value can be the current uncertainty difference, or it can be an adjustment for the uncertainty difference.

[0118] Optionally, the current unlabeled loss can be the unlabeled loss at initialization or the unlabeled loss after the last adjustment. The current unlabeled loss and the adjusted unlabeled loss exist for each uncertainty difference, forming multiple adjustments in sequence.

[0119] Therefore, the current uncertainty difference is positively correlated with the penalty value. The larger the current uncertainty difference, the larger the penalty value, which makes the parameter gradient of the updated network larger and the adjustment of the current unlabeled loss larger, so as to achieve gradual adjustment of the unlabeled loss.

[0120] Step 404: If the uncertainty of the number of people in the j-th group of viewpoints is less than the uncertainty of the number of people in the i-th group of viewpoints, then no labeling loss is maintained.

[0121] Maintaining the unlabeled loss means that the uncertainty difference at this time is not used to adjust the unlabeled loss. Optionally, the above comparison results are each comparison result obtained sequentially; thus, the unlabeled loss is determined sequentially based on each comparison result. In this case, i and j are different values ​​each time, and during each execution, one of step 402 and step 404 is selected.

[0122] In this embodiment, since i is less than j, the number of people in the jth viewpoint is predicted based on features from more viewpoints. Therefore, when the uncertainty of the number of people in the jth viewpoint is less than the uncertainty of the number of people in the ith viewpoint, this is normal and no adjustment is required for the corresponding unannotated loss. Conversely, when the uncertainty of the number of people in the jth viewpoint is less than the uncertainty of the number of people in the ith viewpoint, the uncertainty model is abnormal and adjustments can be made directly based on the uncertainty difference to improve efficiency.

[0123] In an alternative embodiment, Figure 5As shown, before training the number of people prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes steps 502 to 510; correspondingly, training the number of people prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features includes step 512, wherein:

[0124] Step 502 : Determine the nth labeled predicted number of people and the uncertainty of the nth labeled predicted number of people based on the labeled features of the first n perspectives.

[0125] The labeled features of the first n views, including the labeled features of at least two views. The standardized features of the first n views can be fused multiple times. The fused features obtained from each fusion are used for feature extraction to predict each labeled number of people and the corresponding uncertainty. The corresponding auxiliary task loss and labeled uncertainty loss are also obtained each time.

[0126] The labeled predicted number of people and the uncertainty of the labeled predicted number of people are the predictions obtained by inputting the labeled features into different models. The labeled predicted number of people is the predicted number of people, while the uncertainty of the labeled predicted number of people is the uncertainty of the prediction based on the labeled features. Optionally, after the labeled features are input into the uncertainty network, the uncertainty network outputs the uncertainty of the labeled predicted number of people.

[0127] Step 504 : Determine the nth auxiliary task loss based on the mean square error between the nth labeled predicted number of people and the reference number of people.

[0128] The reference headcount is the labeled value corresponding to the labeled feature; this value represents the standard headcount value or the actual headcount value. Because the labeled features are from multiple perspectives, the reference headcount also refers to the reference headcount from multiple perspectives.

[0129] The mean squared error (MSE) is used to evaluate the difference between the model's predictions and the true values. Correspondingly, the squared value of each point is obtained by taking the difference between the labeled predicted number of people and the reference number of people at each point and averaging the squared values ​​across the points in the density map to obtain the mean squared error used to determine the loss of the auxiliary task.

[0130] The auxiliary task loss is used to indicate the degree to which the labeled predicted number of people deviates from the reference number of people. Optionally, the auxiliary task loss can be a loss function or a loss value calculated by the loss function.

[0131] Step 506 , determining the nth reference uncertainty based on the absolute difference between the nth labeled predicted number of people and the reference number of people, and determining the nth labeled uncertainty loss based on the mean square error between the uncertainty of the nth labeled predicted number of people and the nth reference uncertainty.

[0132] The absolute difference is the absolute value of the difference between the labeled forecast and the reference number. Since the labeled forecast is a prediction, the difference between the predicted and reference numbers is difficult to predict in advance. Therefore, using the absolute difference accurately measures the corresponding uncertainty.

[0133] Reference uncertainty is a pseudo-annotation, derived from converting the predicted result with the reference number of people, and represents theoretical uncertainty. Alternatively, the absolute difference can be used as the reference uncertainty, or the processed absolute difference can be used as the reference uncertainty.

[0134] The annotated uncertainty loss is the corresponding uncertainty loss of the standardized characteristics.

[0135] The mean squared error (MSE) is used to assess the difference between the model's predicted value and the true value. Correspondingly, the squared uncertainty value for each point is obtained by squaring the difference between the uncertainty of the annotated predicted number of people and the reference uncertainty at each point. The squared uncertainty values ​​for each point are then averaged across the points in the density map to obtain the mean squared error, which is used to calculate the annotated uncertainty loss.

[0136] Optionally, based on the labeled features of the first n perspectives, the uncertainty corresponding to the nth labeled predicted number of people and the nth labeled predicted number of people is determined; when the labeled predicted number of people is determined based on the labeled features of the first n perspectives, the reference number of people is the reference number of people under the first n perspectives; based on the predicted number of people under the first n perspectives and the reference number of people, the absolute difference between the two is obtained; and based on the absolute difference and the uncertainty corresponding to the nth labeled predicted number of people, the nth labeled uncertainty loss is determined. Thus, the labeled predicted number of people and the corresponding uncertainty are determined for each set of perspectives, resulting in each set of labeled uncertainty losses.

[0137] Step 508 : Determine the labeled loss based on the (k-1)th n-th auxiliary task loss and the (k-1)th n-th labeled uncertainty loss; where n and k are integers, n is greater than 1, n is less than or equal to (k-1), and k is the number of viewpoints.

[0138] In an optional embodiment, the labeled loss is determined based on the (k-1)th n-th auxiliary task losses and the (k-1)th n-th labeled uncertainty losses, including: averaging the (k-1)th n-th auxiliary task losses to obtain the auxiliary task mean loss; averaging the (k-1)th n-th labeled uncertainty losses to obtain the labeled uncertainty mean loss; weighting the auxiliary task mean loss and the labeled uncertainty mean loss according to preset weights to obtain the weighted auxiliary task loss and the weighted labeled uncertainty loss; obtaining the labeled loss based on the weighted auxiliary task loss and the weighted labeled uncertainty loss.

[0139] In an optional embodiment, obtaining the labeled loss based on the weighted auxiliary task loss and the weighted labeled uncertainty loss includes combining the main model loss, the weighted auxiliary task loss, and the weighted labeled uncertainty loss to obtain the labeled loss. The main model loss is trained based on the labeled features, and the main model loss is different from the auxiliary task loss, and the main model loss is also different from the labeled uncertainty loss.

[0140] Step 510 , determining the main task loss based on the mean square error between the k-th labeled predicted number of people and the reference number of people; the k-th labeled predicted number of people includes the labeled predicted number of people corresponding to k perspectives.

[0141] The labeled predicted number of people corresponding to the kth view is the number of people predicted based on all views; since every view is used, the maximum number of views is reached. Although theoretically, training the headcount prediction model using the main task loss is most effective, training using the unlabeled loss, the labeled loss, and the main task loss can achieve higher accuracy.

[0142] Optionally, the kth labeled predicted number of people may be determined based on a number prediction model in training.

[0143] Step 512: Train the number of people prediction model based on the unlabeled loss, the labeled loss, and the main task loss.

[0144] In an optional embodiment, the number of people prediction model is trained based on the unlabeled loss, the labeled loss and the main task loss, including: determining the total model loss based on the sum of the unlabeled loss, the labeled loss and the main task loss; and training the number of people prediction model based on the total model loss.

[0145] Optionally, the unlabeled loss, the labeled loss and the main task loss may be weightedly fused, and the weight of the main task loss is greater than the weight of the unlabeled loss, and the weight of the main task loss is greater than the weight of the labeled loss.

[0146] In this embodiment, the auxiliary task loss is determined based on the mean square error between the labeled predicted number of people and the reference number of people, forming an adjustment method for model training, which helps to improve the accuracy of model prediction; and the mean square error calculation process for determining the auxiliary task loss includes the absolute difference calculation process, and the absolute difference is reused to determine the reference uncertainty, which can ensure processing efficiency. Furthermore, based on the mean square error between the uncertainty of the labeled predicted number of people and the reference uncertainty, the labeled uncertainty loss is determined, so that the uncertainty of the labeled predicted number of people is converted into a relative value for measurement. Furthermore, based on at least the auxiliary task loss and the labeled uncertainty loss, the labeled loss is determined, so that the number prediction model is trained through this labeled loss and the non-standard loss in conjunction with the main task loss, so that the trained number prediction model has more accurate prediction results.

[0147] In an optional embodiment, the method further includes: obtaining a plane annotation map of the crowd scene; the plane annotation map includes multiple pixel points, and the pixel value of each pixel point represents the number of people; through a Gaussian kernel function, the pixel value of each pixel point is replaced by the weighted pixel average of the adjacent pixel points of each pixel point to obtain a number marking map; the pixel values ​​in the number marking map represent the reference number of people.

[0148] The plane annotation map can be a density map, where the pixel value of each pixel represents the density value of each point to reflect the number of people at each point.

[0149] Neighboring pixels are pixels whose distance from each pixel is less than a preset value. For example, if there are no other pixels between each pixel and its neighboring pixels, the neighboring pixels are the pixels closest to each pixel. The reference number of people is a parameter used during model training and represents the number of people.

[0150] Optionally, a Gaussian kernel function is used to replace the pixel value of the annotation center point with the weighted average of the pixel values ​​of the surrounding points to ensure that the sum of the weights of the surrounding pixels is 1.

[0151] In this embodiment, the pixel values ​​in the plane annotation map represent the number of people, facilitating the acquisition of corresponding data via density maps. Using a Gaussian kernel function, the pixel value at the annotation center of the plane annotation map is replaced with the weighted average of the neighboring pixels of the center point, resulting in a head count map. This approach maintains the total head count while more accurately and faithfully reflecting the spatial location of each head, resulting in a more accurate reference head count.

[0152] It's important to understand that crowd counting scenarios present issues like crowd occlusion and varying scales. While existing multi-view counting frameworks effectively address these issues, their introduction presents a new challenge: limited and difficult to annotate datasets for multi-view crowd counting. To address this issue, after thorough research, we discovered that no relevant multi-view semi-supervised work exists in the crowd counting field. Therefore, faced with limited annotated data, we proposed an innovative semi-supervised multi-view crowd counting method. This method utilizes multi-view feature fusion, predicts uncertainty based on the fused features, and ranks the uncertainty to fully exploit unlabeled data information, helping the model better learn crowd characteristics.

[0153] Because more camera perspectives can cover a wider range of people and provide richer information about the people in the scene. Considering that when a certain area is covered by all the involved perspectives, the prediction of the number of people in the area should be certain regardless of whether it is observed from a few perspectives or multiple perspectives.

[0154] Each point in the density map reflects the probability of a person at that location. The sum of all probabilities in a region provides the estimated number of people in that area. If there are people around a specific area, their coordinates, after Gaussian blurring, are also added to that area. The more people around an area, the higher the cumulative probability. Gaussian blurring facilitates model training; otherwise, simply using the original point map to represent the ground-truth population would be very difficult to train.

[0155] At the same time, after the introduction of uncertainty in the field of semi-supervised crowd counting, the training can be assisted by introducing the perspective fusion prediction uncertainty ranking. Under the N perspectives of the same crowd scene, the corresponding numbers are numbered from 1 to N. For a certain area in the scene, the real density map under the perspective i is used as the reference number. , the corresponding predicted density value is If the predicted density value of 1 viewing angle feature is , the uncertainty under the definition 1 perspective in this paper should be , Refers to the Ground-Truth corresponding to perspective 1 (the density map of the original point map after Gaussian blur). After the introduction of the new perspective 2, if the predicted density value of the perspective combination [1,2] is , then the uncertainty from the [1,2] perspective should be , It refers to the original point map that can be seen at the same time from the [1,2] perspective, and the density map after Gaussian blurring.

[0156] The uncertainty sorting of perspective fusion requires the uncertainty under 1 perspective Should be greater than the uncertainty under the [1, 2] perspective combination In this way, we can clearly know that when the introduced perspective combination is [1,..., N], it should have the lowest uncertainty and the highest confidence. In the prediction process, the uncertainty diagrams predicted by different viewing angles should also satisfy the above potential rules. As a punishment for training, when the above When , the penalty is positive and the larger the gap, the greater the penalty. When the penalty is non-positive, the prediction result complies with the underlying sorting rules and no penalty is imposed on the network. In general, uncertainty reflects the accuracy of the prediction at a certain pixel. The more points seen from different angles, the higher the accuracy and the lower the uncertainty. The uncertainty maps (normalized) of different numbers are fused under three data sets as shown in the following figure. Figure 6 As shown, the darker the color of the point, the higher the uncertainty of the position, and the lighter the color, the lower the uncertainty of the position. It can be clearly seen that the lower the uncertainty predicted by integrating more perspective features, the lighter the color.

[0157] In a specific embodiment, Figure 7 As shown, in order to implement a semi-supervised multi-view crowd counting method based on the uncertainty of the number of people, the population prediction model of this embodiment includes at least three modules: a single-view feature extractor, a multi-view decoder, and an uncertainty network involving uncertainty. Each view image is extracted by the single-view feature extractor to obtain corresponding image features, which are then projected onto the scene reference plane. The fused feature maps of each view are fed into the corresponding network module; the multi-view fused projection features of the first 1 to k views are written as f1, f2, ..., f k , respectively represent the feature expression of the splicing fusion of the projection features of the first 1, first 2 to first k perspectives, then f1 to f k The features are input into the multi-view decoder.

[0158] In the multi-view decoder, convolution features are obtained, and the processing of convolution features is divided into two parts:

[0159] 1) If Figure 7 As shown in (a), if the input is labeled data, the multi-view decoder will output f1 to f k The corresponding number of people prediction density map, where the prediction results corresponding to the first 1 to k-1 perspectives are called auxiliary predictions, and the predictions of all k perspectives are called main predictions. The MSE loss between these number of people prediction density maps and the corresponding reference uncertainty (ReferenceU-GT, GT) is calculated and divided into the main task loss and auxiliary task loss ;like Figure 7 The intermediate features of the multi-view decoder are also input to the uncertainty decoder, and the uncertainty prediction map of the corresponding different fusion views is output. The reference GT is the absolute difference between the number of people predicted at each fusion view output by the multi-view decoder and the GT. The MSE loss is calculated between the uncertainty prediction map and the GT, which is called the uncertainty loss. .

[0160] 2) If Figure 7 As shown in (b), if the input is unlabeled data, only the intermediate features of the multi-view decoder are input to the uncertainty decoder to obtain the prediction uncertainty of each fusion view, and the ranking loss between them is calculated, which is called the unlabeled ranking loss .

[0161] Furthermore, given a set of multi-view images, we aim to predict a scene-level density map defined by a fixed region on the ground plane of the 3D scene to determine the number of people. The scene-level density map is as follows: Figure 8 shown.

[0162] Specifically, for the above 1), the convolutional features of the labeled data are labeled features. The labeled features will be input into the multi-view decoder to obtain the prediction results corresponding to each fused view, as shown in the following formula:

[0163]

[0164] As the prediction result, the corresponding GT is as described above for the fusion of features, accordingly, there is , represents the fusion GT labels of the first n views, thus forming the annotation. For example, Figure 9 As shown in Figure 2, on these three datasets, as the number of fused perspectives increases, the corresponding GT images will also show more people.

[0165] Prediction results and corresponding Find the MSE loss between , which is the auxiliary task loss, denoted as , as shown below:

[0166]

[0167] Where k is the number of viewing angles corresponding to the camera.

[0168] On the other hand, the prediction results of all k views are P k , its corresponding GT is G k , the MSE loss between the two is called the main task loss, recorded as , as shown below:

[0169]

[0170] It is worth noting that the prediction result will be the only output during the testing process.

[0171] At the same time, the prediction results and corresponding The absolute difference between them is taken as the reference uncertainty of the uncertainty decoder (Reference U-GT, GT), that is, Figure 7 In addition, the convolutional features are also input into the uncertainty network to obtain the uncertainty corresponding to each fusion perspective (Uncertainty Prediction, ), and Calculate the MSE loss between, which is the uncertainty loss of the labeled data, recorded as , as shown below:

[0172]

[0173] In summary, the total loss of labeled data can be summarized as follows:

[0174]

[0175] in, and is a hyperparameter that adjusts the weight of different loss items. It is the main task loss, is the auxiliary task loss. That is, the predictions of all k perspectives are called main predictions, and the MSE loss between these number prediction density maps and the corresponding reference uncertainty (Reference U-GT, GT) is calculated, which is divided into the main task loss and auxiliary task loss .

[0176] Regarding the above 2), the convolutional features of unlabeled data will not be input into the multi-view decoder because they do not have corresponding GT annotations. Instead, they will be directly input into the uncertainty network, and then the uncertainty graph prediction results corresponding to each fusion view will be output. At this time, there is no corresponding reference number, but the supervision rules are constructed with the help of its potential characteristics to obtain no label loss, that is, no label ranking loss. .

[0177] Based on this rule, if , a corresponding penalty should be imposed on the uncertain network to adjust the network parameters, and the specific penalty value is , and the greater the penalty value, the greater the parameter gradient for updating the network. When , , at this time the penalty is non-positive, that is, the prediction result conforms to the potential sorting rule and there is no need to impose a penalty on the network. That is to say, the network prediction meets the expectation and there is no need for parameter update. At this time, the gradient of the backward propagation of the uncertain network is 0, and the definition formula of the specific penalty amount C is as follows:

[0178]

[0179] By sorting the uncertain graphs with different numbers of fusion perspectives, the entire unlabeled loss can be obtained as shown in the following formula:

[0180]

[0181] Among them, contains k(k - 1) / 2 terms and is calculated by using double summation. For example: assuming there are a total of 3 perspectives, the prediction uncertainty UP3 of fusing 3 perspectives is less than the prediction uncertainty UP2 of fusing 2 perspectives, and at the same time is less than the prediction uncertainty UP1 of 1 perspective, that is, UP3 < UP2, UP3 < UP1, and at the same time UP2 < UP1. There are 3 terms {UP3 < UP2, UP3 < UP1, UP2 < UP1}, and so on, up to the unlabeled loss function of k perspectives.

[0182] In summary, the loss function of the entire model during the training process is:

[0183]

[0184] Among them, is the hyperparameter of the unlabeled loss, is the total loss of the labeled data.

[0185] Based on this, a semi-supervised method is applied to make full use of the unlabeled data as much as possible, greatly alleviating the dependence on the labeled data, especially in the case where annotation is difficult in a large scene. At the same time, the fusion prediction results of multiple perspectives and the uncertain predictions obtained through the uncertainty network are used, and the loss function obtained under a reasonable sorting rule helps to train the model and improves the performance of the crowd counting model.

[0186] Each network module of this embodiment, including the multi-perspective decoder and the uncertainty decoder, can adopt different convolution methods or use different activation functions in the network layer.

[0187] Various experiments demonstrate that this method achieves the best performance on the CVCS and PETS2009 datasets. By introducing a multi-view fusion ranking-constrained semi-supervised approach, the model significantly reduces the need for labeled data and achieves counting performance that matches or even exceeds that of some fully supervised approaches, demonstrating its superiority in reducing the need for labeled data. Furthermore, this method fully exploits the multi-view characteristics and integrates them into the counting model to further leverage the potential features of unlabeled multi-view data. This will open up new avenues for subsequent multi-view counting work.

[0188] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0189] Based on the same inventive concept, the present application also provides a training device for a people detection model for implementing the aforementioned training method for a people detection model. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the training device for one or more people detection models provided below can be found in the above-mentioned limitations on the training method for a people detection model, and will not be repeated here.

[0190] In an exemplary embodiment, Figure 10 As shown, a training device for a people detection model is provided, comprising:

[0191] An acquisition module 1002 is used to acquire unlabeled features and labeled features obtained by capturing a crowd scene from multiple perspectives;

[0192] Prediction module 1004, configured to predict the uncertainty of the number of people based on the unlabeled features of each group of perspectives, and obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0193] A comparison module 1006 is configured to compare the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result;

[0194] An adjustment module 1008 is configured to determine an unlabeled loss based on the comparison result;

[0195] The training module 1010 is configured to train a headcount prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features.

[0196] In one embodiment, the prediction module 1004 is configured to:

[0197] Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives;

[0198] Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0199] In one embodiment, the comparison module 1006 is configured to:

[0200] Comparing the uncertainty of the number of people corresponding to each group of perspectives in sequence according to the number of perspectives, and obtaining each comparison result obtained in sequence;

[0201] Correspondingly, the adjustment module 1008 is used to:

[0202] Based on each comparison result in turn, an unlabeled loss is determined.

[0203] In one embodiment, the number of people in each viewing group includes the number of people in the i-th viewing group and the number of people in the j-th viewing group, i is less than j, and both i and j are positive integers;

[0204] The adjustment module 1008 is configured to:

[0205] If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, then adjusting the unannotated loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints;

[0206] If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

[0207] In one embodiment, the adjustment module 1008 is configured to:

[0208] Performing a difference calculation based on the uncertainty of the number of people in the j-th group of viewpoints and the uncertainty of the number of people in the i-th group of viewpoints to obtain a current uncertainty difference;

[0209] Determining a penalty value that is positively correlated with the current uncertainty difference;

[0210] The current unlabeled loss is adjusted according to the penalty value to obtain an adjusted unlabeled loss.

[0211] In one embodiment, the training module 1010 is configured to:

[0212] Determine the nth labeled predicted number of people and the uncertainty of the labeled predicted number of people based on the labeled features of the first n perspectives;

[0213] Determining the nth auxiliary task loss based on the mean square error between the nth labeled predicted number of people and the reference number of people;

[0214] determining an nth reference uncertainty based on an absolute difference between the nth annotated predicted number of people and the reference number of people, and determining an nth annotated uncertainty loss based on a mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty;

[0215] Determining the labeled loss based on (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, n is greater than 1, n is less than or equal to (k-1), and k is the number of the view angles;

[0216] Determining the main task loss based on a mean square error between the k-th labeled predicted number of people and the reference number of people; the k-th labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives;

[0217] A number prediction model is trained according to the unlabeled loss, the labeled loss, and the main task loss.

[0218] In one embodiment, the acquisition module 1002 is configured to:

[0219] Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people;

[0220] The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number of people mark map; the pixel values ​​in the number of people mark map represent the reference number of people.

[0221] Each module in the aforementioned people detection model training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0222] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a people detection model is implemented.

[0223] Those skilled in the art will understand that Figure 11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0224] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0225] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0226] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0228] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0229] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0230] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A training method for a people detection model, characterized in that: The method comprises: Obtain unlabeled features and labeled features from crowd scenes captured from multiple perspectives; Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives; Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers; each group of perspectives contains a different number of perspectives; Comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result; determining the unmarked loss according to the comparison result; A headcount prediction model is trained according to the unlabeled loss and the labeled loss corresponding to the labeled features.

2. The method according to claim 1, characterized in that The comparing the uncertainty of the number of people corresponding to each set of perspectives to obtain a comparison result includes: Comparing the uncertainty of the number of people corresponding to each group of perspectives in sequence according to the number of perspectives, and obtaining each comparison result obtained in sequence; Determining the unlabeled loss according to the comparison result includes: Based on each comparison result in turn, an unlabeled loss is determined.

3. The method according to claim 1, characterized in that The uncertainty of the number of people corresponding to each group of perspectives includes the uncertainty of the number of people in the i-th group of perspectives and the uncertainty of the number of people in the j-th group of perspectives, where i is less than j, and both i and j are positive integers; Determining the unlabeled loss according to the comparison result includes: If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, then adjusting the unannotated loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints; If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

4. The method according to claim 3, characterized in that The adjusting the unlabeled loss according to the uncertainty difference of the number of people between the j-th group of viewing angles and the i-th group of viewing angles includes: Performing a difference calculation based on the uncertainty of the number of people in the j-th group of viewpoints and the uncertainty of the number of people in the i-th group of viewpoints to obtain a current uncertainty difference; Determining a penalty value that is positively correlated with the current uncertainty difference; The current unlabeled loss is adjusted according to the penalty value to obtain an adjusted unlabeled loss.

5. The method according to claim 1, wherein Before training the number of people prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes: Determine the nth labeled predicted number of people and the uncertainty of the labeled predicted number of people based on the labeled features of the first n perspectives; Determining the nth auxiliary task loss based on the mean square error between the nth labeled predicted number of people and the reference number of people; determining an nth reference uncertainty based on an absolute difference between the nth annotated predicted number of people and the reference number of people, and determining an nth annotated uncertainty loss based on a mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty; Determining the labeled loss based on (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, n is greater than 1, n is less than or equal to (k-1), and k is the number of the view angles; Determining the main task loss based on a mean square error between the kth labeled predicted number of people and the reference number of people; the kth labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives; The training of the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features includes: A number prediction model is trained according to the unlabeled loss, the labeled loss, and the main task loss.

6. The method according to claim 5, characterized in that The method further comprises: Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people; The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number of people mark map; the pixel values ​​in the number of people mark map represent the reference number of people.

7. A training device for a people detection model, characterized in that: The device comprises: An acquisition module is used to obtain unlabeled features and labeled features obtained by capturing crowd scenes from multiple perspectives; A prediction module is configured to predict the uncertainty of the number of people in the crowd scene based on the unlabeled features of the first i perspectives, thereby obtaining the uncertainty of the number of people in the i-th perspective group; and to predict the uncertainty of the number of people in the crowd scene based on the unlabeled features of the first j perspectives, thereby obtaining the uncertainty of the number of people in the j-th perspective group; wherein i is less than j, and both i and j are positive integers; and each perspective group contains a different number of perspectives; A comparison module, configured to compare the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result; An adjustment module, configured to determine an unlabeled loss according to the comparison result; A training module is used to train a headcount prediction model based on the unlabeled loss and the labeled loss corresponding to the labeled features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images