Training method and device of people number detection model, computer equipment and storage medium

Abstract

The invention relates to a people number detection model training method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring unlabeled features and labeled features obtained by collecting a crowd scene from multiple visual angles; predicting the uncertainty of the number of people based on the unmarked features of each group of view angles to obtain the uncertainty of the number of people corresponding to each group of view angles; the numbers of visual angles contained in each group of visual angles are different; comparing the uncertainty of the number of people corresponding to each group of visual angles to obtain a comparison result; determining no label loss according to the comparison result; and training a people number prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features. By adopting the method, the dependence of annotation can be relieved, and the accuracy of people number prediction is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method, apparatus, computer equipment, computer-readable storage medium and computer program product for a people detection model. Background Art

[0002] With the development of society, more and more scenarios require crowd computing to provide data support, optimize resource allocation, and promote corresponding development in areas such as security monitoring and business decision-making.

[0003] The main models used in traditional crowd detection models include sliding window detectors based on overall and local detection, support vector machines, and random forest models. However, these methods are suitable for sparse crowds, but they are not effective when the crowd density increases and occlusion is serious. Based on these models, regression methods are introduced to first extract low-level features (such as shape, texture, etc.), and then predict the number of people through regression models. However, this method is difficult to capture high-level semantic features in complex scenes and has limited applicability. The accuracy still needs to be improved. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for a people detection model that can alleviate the dependence on labeling and improve accuracy in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for training a people detection model, the method comprising:

[0006] Obtain unlabeled features and labeled features from crowd scenes collected from multiple perspectives;

[0007] Based on the unlabeled features of each group of perspectives, the uncertainty of the number of people is predicted to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0008] Comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result;

[0009] determining the unmarked loss according to the comparison result;

[0010] The number of people prediction model is trained according to the unlabeled loss and the labeled loss corresponding to the labeled features.

[0011] In one embodiment, the predicting of the number of people uncertainty based on the unlabeled features of each group of viewing angles to obtain the number of people uncertainty corresponding to each group of viewing angles includes:

[0012] Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives;

[0013] Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0014] In one embodiment, comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result includes:

[0015] According to the number of the viewing angles, the uncertainty of the number of people corresponding to each group of viewing angles is compared in turn to obtain each comparison result obtained in turn;

[0016] The determining the unlabeled loss according to the comparison result includes:

[0017] Based on each of the comparison results in turn, an unlabeled loss is determined.

[0018] In one embodiment, the uncertainty of the number of people corresponding to each group of viewing angles includes the uncertainty of the number of people in the i-th group of viewing angles and the uncertainty of the number of people in the j-th group of viewing angles, i is less than j, and i and j are both positive integers;

[0019] The determining the unlabeled loss according to the comparison result includes:

[0020] If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, adjusting the unmarked loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints;

[0021] If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

[0022] In one embodiment, adjusting the unlabeled loss according to the difference in number of people uncertainty between the j-th group of viewing angles and the i-th group of viewing angles includes:

[0023] Based on the uncertainty of the number of people in the j-th group of viewing angles and the uncertainty of the number of people in the i-th group of viewing angles, a difference calculation is performed to obtain a current uncertainty difference;

[0024] Determining a penalty value that is positively correlated with the current uncertainty difference;

[0025] According to the penalty value, the current unlabeled loss is adjusted to obtain an adjusted unlabeled loss.

[0026] In one embodiment, before training the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes:

[0027] Determine the uncertainty of the nth labeled predicted number of people and the labeled predicted number of people based on the labeled features of the first n perspectives;

[0028] Determining the nth auxiliary task loss according to the mean square error between the nth labeled predicted number of people and the reference number of people;

[0029] Determine an nth reference uncertainty according to the absolute difference between the nth annotated predicted number of people and the reference number of people, and determine an nth annotated uncertainty loss according to the mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty;

[0030] Determine the labeled loss according to (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, and n is greater than 1, and n is less than or equal to (k-1), and k is the number of the viewing angles;

[0031] Determine the main task loss according to the mean square error between the kth labeled predicted number of people and the reference number of people; the kth labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives;

[0032] The training of the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features includes:

[0033] The number of people prediction model is trained according to the unlabeled loss, the labeled loss and the main task loss.

[0034] In one embodiment, the method further comprises:

[0035] Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people;

[0036] The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number marking map; the pixel value in the number marking map represents the reference number of people.

[0037] In a second aspect, the present application also provides a training device for a people detection model, comprising:

[0038] An acquisition module is used to acquire unlabeled features and labeled features obtained by capturing crowd scenes from multiple perspectives;

[0039] A prediction module, used for predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives, to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0040] A comparison module, used for comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result;

[0041] An adjustment module, used for determining the unlabeled loss according to the comparison result;

[0042] A training module is used to train a headcount prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features.

[0043] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the training steps of the people detection model in any of the above embodiments are implemented.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of training a people detection model in any of the above embodiments.

[0045] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of training the people detection model in any of the above embodiments are implemented.

[0046] The training method, device, computer equipment, computer-readable storage medium and computer program product of the above-mentioned number detection model obtain the unlabeled features and labeled features collected from the crowd scene at multiple perspectives, and can reduce the impact of occlusion through multiple perspectives, which helps to ensure the accuracy of number recognition. Then, by predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives, the uncertainty of the number of people corresponding to each group of perspectives is obtained, and a method of fusion prediction of different numbers of perspectives is formed, so that the unlabeled data and labeled data of different numbers of perspectives are trained separately, forming a semi-supervised training mode, which greatly reduces the dependence on labeled data, and does not require additional labeling, and can combine the unlabeled features with tasks such as density regression of the number of people to improve the performance of crowd counting. By comparing the uncertainty of the number of people corresponding to each group of perspectives, a comparison result of uncertainty is obtained. The reliability of different multi-perspective fusion processes can be measured through the comparison result, and whether there are errors in the corresponding prediction process, so as to form the order of uncertainty results for each group of perspectives, and then the corresponding unlabeled loss is gradually adjusted through the comparison result, so that the unlabeled loss is determined by the principle of size sorting; finally, the unlabeled loss and the labeled loss corresponding to the labeled features are simultaneously applied to the training process of the number prediction model. In this way, multi-perspective fusion prediction is performed on different numbers of perspective features, and then the uncertainty results of the prediction results are sorted, so as to introduce constraints for the multi-perspective crowd counting model. Semi-supervised training is performed for the relationship between the number of perspectives and the uncertainty of the number of people. Unlabeled features that do not require additional labeling are trained together with the corresponding tasks of labeled features to form a semi-supervised model training method, which can improve the performance of crowd counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] FIG1 is a diagram showing an application environment of a training method for a number detection model in one embodiment;

[0049] Figure 2 A schematic diagram of a flow chart of a training method for a people detection model in one embodiment;

[0050] Figure 3 A schematic diagram of a process for sorting uncertainty in one embodiment;

[0051] Figure 4 A schematic diagram of a process flow of unmarked loss adjustment in an embodiment;

[0052] Figure 5 A schematic diagram of a process for determining a marked loss in one embodiment;

[0053] Figure 6 It is a schematic diagram of the effect of an uncertainty diagram in one embodiment;

[0054] Figure 7 A schematic diagram of a network structure of a number of people prediction model in an embodiment;

[0055] Figure 8 A schematic diagram of a process for predicting the number of people in one embodiment;

[0056] Fig. 9 A schematic diagram of the characteristic effect of fusion of each set of viewing angle characteristics in one embodiment;

[0057] Fig.10 is a structural block diagram of a training device for a number detection model in one embodiment;

[0058] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] When counting people in crowd scenes, traditional single-view counting methods often fail to achieve satisfactory results due to serious occlusion problems. To address this challenge, multi-view crowd counting can be performed. However, the collection and annotation of multi-view images face great difficulties, resulting in a scarcity of available multi-view counting datasets. In the face of limited data, synthetic data can be collected to circumvent the annotation step, thereby making up for the lack of real data for fully supervised training; semi-supervised or unsupervised learning methods can also be proposed on limited multi-view datasets in order to better adapt to actual scenarios.

[0061] Optionally, the crowd density map based on Gaussian convolution can be used for estimation; since it is trained by the difference between the predicted density map and the true density map, it avoids dependence on simple regression or classification and has good interpretability and learnability. Under a single perspective, this method performs well in solving the crowd scale problem, but it still has shortcomings in the occlusion problem. To solve the occlusion problem, a multi-perspective crowd counting method is proposed to reduce the impact of occlusion through multiple perspectives.

[0062] If the multi-view counting method relies entirely on fully supervised training, it requires a large amount of labeled multi-view scene data, while the existing real-scene multi-view crowd data annotation is relatively scarce; moreover, there is currently no relevant semi-supervised method in the field of multi-view crowd counting. In addition, in many works, the fusion of features from different stages and perspectives is regarded as a key method to improve the performance of the final task, but the prediction results of multiple perspective fusion are not used. Therefore, in order to reasonably and effectively utilize the feature information of multiple perspectives, the occlusion problem is overcome to a certain extent, and a semi-supervised strategy is adopted to reduce the dependence on labeled data, which improves the generalization ability of the method to a certain extent. In addition, a method based on uncertainty sorting is adopted to use the prediction results of multiple perspectives to improve the crowd counting performance under the current method framework.

[0063] The training method of the people detection model provided in the embodiment of the present application can be applied to the application environment shown in FIG1. ​​The terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers.

[0064] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0065] In an exemplary embodiment, Figure 2 As shown, a training method for a people detection model is provided, and the method is applied to the server 104 in FIG. 1 as an example for explanation, including the following steps 202 to 210. Among them:

[0066] Step 202: Obtain unlabeled features and labeled features obtained by capturing a crowd scene from multiple perspectives.

[0067] Unlabeled features are features of unlabeled data, and there is no corresponding label for unlabeled data. Labeled features are features extracted from labeled data, and there are corresponding labels for labeled features. Both unlabeled features and labeled features are used to characterize human features, such as head features, face features, body features, etc.

[0068] For each perspective, a multi-array convolutional neural network (CNN) can be used to extract labeled features or unlabeled features of different scales; the labeled features or unlabeled features can be extracted in the following way: MCNN uses three networks with different convolution kernel sizes to extract the features of the crowd image, and finally fuses these features through 1×1 convolution. However, this multi-network structure has many model parameters and large computational complexity, making it difficult to achieve real-time prediction. In addition, the multi-array network cannot efficiently extract the features of different human heads and has an inefficient branch structure.

[0069] CP-CNN can be used; CP-CNN enhances the constraints of the crowd density map through global and local semantic information. Its network contains three branches: one for global context, by classifying the entire image into features of different density levels; the other branch performs the same operation on local blocks of the image to obtain features of local context. Finally, the global and local context are concatenated with the density map features in the channel dimension, comprehensively considering the global and local density information in the image, so that the network can adaptively learn the density level features of any image.

[0070] Switch-CNN can be used; Switch-CNN also uses three sub-networks and classification methods, but the difference is that it selects corresponding sub-networks for processing image blocks of different density levels, thereby improving prediction accuracy. During the training process, all networks are first pre-trained through pre-training technology, and then the classification network determines the input path of each image block; this method is similar to MCNN in terms of the number of branches.

[0071] CSRNet can be used; CSRNet uses a Single-Column framework; through the pre-trained VGG-Net network, and through dilated convolution, better counting effects can be obtained. Compared with non-dilated convolution, dilated convolution is better at capturing the edge information of the human head to form the characteristics of the human head.

[0072] You can use ic-CNN, which uses a method of gradually refining from low resolution (LR) to high resolution (HR). The structure is clear and scalable, inspired by the field of image generation, where it is common to gradually generate density maps from low to high resolution.

[0073] In an optional embodiment, obtaining unlabeled features and labeled features obtained by collecting crowd scenes from multiple perspectives includes: selecting VGG-Net as a feature extraction framework for single-view images, performing feature extraction on image data of crowd scenes at each perspective, and obtaining unlabeled features and labeled features. Among them, the reason for selecting the VGG-Net network model is that in the multi-view crowd counting task, the same object may be located at different positions under different perspectives, and the size of the crowd at the same position under each perspective may also be different. Correspondingly, VGG-Net, as a classic convolutional neural network, has a deep structure and can gradually extract more and more rich semantic information through convolution and downsampling operations. As the depth of the network increases, the size of the feature map extracted by VGG-Net will gradually decrease, and some high-level semantics reflecting the distribution of the crowd will be extracted. Combining high-level semantics with underlying semantics can form more robust crowd features. As a result, the robustness of unlabeled features and labeled features can be made higher.

[0074] In an optional embodiment, the projection features obtained by projecting the unlabeled features with the labeled features are used for number prediction.

[0075] When the camera is fixed, the calibration parameters (intrinsic and extrinsic) of the camera are known, and the camera frames across views are synchronized; therefore, given a set of multi-view images, a scene-level density map defined by a fixed region on the ground plane of the 3D scene can be predicted. Multi-view feature fusion consists of two steps: projection of single-view features to the ground plane and fusion of the features after projection.

[0076] Optionally, the above-mentioned step of projecting the single view to the ground plane includes that the feature map obtained by the previous single view feature extraction and decoding module passes through a projection layer calculated by camera calibration parameters, the projection layer is based on a spatial transformer network (STN), and then the feature map is projected to the scene ground plane. Thus, the parameters of the projection layer are determined by the changing scene and camera perspective to adapt to the changes in different scenes and the number of cameras, so as to be able to process multi-view image features simultaneously.

[0077] Optionally, the step of feature fusion after projection includes: extracting multiple single-view features, projecting them from the image plane to a common scene plane through a spatial transformation network, and then performing feature fusion.

[0078] For example, assuming that the projection feature of view i is expressed as , the projection feature set of all camera views is . is the camera view set, where k represents the number of cameras. The multi-camera view features will be concatenated (abbreviated as cat) together, and then the entire multi-view feature representation is finally obtained. The fusion feature of the first j views is ;

[0079] Among them, i {1,2,...j}. After the fused multi-view features are input into the multi-view decoder, the corresponding convolutional network layer further convolves the features to obtain more advanced feature semantics. Here, it is assumed that the convolutional features Represents the features of the projection features after the fusion of the first i perspectives through the corresponding convolutional network layer, that is:

[0080]

[0081] Step 204 , predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives, and obtaining the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives.

[0082] The uncertainty of the number of people (Uncertainty) represents the uncertainty or level of uncertainty in the number of people in each group of perspectives. This uncertainty can be related to the choice of model structure and the imprecision of parameter estimation. Optionally, uncertainty is negatively correlated with confidence. The uncertainty of the number of people can be determined through the neural network model.

[0083] Optionally, based on the density map method, a density value can be assigned to each pixel in the feature map, and the density value can be used to represent the density of people in different areas. The total number of people can be inferred by the density value of each pixel, and the accumulation of all values ​​of the density map is the prediction result.

[0084] Optionally, the unlabeled features of each group of perspectives are fused separately to form fused unlabeled features; the fused unlabeled features may be features under a single perspective, and a crowd counting method under a single perspective may be used.

[0085] Step 206, comparing the uncertainty of the number of people corresponding to each group of viewing angles to obtain a comparison result.

[0086] Alternatively, a self-supervised proxy task can be designed based on the fact that a wider scene should contain more individuals for the same area. The number of errors made on this proxy task can be considered as uncertainty. Optionally, a matching patch proxy function can be used to better approximate the uncertainty of the number of people for the crowd counting task; this approach pays enough attention to detail while maintaining appropriate granularity.

[0087] Optionally, a confidence branch can be added to the classification prediction branch and a confidence score can be output for each input to distinguish between in-distribution and out-of-distribution samples. By learning a specific target confidence criterion directly from labeled data, more accurate confidence can be directly generated at test time. Optionally, a multi-task credible pseudo-annotation learning (MTCP) framework can be constructed to guide the generation of credible density map pseudo-annotations and reduce the impact of noise.

[0088] Step 208: determine the unlabeled loss according to the comparison result.

[0089] Optionally, in each single-view scene, there may be current unlabeled losses of any class, which can be adjusted by comparing the results to obtain the unlabeled losses used in step 210. For example, the unlabeled loss can be determined in a semi-supervised manner based on the spatial uncertainty of the regularized proxy task; this step focuses on the crowd information in high-confidence areas through the spatial uncertainty-aware teacher-student framework. In the case where there are no corresponding annotations for features, the substitute task of the teacher model is used to estimate the spatial uncertainty map and guide the feature learning of the main task (density regression) and the student model at the same time. This step also introduces a differential transformation layer to strengthen the spatial consistency regularization between the main task and the proxy task in the student model, thereby improving the prediction accuracy and generating a high-quality uncertainty map.

[0090] Optionally, a supervised uncertainty estimation strategy can be used to calibrate the uncertainty of the crowd counting model, and a block-by-block proxy function is designed to generate reliable pseudo-annotations. A learning region ranking framework is used to train the region population ranking rule in unlabeled crowd images. The region population ranking rule means that the number of people contained in any region block is always no less than the number of people in its containing sub-blocks. This semi-supervised step solves the problem of limited existing datasets and difficult annotation.

[0091] The semi-supervised crowd counting problem can be solved from the perspective of feature learning; correspondingly, a set of binary segmentation tasks related to the original density map regression task can be derived as alternative prediction targets, and the common features of the crowd can be learned through labeled and unlabeled data. Pseudo-annotations are generated by using an auxiliary target predictor trained with an unlabeled loss and the potential constraints between these pseudo-annotations are used to assist training.

[0092] In an optional embodiment, comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result includes: comparing the comparison results of the uncertainty of the number of people between each group of perspectives in descending order of the number of perspectives to obtain each comparison result in sequence. Correspondingly, determining the unlabeled loss according to the comparison result includes: judging whether the current unlabeled loss is adjusted according to each comparison result obtained in sequence; if adjusted, adjusting the current unlabeled loss based on each comparison result obtained, until all comparison results are used to determine the unlabeled loss, and obtaining the unlabeled loss for model training.

[0093] Therefore, after obtaining the uncertainty of multi-view fusion prediction by inputting the perspectives of different numbers of cameras, the unlabeled loss can be determined by comparing the results and sorting them. The core idea is to introduce constraints for the multi-view crowd counting model. Optionally, there are the following rules that the model is forced to follow during training: more camera perspectives should cover more people and handle occlusion better, so the uncertainty of the number of people predicted by the fusion should be lower, that is, the more perspectives there are, the lower the uncertainty of the number of people corresponding to the crowd counting result, and the two are inversely proportional.

[0094] Step 210 , training a headcount prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features.

[0095] The labeled loss is a model loss determined based on the labeled features. Optionally, the labeled loss can be a dynamic loss function or a specific loss value.

[0096] In an optional embodiment, the number of people prediction model includes an image encoder, a multi-view decoder trained with main task loss and auxiliary task loss, and an uncertain network trained based on unlabeled sorting loss and labeled uncertainty loss; wherein, the labeled losses corresponding to the labeled features include main task loss, auxiliary task loss and uncertain loss; and the unlabeled loss is for unlabeled data, and the three networks can be trained through the three losses of labeled features and the unlabeled loss for labeled / unlabeled data respectively.

[0097] For the crowd prediction model, it can use the multi-view counting framework and the CityStreet dataset to fuse multiple camera views to predict 3D scene-level density maps; and can use multi-view multi-scale (MVMS) counting, in which case CNN is used to fuse camera views to predict 2D scene density maps, and 3D feature fusion is used to handle complex multi-view counting tasks. Optionally, 2D pixels can be projected onto 3D voxels based on a highly estimated attention module to ensure consistency between 3D predictions and 2D true values.

[0098] Optionally, the crowd prediction model may include: a cross-view cross-scene (CVCS) model; dynamically selecting and fusing multiple views through an attention mechanism, and using a noisy view regularization method to handle camera calibration errors or feature errors. Among them, the cross-view cross-scene dataset contains multi-scene, multi-person, and multi-view data. Optionally, by adding correlations between views, the geometric and semantic information can be decomposed into a general fusion component and an affine transformation component for a specific view, effectively reducing parameter redundancy. Optionally, a multi-view learning model based on a graph can be used to perform contextual reasoning through a graph interaction space and a graph communication layer (GCL) to handle information fusion within and across viewpoints.

[0099] Alternatively, although the background and foreground regions are from the same image, they have very different appearances, and the foreground region can find regions with similar density in other crowd images. Based on this principle, a semi-supervised counting step guided by a density proxy can be used; correspondingly, this step constructs a density proxy to align the foreground region features with the corresponding density subclass features, while maximizing the difference between the foreground and background features to further ensure the accuracy of the crowd prediction model.

[0100] In the training method of the above-mentioned number detection model, the unlabeled features and the labeled features collected from the crowd scene at multiple perspectives are obtained, and the influence of occlusion can be reduced through multiple perspectives, which is helpful to ensure the accuracy of number recognition. Then, the number of people is predicted based on the unlabeled features of each group of perspectives, and the number of people corresponding to each group of perspectives is obtained, forming a fusion prediction method of different number of perspectives, so that the unlabeled data and the labeled data of different number of perspectives are trained separately, forming a semi-supervised training mode, which greatly reduces the dependence on labeled data, and does not require additional labeling, and can make the unlabeled features combined with tasks such as density regression of the number of people be trained together to improve the performance of crowd counting. And the uncertainty of the number of people corresponding to each group of perspectives is compared to obtain the comparison result of uncertainty. Through this comparison result, the reliability of different multi-perspective fusion processes can be measured, and whether there are errors in the corresponding prediction process can be reflected, so as to form the order of uncertainty results for each group of perspectives, and then the corresponding unlabeled loss is gradually adjusted through the comparison result, so that the unlabeled loss is determined by the principle of size sorting; finally, the unlabeled loss and the labeled loss corresponding to the labeled features are simultaneously applied to the training process of the number prediction model. Therefore, multi-view fusion prediction is performed on different numbers of view features, and then the uncertainty results of the prediction results are sorted to introduce constraints into the multi-view crowd counting model. According to the relationship between the number of viewpoints and the uncertainty of the number of people, semi-supervised training is performed. The unlabeled features that do not require additional labeling are combined with the corresponding tasks of the labeled features for training, forming a semi-supervised model training method, which can improve the crowd counting performance.

[0101] In an exemplary embodiment, the uncertainty of the number of people is predicted based on the unlabeled features of each group of perspectives to obtain the uncertainty of the number of people corresponding to each group of perspectives, including: predicting the uncertainty of the number of people in a crowd scene based on the unlabeled features of the first i perspectives to obtain the uncertainty of the number of people in the i-th group of perspectives; predicting the uncertainty of the number of people in a crowd scene based on the unlabeled features of the first j perspectives to obtain the uncertainty of the number of people in the j-th group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0102] Optionally, the unlabeled features of the first i perspectives are spliced ​​to obtain the i-th spliced ​​features; the i-th spliced ​​features are input into the neural network model to obtain the uncertainty of the number of people in the i-th group of perspectives. Thus, the i-th spliced ​​features are formed by the feature splicing method to integrate all the features of the first i perspectives.

[0103] Optionally, the unlabeled features of the first j perspectives are spliced ​​to obtain the jth spliced ​​feature; the jth spliced ​​feature is input into the neural network model to obtain the uncertainty of the number of people in the jth group of perspectives. Thus, the jth spliced ​​feature is formed by the feature splicing method to integrate all the features of the first j perspectives.

[0104] The overlap between the first i and j perspectives is equivalent to controlling the perspective variable. Since the uncertainty corresponding to more perspectives should be smaller, that is, when i is less than j, the uncertainty of the number of people in the jth perspective group is based on the prediction of unlabeled features of more perspectives, and the uncertainty reflects the accuracy of the prediction at a certain pixel point. Therefore, under theoretical rules, the uncertainty corresponding to the jth perspective group should be less than the uncertainty corresponding to the ith perspective group. Correspondingly, because the perspective variable is controlled, once the uncertainty corresponding to the number of people in the jth perspective group is greater than the uncertainty corresponding to the number of people in the ith perspective group, it is easier to find that the model needs to be adjusted.

[0105] In this embodiment, each group of perspectives has overlapping unlabeled features and prediction results, and are arranged in a corresponding order. Since the uncertainty can reflect the accuracy of the prediction of a certain pixel point, the uncertainty corresponding to the number of people in each group of perspectives should have a unidirectional change law. Therefore, once the uncertainty does not conform to the unidirectional change law, it can be reflected through the comparison result corresponding to the uncertainty, thereby improving the accuracy of the unlabeled loss.

[0106] In an exemplary embodiment, Figure 3 As shown, the uncertainty of the number of people corresponding to each group of viewpoints is compared to obtain a comparison result, including step 302; correspondingly, the unlabeled loss is determined according to the comparison result, including step 304; wherein:

[0107] Step 302 , according to the number of viewing angles, the uncertainty of the number of people corresponding to each group of viewing angles is compared in turn, and each comparison result is obtained in turn.

[0108] Optionally, the uncertainty of the number of people corresponding to each group of perspectives may be compared in sequence according to the number of perspectives contained in each group of perspectives and in the order of the number of perspectives from large to small or from small to large, to obtain each comparison result obtained in sequence.

[0109] Optionally, the uncertainty corresponding to the number of people in the i-th group of perspectives is compared with the uncertainty corresponding to the number of people in the j-th group of perspectives in turn to obtain a comparison result between the i-th group of perspectives and the j-th group of perspectives. Thus, the comparison result between the i-th group of perspectives and the j-th group of perspectives forms each comparison result obtained in turn.

[0110] Step 304 , determining the unlabeled loss based on each comparison result in turn.

[0111] Optionally, based on each comparison result, it is determined whether the current unlabeled loss needs to be adjusted; if it needs to be adjusted, it is adjusted based on each comparison result in turn; if it does not need to be adjusted, it is determined based on the next comparison result whether the current unlabeled loss needs to be adjusted, until each comparison result is determined to be completed.

[0112] In this embodiment, by comparing in sequence according to the number of viewpoints, the comparison results in the corresponding iteration process can be obtained in an orderly manner, and the unlabeled loss is continuously adjusted, so that the unlabeled loss is gradually improved, and the required accuracy is controllable.

[0113] In an exemplary embodiment, Figure 4 As shown, the uncertainty of the number of people corresponding to each group of perspectives includes the uncertainty of the number of people in the i-th group of perspectives and the uncertainty of the number of people in the j-th group of perspectives, i is less than j, and i and j are both positive integers. The unlabeled loss is determined based on the comparison results, including:

[0114] Step 402: If the uncertainty of the number of people in the j-th group of viewing angles is greater than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is adjusted according to the difference in the uncertainty of the number of people between the j-th group of viewing angles and the i-th group of viewing angles.

[0115] The uncertainty difference may be the difference between the uncertainty of the number of people in the j-th group of perspectives and the uncertainty of the number of people in the ith group of perspectives; the uncertainty difference may be the difference or the absolute value of the difference.

[0116] In an optional embodiment, the unlabeled loss is adjusted according to the uncertainty difference of the number of people between the j-th group of perspectives and the i-th group of perspectives, including: performing difference calculation based on the uncertainty of the number of people in the j-th group of perspectives and the uncertainty of the number of people in the i-th group of perspectives to obtain a current uncertainty difference; determining a penalty value positively correlated with the current uncertainty difference; and adjusting the current unlabeled loss according to the penalty value to obtain an adjusted unlabeled loss.

[0117] The penalty value is a value used to adjust the current unlabeled loss. The larger the penalty value, the larger the parameter gradient of the updated network, and the larger the adjustment of the current unlabeled loss. Optionally, the penalty value can be a regularization term adjusted by the uncertainty difference, or it can be the uncertainty difference itself. The penalty value can be the current uncertainty difference, or it can be the adjustment of the uncertainty difference.

[0118] Optionally, the current unlabeled loss may be the unlabeled loss at the time of initialization, or the unlabeled loss after the last adjustment. The current unlabeled loss and the adjusted unlabeled loss exist for each uncertainty difference, and multiple adjustments are formed in sequence.

[0119] Therefore, the current uncertainty difference is positively correlated with the penalty value. The larger the current uncertainty difference, the larger the penalty value, which makes the parameter gradient of the updated network larger and the adjustment of the current unlabeled loss larger, so as to achieve gradual adjustment of the unlabeled loss.

[0120] Step 404: If the uncertainty of the number of people in the j-th group of viewpoints is less than the uncertainty of the number of people in the i-th group of viewpoints, then no labeling loss is maintained.

[0121] Maintaining the unlabeled loss means that the uncertainty difference at this time is not used to adjust the unlabeled loss. Optionally, the above comparison result is each comparison result obtained in sequence; thus, the unlabeled loss is determined in sequence according to each comparison result, at this time, i and j in each time are different values, and in each execution process, one of step 402 and step 404 is selected for execution.

[0122] In this embodiment, since i is less than j, the number of people in the jth group of viewpoints is predicted based on features under more viewpoints. Therefore, when the uncertainty of the number of people in the jth group of viewpoints is less than the uncertainty of the number of people in the ith group of viewpoints, it is a normal situation, and there is no need to adjust the corresponding unlabeled loss. Correspondingly, when the uncertainty of the number of people in the jth group of viewpoints is less than the uncertainty of the number of people in the ith group of viewpoints, there is an abnormality in the uncertainty model, and it can be adjusted directly based on the uncertainty difference to improve efficiency.

[0123] In an alternative embodiment, Figure 5As shown, before training the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes steps 502 to 510; correspondingly, training the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features includes step 512, wherein:

[0124] Step 502 , determining the uncertainty of the n-th labeled predicted number of people and the n-th labeled predicted number of people based on the labeled features of the first n perspectives.

[0125] The labeled features of the first n perspectives include the labeled features of at least two perspectives. The standardized features of the first n perspectives can be fused multiple times, and the fused features obtained each time are used for feature extraction to predict each labeled predicted number of people and the corresponding uncertainty, and the corresponding auxiliary task loss and labeled uncertainty loss are obtained each time.

[0126] The labeled predicted number of people and the uncertainty of the labeled predicted number of people are the predictions obtained by inputting the labeled features into different models respectively; the labeled predicted number of people is the predicted number of people; and the uncertainty of the labeled predicted number of people is the uncertainty obtained by prediction based on the labeled features. Optionally, after the labeled features are input into the uncertainty network, the uncertainty network outputs the uncertainty of the labeled predicted number of people.

[0127] Step 504 , determining the nth auxiliary task loss according to the mean square error between the nth labeled predicted number of people and the reference number of people.

[0128] The reference number of people is the labeled value corresponding to the labeled feature; the labeled value represents the standard value of the number of people or the actual number of people. Since the labeled features are features from multiple perspectives, the reference number of people is also the reference number of people from multiple perspectives.

[0129] The mean squared error (MSE) is used to evaluate the difference between the model prediction and the true value. Correspondingly, the square value of each point can be obtained by determining the square of the difference between the labeled predicted number of people and the reference number of people at each point; then, the square value of each point is averaged according to the points contained in the density map to obtain the mean squared error used to determine the loss of the auxiliary task.

[0130] The auxiliary task loss is used to indicate the degree to which the labeled predicted number of people deviates from the reference number of people. Optionally, the auxiliary task loss may be a loss function or a loss value calculated by the loss function.

[0131] Step 506, determining the nth reference uncertainty based on the absolute difference between the nth labeled predicted number of people and the reference number of people, and determining the nth labeled uncertainty loss based on the mean square error between the uncertainty of the nth labeled predicted number of people and the nth reference uncertainty.

[0132] The absolute difference is the absolute value of the difference between the labeled predicted number of people and the reference number of people. Since the labeled predicted number of people is a predicted value, the numerical size of the predicted value and the reference number of people is difficult to know in advance, so the absolute difference can accurately measure the corresponding uncertainty.

[0133] The reference uncertainty is a pseudo-label, which is obtained by converting the prediction result and the reference number of labeled data, and can represent the theoretical uncertainty. Optionally, the above absolute difference can be used as the reference uncertainty, or the processing result of the above absolute difference can be used as the reference uncertainty.

[0134] The annotated uncertainty loss is the corresponding uncertainty loss of the standardized characteristic.

[0135] The mean squared error (MSE) is used to evaluate the difference between the model prediction value and the true value. Correspondingly, the square value of the uncertainty at each point can be obtained by taking the difference between the uncertainty of the labeled predicted number of people and the reference uncertainty at each point, and then squaring the difference in uncertainty; then the square value of the uncertainty at each point is averaged according to the points contained in the density map, and the mean squared error used to calculate the loss of labeled uncertainty can be obtained.

[0136] Optionally, based on the labeled features of the first n perspectives, the uncertainty corresponding to the nth labeled predicted number of people and the nth labeled predicted number of people is determined; in the case where the labeled predicted number of people is determined based on the labeled features of the first n perspectives, the reference number of people is the reference number of people under the first n perspectives; based on the predicted number of people under the first n perspectives and the reference number of people, the absolute difference between the two is obtained; based on the absolute difference and the uncertainty corresponding to the nth labeled predicted number of people, the nth labeled uncertainty loss is determined. Thus, the labeled predicted number of people and the corresponding uncertainty are determined for each group of perspectives, and each group of labeled uncertainty losses is obtained.

[0137] Step 508, determining the labeled loss according to the (k-1)th n-th auxiliary task loss and the (k-1)th n-th labeled uncertainty loss; wherein n and k are integers, and n is greater than 1, and n is less than or equal to (k-1), and k is the number of viewing angles.

[0138] In an optional embodiment, the labeled loss is determined based on the (k-1)th auxiliary task losses and the (k-1)th labeled uncertainty losses, including: averaging the (k-1)th auxiliary task losses to obtain the auxiliary task mean loss; averaging the (k-1)th labeled uncertainty losses to obtain the labeled uncertainty mean loss; weighting the auxiliary task mean loss and the labeled uncertainty mean loss according to preset weights to obtain the weighted auxiliary task loss and the weighted labeled uncertainty loss; obtaining the labeled loss based on the weighted auxiliary task loss and the weighted labeled uncertainty loss.

[0139] In an optional embodiment, the labeled loss is obtained according to the weighted auxiliary task loss and the weighted labeled uncertainty loss, including: combining the main model loss, the weighted auxiliary task loss, and the weighted labeled uncertainty loss to obtain the labeled loss. The main model loss is trained based on the labeled features, and the main model loss is different from the auxiliary task loss, and the main model loss is also different from the labeled uncertainty loss.

[0140] Step 510, determining the main task loss according to the mean square error between the k-th labeled predicted number of people and the reference number of people; the k-th labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives.

[0141] The number of labeled predicted people corresponding to the kth number of views is the number of people predicted based on all views; since each view is used, the number of views is equivalent to reaching the maximum value. Although theoretically, the training effect of the number prediction model using the main task loss is the best, using the unlabeled loss, the labeled loss and the main task loss for training can achieve higher accuracy.

[0142] Optionally, the k-th labeled predicted number of people may be determined based on a number prediction model in training.

[0143] Step 512: training the number of people prediction model according to the unlabeled loss, the labeled loss and the main task loss.

[0144] In an optional embodiment, the number of people prediction model is trained according to the unlabeled loss, the labeled loss and the main task loss, including: determining the total model loss based on the sum of the unlabeled loss, the labeled loss and the main task loss; and training the number of people prediction model according to the total model loss.

[0145] Optionally, the unlabeled loss, the labeled loss and the main task loss may be weightedly fused, and the weight of the main task loss is greater than the weight of the unlabeled loss, and the weight of the main task loss is greater than the weight of the labeled loss.

[0146] In this embodiment, the auxiliary task loss is determined based on the mean square error between the labeled predicted number of people and the reference number of people, forming an adjustment method for model training, which helps to improve the accuracy of model prediction; and the mean square error calculation process for determining the auxiliary task loss includes an absolute difference calculation process, and the absolute difference is reused to determine the reference uncertainty, which can ensure processing efficiency. Furthermore, based on the mean square error between the uncertainty of the labeled predicted number of people and the reference uncertainty, the labeled uncertainty loss is determined, so that the uncertainty of the labeled predicted number of people is converted into a relative value for measurement. Furthermore, based on at least the auxiliary task loss and the labeled uncertainty loss, the labeled loss is determined, so that the number prediction model is trained through this labeled loss and the non-standard loss in conjunction with the main task loss, so that the trained number prediction model has a more accurate prediction result.

[0147] In an optional embodiment, the method also includes: obtaining a plane annotation map of the crowd scene; the plane annotation map includes multiple pixel points, and the pixel value of each pixel point represents the number of people; through a Gaussian kernel function, the pixel value of each pixel point is replaced by the weighted pixel average of the adjacent pixel points of each pixel point to obtain a number marking map; the pixel values ​​in the number marking map represent the reference number of people.

[0148] The plane annotation map can be a density map, and the pixel value of each pixel point represents the density value of each point to reflect the number of people at each point.

[0149] The neighboring pixel is a pixel whose distance to each pixel is less than a preset value. For example, there is no other pixel between each pixel and its neighboring pixel. In this case, the neighboring pixel is the pixel closest to each pixel. The reference number of people is a parameter in the model training process, and it represents the number of people.

[0150] Optionally, a Gaussian kernel function is used to replace the pixel value of the annotation center point with a weighted average of the pixel values ​​of the surrounding points to ensure that the sum of the weights of the surrounding pixels is 1.

[0151] In this embodiment, the pixel values ​​in the plane annotation map represent the number of people, so that the corresponding data can be obtained in the form of a density map; and through the Gaussian kernel function, the pixel value of the plane annotation map at the annotation center point is replaced by the weighted pixel average of the neighboring pixels of the center point to obtain the number of people label map. In this way, the total number of heads generated is not changed, and the position characteristics of each head in space are more accurately and truly reflected, so that the reference number is more accurate.

[0152] It is important to understand that crowd counting scenarios involve crowd occlusion and changes in the scale of people. Although the existing multi-view counting framework can solve the above problems well, a new problem will arise after the introduction of the multi-view counting framework: the multi-view crowd counting dataset is limited and difficult to label. To solve this problem, after a thorough investigation, it was found that there is currently no relevant multi-view semi-supervised work in the field of crowd counting. In the face of limited labeled data, an innovative semi-supervised multi-view crowd counting method is proposed. By utilizing multi-view fusion features, predicting uncertainty through the fused features, and sorting the uncertainty, the unlabeled data information is fully mined to assist the model in better learning crowd characteristics.

[0153] Because more camera perspectives can cover a wider range of people and provide richer information about the people in the scene. Considering that when a certain area can be covered by all the perspectives involved, the prediction of the number of people in the area should be certain regardless of whether it is observed from a few perspectives or from many perspectives.

[0154] Each point in the density map reflects the probability that there is a person at that point. All probabilities in an area are accumulated, and the value is the estimated number of people in the area. If there are people around a specific area, the value of their coordinates after Gaussian blur will also be added to the area. The more people around the area, the higher the cumulative probability value of the area. Gaussian blur is to make the model easier to train. Otherwise, it is very difficult to train by directly using the original point map to represent the reference number of people (ground-truth).

[0155] At the same time, after the introduction of uncertainty in the field of semi-supervised crowd counting, the training can be assisted by introducing the perspective fusion prediction uncertainty sorting. In the same crowd scene, N perspectives are numbered 1→N accordingly, and the real density map of a certain area in the scene under i perspectives is used as the reference number. , the corresponding predicted density value is If the predicted density value of 1 viewing angle feature is , the uncertainty under the definition 1 perspective in this paper should be , Refers to the Ground-Truth corresponding to perspective 1 (the density map of the original point map after Gaussian blur). After introducing the new perspective 2, if the predicted density value of the perspective combination [1,2] is , then the uncertainty from the [1,2] perspective should be , It refers to the original point map that can be seen at the same time from the [1,2] perspective, and the density map after Gaussian blur.

[0156] The uncertainty sorting of perspective fusion requires the uncertainty under 1 perspective Should be greater than the uncertainty under the [1, 2] perspective combination In this way, we can clearly know that when the introduced perspective combination is [1,..., N], it should have the lowest uncertainty, that is, the highest confidence. At this time, In the prediction process, the uncertainty graphs predicted by different numbers of viewing angles should also satisfy the above potential rules. As a punishment for training, when the above When , the penalty is positive and the larger the gap, the greater the penalty. When the penalty is non-positive, the prediction result meets the potential sorting rule and no penalty is imposed on the network. In general, uncertainty reflects the accuracy of the prediction at a certain pixel point. The more points are seen from different perspectives, the higher the accuracy is, that is, the lower the uncertainty is. The uncertainty maps of different numbers (normalized) are fused under three data sets as shown in Figure 6 As shown, the darker the color of the point, the higher the uncertainty of the position, and the lighter the color, the lower the uncertainty of the position. It can be clearly seen that the lower the uncertainty predicted by integrating more viewing angle features, that is, the lighter the color.

[0157] In a specific embodiment, Figure 7 As shown, in order to implement a semi-supervised multi-view crowd counting method based on the uncertainty sorting of the number of people, the number prediction model of this embodiment includes at least three modules, namely a single-view feature extractor, a multi-view decoder, and an uncertainty network involving uncertainty. Each view image obtains the corresponding image features through the single-view feature extractor, and then projects it to the scene reference plane. The fused feature map of each view is fed into the corresponding network module respectively; the multi-view fusion projection features of the first 1 to k views are written as f1, f2, ..., f k , respectively represent the feature expression of the concatenation and fusion of the projection features of the first 1, first 2 to first k perspectives, then f1 to f k The features are input into the multi-view decoder.

[0158] In the multi-view decoder, convolution features are obtained, and the processing of convolution features is divided into two parts:

[0159] 1) If Figure 7 As shown in (a) in the figure, if the input is labeled data, the multi-view decoder will output f1 to f k The corresponding number prediction density map, where the prediction results corresponding to the first 1 to k-1 perspectives are called auxiliary predictions, and the predictions of all k perspectives are called main predictions. The MSE loss between these number prediction density maps and the corresponding reference uncertainty (ReferenceU-GT, GT) is calculated and divided into the main task loss and auxiliary task loss ;like Figure 7 The intermediate features of the multi-view decoder are also input to the uncertainty decoder, and the uncertainty prediction map of the corresponding different fused views is output. The reference GT is the absolute difference between the number of people predicted at each fused view output by the multi-view decoder and the GT. The MSE loss is calculated between the uncertainty prediction map and the GT, which is called the uncertainty loss. .

[0160] 2) If Figure 7 As shown in (b) in the figure, if the input is unlabeled data, only the intermediate features of the multi-view decoder are input into the uncertainty decoder to obtain the prediction uncertainty of each fused view, and the ranking loss between them is calculated, which is called the unlabeled ranking loss .

[0161] In addition, given a set of multi-view images, the goal is to predict a scene-level density map defined by a fixed area on the ground plane of the 3D scene to determine the number of people. The scene-level density map is as follows: Figure 8 shown.

[0162] Specifically, for the above 1), the convolutional features with labeled data are labeled features. The labeled features will be input into the multi-view decoder to obtain the prediction results corresponding to each fused view, as shown in the following formula:

[0163]

[0164] As the prediction result, the corresponding GT is the fusion of features as mentioned above, and accordingly, , represents the fusion GT labels of the first n views, thus forming the annotation. For example, Fig. 9 As shown in Figure 2, on these three datasets, as the number of fused perspectives increases, the corresponding GT images will also present more people.

[0165] Prediction results and correspondence Find the MSE loss between , which is the auxiliary task loss, denoted as , as shown below:

[0166]

[0167] Where k is the number of viewing angles corresponding to the camera.

[0168] On the other hand, the prediction results of all k views are P k , and its corresponding GT is G k , the MSE loss between the two is called the main task loss, denoted as , as shown below:

[0169]

[0170] It is worth noting that the prediction result will be the only output during the test.

[0171] At the same time, the prediction results and correspondence The absolute difference between them is taken as the reference uncertainty of the uncertainty decoder (Reference U-GT, GT), that is, Figure 7 In addition, the convolutional features are also input into the uncertainty network to obtain the uncertainty corresponding to each fusion perspective (Uncertainty Prediction, ), and The MSE loss is calculated between , which is the uncertainty loss of the labeled data, denoted as , as shown below:

[0172]

[0173] In summary, the total loss of labeled data can be summarized as follows:

[0174]

[0175] in, and is a hyperparameter that adjusts the weight of different loss items. is the main task loss, is the auxiliary task loss. That is, the predictions of all k perspectives are called the main predictions, and the MSE loss between these number prediction density maps and the corresponding reference uncertainty (Reference U-GT, GT) is calculated, which is divided into the main task loss and auxiliary task loss .

[0176] Regarding the above 2), the convolutional features of unlabeled data will not be input into the multi-view decoder because they have no corresponding GT annotations. Instead, they will be directly input into the uncertainty network, and then the uncertainty graph prediction results corresponding to each fused view will be output. At this time, there is no corresponding reference number, but the supervision rules are constructed with the help of its potential characteristics to achieve no label loss, that is, no label sorting loss .

[0177] Based on this rule, if , a corresponding penalty should be imposed on the uncertain network to adjust the network parameters, and the specific penalty value is , and the greater the penalty value, the greater the parameter gradient for updating the network. When , , at this time the penalty is non-positive, that is, the prediction result conforms to the potential sorting rule and there is no need to impose a penalty on the network. That is to say, the network prediction meets the expectation and there is no need for parameter update. At this time, the gradient of the backward propagation of the uncertain network is 0, and the definition formula of the specific penalty amount C is as follows:

[0178]

[0179] By sorting the uncertain graphs with different numbers of fusion perspectives, the entire unlabeled loss can be obtained as shown in the following formula:

[0180]

[0181] Among them, contains k(k - 1) / 2 terms and is calculated by using double summation. For example: assuming there are a total of 3 perspectives, the prediction uncertainty UP3 of fusing 3 perspectives is less than the prediction uncertainty UP2 of fusing 2 perspectives, and at the same time is less than the prediction uncertainty UP1 of 1 perspective, that is, UP3 < UP2, UP3 < UP1, and at the same time UP2 < UP1. There are 3 terms {UP3 < UP2, UP3 < UP1, UP2 < UP1}, and so on, up to the unlabeled loss function of k perspectives.

[0182] In summary, the loss function of the entire model during the training process is:

[0183]

[0184] Among them, is the hyperparameter of the unlabeled loss, is the total loss of the labeled data.

[0185] Based on this, a semi-supervised method is applied to make the most of the unlabeled data as much as possible, greatly alleviating the dependence on the labeled data, especially in the case where it is difficult to label in a large scenario. At the same time, the fusion prediction results of multiple perspectives and the uncertain predictions obtained through the uncertainty network are utilized, and the loss function obtained under a reasonable sorting rule helps to train the model and improves the performance of the crowd counting model.

[0186] Each network module of this embodiment, including the multi-perspective decoder and the uncertainty decoder, can adopt different convolution methods or use different activation functions in its network layers.

[0187] Various experiments have shown that this method performs best on the CVCS and PETS2009 datasets. After introducing the multi-view fusion ranking constraint semi-supervised method, the entire model can not only greatly reduce the demand for labeled data, but also reach or even exceed some fully supervised methods in counting performance, showing its superiority in reducing the demand for labeled data. At the same time, this method fully utilizes the characteristics of multi-view and integrates it into the counting model to make more full use of the potential features of unlabeled multi-view data, which will open up a new idea for subsequent multi-view counting work.

[0188] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0189] Based on the same inventive concept, the embodiment of the present application also provides a training device for a people detection model for implementing the training method for a people detection model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more people detection models provided below can refer to the limitations of the training method for the people detection model above, and will not be repeated here.

[0190] In an exemplary embodiment, Fig.10 As shown, a training device for a people detection model is provided, comprising:

[0191] An acquisition module 1002 is used to acquire unlabeled features and labeled features obtained by capturing a crowd scene from multiple perspectives;

[0192] A prediction module 1004 is used to predict the uncertainty of the number of people based on the unlabeled features of each group of perspectives, and obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives;

[0193] A comparison module 1006, used to compare the uncertainty of the number of people corresponding to each group of viewing angles to obtain a comparison result;

[0194] An adjustment module 1008, configured to determine an unlabeled loss according to the comparison result;

[0195] The training module 1010 is used to train the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features.

[0196] In one embodiment, the prediction module 1004 is used to:

[0197] Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives;

[0198] Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers.

[0199] In one embodiment, the comparison module 1006 is used to:

[0200] According to the number of the viewing angles, the uncertainty of the number of people corresponding to each group of viewing angles is compared in turn to obtain each comparison result obtained in turn;

[0201] Correspondingly, the adjustment module 1008 is used to:

[0202] Based on each of the comparison results in turn, an unlabeled loss is determined.

[0203] In one embodiment, the number of people in each viewpoint group includes the number of people in the i-th viewpoint group and the number of people in the j-th viewpoint group, i is less than j, and i and j are both positive integers;

[0204] The adjustment module 1008 is used to:

[0205] If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, adjusting the unmarked loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints;

[0206] If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

[0207] In one embodiment, the adjustment module 1008 is used to:

[0208] Based on the uncertainty of the number of people in the j-th group of viewing angles and the uncertainty of the number of people in the i-th group of viewing angles, a difference calculation is performed to obtain a current uncertainty difference;

[0209] Determining a penalty value that is positively correlated with the current uncertainty difference;

[0210] According to the penalty value, the current unlabeled loss is adjusted to obtain an adjusted unlabeled loss.

[0211] In one embodiment, the training module 1010 is used to:

[0212] Determine the uncertainty of the nth labeled predicted number of people and the labeled predicted number of people based on the labeled features of the first n perspectives;

[0213] Determining the nth auxiliary task loss according to the mean square error between the nth labeled predicted number of people and the reference number of people;

[0214] Determine an nth reference uncertainty according to the absolute difference between the nth annotated predicted number of people and the reference number of people, and determine an nth annotated uncertainty loss according to the mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty;

[0215] Determine the labeled loss according to (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, and n is greater than 1, and n is less than or equal to (k-1), and k is the number of the viewing angles;

[0216] Determine the main task loss according to the mean square error between the kth labeled predicted number of people and the reference number of people; the kth labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives;

[0217] The number of people prediction model is trained according to the unlabeled loss, the labeled loss and the main task loss.

[0218] In one embodiment, the acquisition module 1002 is used to:

[0219] Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people;

[0220] The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number marking map; the pixel value in the number marking map represents the reference number of people.

[0221] Each module in the training device of the above-mentioned people detection model can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0222] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a number detection model is implemented.

[0223] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0224] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0225] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0226] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0228] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0229] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0230] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a people detection model, characterized in that: The method comprises: Obtain unlabeled features and labeled features from crowd scenes collected from multiple perspectives; Based on the unlabeled features of each group of perspectives, the uncertainty of the number of people is predicted to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives; Comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result; determining the unmarked loss according to the comparison result; The number of people prediction model is trained according to the unlabeled loss and the labeled loss corresponding to the labeled features.

2. The method according to claim 1, characterized in that The predicting of the uncertainty of the number of people based on the unlabeled features of each group of viewing angles to obtain the uncertainty of the number of people corresponding to each group of viewing angles includes: Based on the unlabeled features of the first i perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the i-th group of perspectives; Based on the unlabeled features of the first j perspectives, the uncertainty of the number of people in the crowd scene is predicted to obtain the uncertainty of the number of people in the jth group of perspectives; wherein i is less than j, and i and j are both positive integers.

3. The method according to claim 1, characterized in that The comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result includes: According to the number of the viewing angles, the uncertainty of the number of people corresponding to each group of viewing angles is compared in turn to obtain each comparison result obtained in turn; The determining the unlabeled loss according to the comparison result includes: Based on each of the comparison results in turn, an unlabeled loss is determined.

4. The method according to claim 1, characterized in that: The uncertainty of the number of people corresponding to each group of perspectives includes the uncertainty of the number of people in the i-th group of perspectives and the uncertainty of the number of people in the j-th group of perspectives, i is less than j, and i and j are both positive integers; The determining the unlabeled loss according to the comparison result includes: If the uncertainty of the number of people in the j-th group of viewpoints is greater than the uncertainty of the number of people in the i-th group of viewpoints, adjusting the unmarked loss according to the difference in the number of people uncertainty between the j-th group of viewpoints and the i-th group of viewpoints; If the uncertainty of the number of people in the j-th group of viewing angles is less than the uncertainty of the number of people in the i-th group of viewing angles, the unlabeled loss is maintained.

5. The method according to claim 4, characterized in that The adjusting the unlabeled loss according to the difference in the number of people uncertainty between the j-th group of viewing angles and the i-th group of viewing angles includes: Based on the uncertainty of the number of people in the j-th group of viewing angles and the uncertainty of the number of people in the i-th group of viewing angles, a difference calculation is performed to obtain a current uncertainty difference; Determining a penalty value that is positively correlated with the current uncertainty difference; According to the penalty value, the current unlabeled loss is adjusted to obtain an adjusted unlabeled loss.

6. The method according to claim 1, characterized in that Before training the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features, the method further includes: Determine the uncertainty of the nth labeled predicted number of people and the labeled predicted number of people based on the labeled features of the first n perspectives; Determining the nth auxiliary task loss according to the mean square error between the nth labeled predicted number of people and the reference number of people; Determine an nth reference uncertainty according to the absolute difference between the nth annotated predicted number of people and the reference number of people, and determine an nth annotated uncertainty loss according to the mean square error between the uncertainty of the nth annotated predicted number of people and the nth reference uncertainty; Determine the labeled loss according to (k-1) the nth auxiliary task losses and (k-1) the nth labeled uncertainty losses; wherein n and k are integers, and n is greater than 1, and n is less than or equal to (k-1), and k is the number of the viewing angles; Determine the main task loss according to the mean square error between the kth labeled predicted number of people and the reference number of people; the kth labeled predicted number of people includes the labeled predicted number of people corresponding to the k number of perspectives; The training of the number of people prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features includes: The number of people prediction model is trained according to the unlabeled loss, the labeled loss and the main task loss.

7. The method according to claim 6, characterized in that The method further comprises: Acquire a plane annotation map of the crowd scene; the plane annotation map includes a plurality of pixel points, and the pixel value of each pixel point represents the number of people; The pixel value of each pixel is replaced by the weighted pixel average of the adjacent pixels of each pixel through the Gaussian kernel function to obtain a number marking map; the pixel value in the number marking map represents the reference number of people.

8. A training device for a people detection model, characterized in that: The device comprises: An acquisition module is used to acquire unlabeled features and labeled features obtained by capturing crowd scenes from multiple perspectives; A prediction module, used for predicting the uncertainty of the number of people based on the unlabeled features of each group of perspectives, to obtain the uncertainty of the number of people corresponding to each group of perspectives; each group of perspectives contains a different number of perspectives; A comparison module, used for comparing the uncertainty of the number of people corresponding to each group of perspectives to obtain a comparison result; An adjustment module, used for determining the unlabeled loss according to the comparison result; A training module is used to train a headcount prediction model according to the unlabeled loss and the labeled loss corresponding to the labeled features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images