Crowd counting method and system based on point-to-point framework

Abstract

The application provides a crowd counting method and system based on a point-to-point framework, belongs to the technical field of crowd counting based on computer vision, and acquires an image to be recognized; a pre-trained crowd counting recognition model is used to process the acquired image to obtain a crowd counting recognition result; wherein the model training comprises: inputting a training set image into a backbone network to extract a feature map; inputting the feature map into three-layer convolution networks of a coordinate prediction head and a confidence calculation head respectively to obtain predicted point coordinates and confidence; loss function calculation, using an automatic derivation mechanism of a PyTorch framework, calculating a model parameter gradient direction according to a loss value obtained, and updating the model parameter. The application estimates sample size information without introducing additional labeling workload by using point labeling information in a data set, calculates the influence weight of each labeling point on the final loss function, and improves the ability of the model to detect different scale targets.

Description

Technical Field

[0001] The present invention relates to the technical field of crowd counting based on computer vision, and in particular to a crowd counting method and system based on a point-to-point framework. Background Art

[0002] Crowd counting involves estimating the number of people in an image or video using computer vision techniques. There are two main approaches to crowd counting: detection-based methods and density estimation-based methods.

[0003] Most early research focused on detection-based methods, using detectors similar to moving windows to detect and count people. These methods require well-trained classifiers to extract low-level features (such as Haar wavelets and HOG (histogram of gradients)) from intact human bodies. With the development of deep learning, a series of deep learning-based target detection models (such as Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot Multibox Detector)) have also been used in this task. Compared with traditional methods, the performance of these models has greatly improved.

[0004] Density estimation-based methods primarily work by learning the density of crowd distributions in an image. A common approach for density estimation-based crowd counting is to use a Gaussian kernel function to generate a Gaussian distribution for each person's location, then overlay these Gaussian distributions to form a density map. This generated density map is then used as a target to train a neural network to learn the mapping from the input image to the density map. The Euclidean distance between the actual and predicted density maps is typically used as the loss function during training. Density estimation-based methods are currently the most mainstream approach.

[0005] At present, the mainstream solutions for crowd counting tasks are detection-based methods and density regression-based methods. The disadvantages of these two solutions are as follows:

[0006] The idea behind detection-based methods is relatively simple: just count the types of objects that need to be counted in the image and count the number of detected objects. However, crowd counting often involves crowded scenes, where objects in the crowd are severely occluded by each other. Object detection methods typically use a non-maximum suppression strategy, which can lead to serious missed detections, where only one object is detected from multiple overlapping objects. In addition, since the number of objects in crowd counting scenes is usually large, in order to reduce the labeling workload, mainstream datasets for crowd counting tasks typically use point annotations. This annotation format only provides the location information of the object but not its size information. If you want to use methods based on object detection on such datasets, you need to use some estimation methods to generate pseudo boxes based on the point annotations, and the generated pseudo boxes will also introduce additional errors.

[0007] While density regression methods can leverage convolutional neural networks to fully utilize contextual information in images and generate relatively accurate density map estimates, it is difficult to obtain target location information from the density maps generated by these methods. Therefore, their prediction results are difficult to apply to more complex crowd behavior analysis tasks (such as crowd tracking, crowd behavior prediction, and anomaly detection). Furthermore, the process of generating density maps through point annotation also introduces some additional errors.

[0008] In order to handle more complex crowd behavior analysis tasks, it is not enough to just count the number of people. The location of individuals in the crowd is also key information required for crowd behavior analysis. Although the detection-based method can provide location information, it cannot handle high-density scenarios of crowd counting tasks well. Therefore, a method that directly predicts the location of each target in the crowd based on the input image is proposed. Its main process is as follows: Figure 1 As shown in the figure. After using the backbone network to extract the feature vectors, this method directly uses these feature vectors to calculate the candidate point locations and confidence levels. These results are then combined to directly provide the point coordinate information for predicting the individual positions in the crowd. During the training process, this method uses the Hungarian algorithm to perform a one-to-one match between the output results and the annotation information. Based on the matching results, the Euclidean distance is used to calculate the positioning loss of the candidate points and the annotation points, and the cross entropy is used to calculate the classification loss of the candidate points. In the above method, the imbalance in the number of targets of different sizes in the dataset is not addressed, resulting in the trained model being more inclined to detect smaller targets and having a high miss detection rate for larger targets. Summary of the Invention

[0009] The purpose of the present invention is to provide a crowd counting method and system based on a point-to-point framework to improve the accuracy of counting tasks and realize single target positioning tasks. A new sample scale balance loss function is designed to alleviate the impact caused by the imbalance of target size types in the crowd annotation dataset. The distance information between the annotation points is used to estimate the size of the target corresponding to the annotation point, and a weight is calculated for each annotation point to balance the problem that samples of a specific size have a greater impact on the model training results. The model's detection ability and counting accuracy for targets of different sizes are improved, and at least one technical problem existing in the above-mentioned background technology is solved.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In a first aspect, the present invention provides a crowd counting method based on a point-to-point framework, comprising:

[0012] Obtaining an image to be recognized;

[0013] The acquired images are processed using a pre-trained crowd counting and recognition model to obtain crowd counting and recognition results. Training the crowd counting and recognition model includes: inputting the training set images into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head to obtain the predicted point coordinates and confidence levels, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0014] Furthermore, the training of the crowd counting recognition model also includes model accuracy evaluation: reading the input images in the test set, assuming that the total number of images in the test set is n, performing the inference process with the current parameters to obtain the counting result of each image; calculating the prediction result {M i |i=1,2,…,n} and the true result {N i The mean absolute error and mean square error of |i=1,2,…,n} are calculated. The accuracy of the model in this round is compared with the historical best accuracy. If the current accuracy is better, the current weight is saved and the historical best accuracy value is updated.

[0015] Furthermore, the extraction of feature maps includes: inputting the image into the backbone network to extract the feature map; upsampling the feature map of the last layer of the backbone network; adding it bit by bit with the result of the second-to-last layer, and obtaining the final feature map after one convolution.

[0016] Furthermore, the prediction point coordinates and confidence calculation include: inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively, and performing ReLU on the output results to obtain the predicted point coordinates and confidence respectively; generating M predicted point coordinates as the final result according to the number of anchor points, and selecting an appropriate number of anchor points to ensure that M is greater than the number of point annotations N.

[0017] Furthermore, the loss function calculation includes: obtaining the scale balance weight of the point annotation; using the Hungarian algorithm to perform a one-to-one matching between the predicted point and the point annotation, assuming that the point annotation with the subscript i after matching corresponds to the predicted point with the subscript ξ(i); using the cross entropy function to calculate the classification loss L cls , using Euclidean distance to represent the sample scale balanced positioning loss L loc , the overall loss function L is: L = L cls +λ2L loc .

[0018] Furthermore, the scale balance weight of the point annotation is obtained, including: for each point annotation p i , calculate the distance information between it and all other points; select the k points with the smallest distance to the point, and calculate the sum of the distances d of these k points i , the distances of all points {d i |i=1,2,…,N} is normalized to obtain the scale balance weight {w i |i=1,2,…,N}.

[0019] In a second aspect, the present invention provides a crowd counting system based on a point-to-point framework, comprising:

[0020] An acquisition module, used for acquiring an image to be identified;

[0021] The recognition module is used to process the acquired images using a pre-trained crowd counting recognition model to obtain crowd counting recognition results. Training the crowd counting recognition model includes: inputting the training set images into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head to obtain the predicted point coordinates and confidence levels, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0022] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the crowd counting method based on the point-to-point framework as described in the first aspect is implemented.

[0023] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the crowd counting method based on the point-to-point framework as described in the first aspect.

[0024] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the crowd counting method based on the point-to-point framework as described in the first aspect.

[0025] Explanation of terms:

[0026] Crowd counting: The task of estimating the number of people in an image or video using computer vision techniques.

[0027] Object detection: Identifying objects in images or videos and determining their exact locations.

[0028] Density map estimation: Density map estimation is a method in crowd counting that estimates the number of people by assigning a density value to each pixel in the image to reflect the crowd density at that location.

[0029] VGG16: A deep convolutional neural network proposed by the Visual Geometry Group at the University of Oxford. It consists of 16 convolutional layers, including 13 convolutional layers and 3 fully connected layers. This network is widely used in image recognition and computer vision tasks.

[0030] Hungarian algorithm: An algorithm for solving the optimal matching problem of bipartite graphs, proposed by Hungarian mathematicians Harold Kuhn and James Munkres in 1955. The goal of the algorithm is to find a perfect matching with minimum weight, where each vertex can only be matched once and every edge belongs to the matching.

[0031] Beneficial effects of the present invention: A new sample scale balance loss function is proposed. This loss function uses the point annotation information in the data set to estimate the sample size information without introducing additional annotation workload, thereby calculating the influence weight of each annotation point on the final loss function, thereby improving the model's ability to detect targets of different scales.

[0032] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 This is a flowchart of the method for predicting the position of each target in a crowd based on an input image as described in the background technology.

[0035] Figure 2 This is a flow chart of the crowd counting recognition model training used in the crowd counting method based on the point-to-point framework described in an embodiment of the present invention.

[0036] Figure 3 This is a framework diagram of the working principle of the model described in an embodiment of the present invention.

[0037] Figure 4 This is a workflow diagram of the feature extraction module of the crowd counting and recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0039] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0040] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0041] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0042] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise contradictory.

[0043] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0044] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0045] Crowd counting is a key research task in crowd behavior analysis, aiming to accurately estimate the number of people in an image or video using computer vision techniques. This technology provides a powerful tool for a variety of fields, including public safety, urban planning, business decision-making, event management, traffic optimization, and social science research. By monitoring and understanding crowd density, this technology can provide real-time data support for urban management, business operations, and social science research, promoting more effective decision-making and planning. However, for some more complex crowd behavior analysis tasks, simply counting objects in an image is insufficient. For example, in tasks such as crowd tracking, crowd behavior prediction, and anomaly detection, in-depth analysis requires the location information of individuals within the crowd. This present invention, based on a point-to-point crowd counting framework and incorporating a sample-scale balanced loss function, fully utilizes point annotation information with a low annotation burden. The Hungarian algorithm achieves precise matching between predicted and annotated points, thereby guiding the model training process. Incorporating a sample-scale balanced loss function, the impact of samples of different scales on the loss function is balanced, improving the model's adaptability to scale changes. While achieving high counting accuracy, this present invention also enables the ability to predict the location of individuals within a crowd, providing a foundation for complex crowd behavior analysis.

[0046] Example 1

[0047] In this embodiment 1, a crowd counting system based on a point-to-point framework is first provided, comprising: an acquisition module for acquiring an image to be identified; a recognition module for processing the acquired image using a pre-trained crowd counting recognition model to obtain a crowd counting recognition result; wherein, training the crowd counting recognition model comprises: inputting the training set images into a backbone network to extract feature maps; inputting the feature maps into a three-layer convolutional network of a coordinate prediction head and a confidence calculation head to obtain predicted point coordinates and confidences, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0048] In this embodiment 1, the above-mentioned system is used to implement a crowd counting method based on a point-to-point framework, including: a practical acquisition module acquires an image to be identified; a practical identification module processes the acquired image using a pre-trained crowd counting and recognition model to obtain a crowd counting and recognition result; wherein, training the crowd counting and recognition model includes: inputting the training set image into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively to obtain the predicted point coordinates and confidence, respectively; and calculating the loss function by using the automatic derivation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters according to the obtained loss value, and updating the model parameters.

[0049] The training of the crowd counting recognition model also includes model accuracy evaluation: reading the input images in the test set, assuming that the total number of images in the test set is n, performing the inference process with the current parameters to obtain the counting results of each image; calculating the prediction results {M i |i=1,2,…,n} and the true result {N i The mean absolute error and mean square error of |i=1,2,…,n} are calculated. The accuracy of the model in this round is compared with the historical best accuracy. If the current accuracy is better, the current weight is saved and the historical best accuracy value is updated.

[0050] The extraction of feature maps includes: inputting the image into the backbone network to extract the feature map; upsampling the feature map of the last layer of the backbone network; adding it bit by bit with the result of the second-to-last layer, and obtaining the final feature map after a convolution.

[0051] The calculation of predicted point coordinates and confidence includes: inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively, and performing ReLU on the output results to obtain the predicted point coordinates and confidence respectively; generating M predicted point coordinates as the final result based on the number of anchor points, and selecting an appropriate number of anchor points to ensure that M is greater than the number of point annotations N.

[0052] The loss function calculation includes: obtaining scale balance weights of point labels; using a Hungarian algorithm to perform one-to-one matching between predicted points and point labels, assuming that the subscript of the predicted point corresponding to the point label with subscript i after matching is ξ(i); using a cross-entropy function to calculate a classification loss L cls , using a Euclidean distance to represent a sample scale balance positioning loss L loc , and an overall loss function L is L = L cls + λ2L loc .

[0053] The scale balance weights of the point labels are obtained, including: for each point label p i , distance information with all other points is calculated; k points with the smallest distance to the point are selected, and the sum of distances d i of the k points is calculated; distances {d i |i = 1, 2, …, N} of all points are normalized to obtain scale balance weights {w i |i = 1, 2, …, N}.

[0054] Embodiment 2

[0055] Embodiment 2 provides a crowd counting system, which first performs configuration work of related links, including installation of a Linux operating system, a development environment of Python 3.7 (and above versions), and a deep framework of PyTorch 1.5 (and above versions). Since the algorithm used in this embodiment is a model algorithm based on deep learning, it is recommended to perform a model training process in a GPU environment, and it is necessary to install a GPU version of Pytorch 1.5 (and above versions) and a corresponding version of CUDA parallel computing architecture.

[0056] The basic framework of the training method of the model used in the crowd counting of this embodiment 2 is shown in Figure 2 , and the most core steps in the process are a result inference part and a loss function calculation part for processing annotation information.

[0057] The reasoning process of the crowd counting task usually uses a pre-trained backbone network to extract features and then perform some corresponding processing methods, such as density regression or generating detection boxes. The processing of annotation information is also a key part, because simple number annotation cannot provide sufficient and effective information for parameter optimization; and because the application scenarios of crowd counting tasks are usually high-flow and high-density, complex annotation methods will bring a lot of overhead to the annotation task. Crowd counting usually uses point annotation, that is, the coordinates of each target center point are used to represent the target position, without additional size information. If point annotation is directly used to calculate the loss function during training, the model will also be difficult to fit. Therefore, different methods will also adopt different processing methods. For example, the density regression-based method will use a Gaussian geometric adaptive kernel to process each person's point annotation information, and then superimpose these Gaussian distributions to form a density map; while the detection-based method will generate pseudo boxes based on the point annotation to guide training; the point-to-point framework used in this embodiment also needs to use a matching algorithm such as the Hungarian algorithm to achieve a one-to-one matching between the annotation point and the prediction point before calculating the loss function.

[0058] In this embodiment, a new sample scale balance loss function is designed based on the point-to-point crowd counting framework. The overall framework of the model is as follows: Figure 3 As shown in the figure. During inference, the model first uses a pretrained backbone network to extract feature vectors, which are then fed into the candidate point prediction and confidence calculation modules to obtain the final results. During training, the loss function calculation and parameter update process are required. This method also requires processing the annotation information before calculating the loss function. This processing involves a one-to-one match with the predicted point coordinates using the Hungarian algorithm, and calculating the scale weight corresponding to each target based on the relative position information of the point coordinates. After processing, the above information is combined to calculate the loss function and update the model parameters.

[0059] This method uses the first thirteen convolutional layers of VGG-16_bn as the backbone network to extract depth information. The extraction process is as follows Figure 4 As shown, assuming that the input image size is H×W, the final feature obtained after the backbone network is Figure 3 and features Figure 4 The sizes are H / 8×W / 8 and H / 16×W / 16 respectively. Then the nearest neighbor interpolation algorithm is used to interpolate the features. Figure 4 After the spatial resolution is doubled, the feature Figure 3 The bits are added bit by bit and then passed through a 3×3 convolutional network to obtain the final output feature map.

[0060] After obtaining the feature map generated by the backbone network, the feature map is input into the prediction module for prediction of the candidate points and calculation of the confidence. The prediction of the candidate points and the calculation of the confidence belong to different branches respectively, but the network structures adopted by the two branches are the same, and both of them are obtained through a three-layer convolutional neural network and then through a ReLU activation function to obtain the final result.

[0061] The loss function L of the model mainly includes two parts: a classification loss L cls and a modified sample size balanced positioning loss L loc . Assuming that M prediction points are generated for a given input image, their coordinates and confidence are represented as Assuming that the annotation information has N labeled points {p i = (x i , y i ) | i = 1, 2, …, N}. In order to ensure the normal calculation process, the training process will ensure that the number of candidate points M is greater than the number of labeled points N by controlling the number of anchor points. The hyperparameters in the loss function are represented by λ.

[0062] In order to perform the next calculation, the Hungarian algorithm is used to one-to-one match the prediction points and the labeled points. Assuming that after matching, the mapping of the labeled points to the prediction points is ξ, that is, the label of the prediction point corresponding to the labeled point with number i is ξ(i); and when i is greater than N and less than or equal to M, different i correspond to different unmatched prediction points.

[0063] The classification loss L cls is calculated by the cross-entropy function, and the calculation formula is as shown in formula 1:

[0064]

[0065] Before calculating the positioning loss, the sample size balance weight needs to be calculated. For N labeled points, their weights are {w i | i = 1, 2, …, M}. The size of the weight is determined by the size estimation of the sample point, and the size of the sample point is represented by calculating the sum of the distances of the top-k neighbor points of the sample point. Assuming that the jth neighbor point of the sample point i is f(i, j), the calculation formula of the sample size balance weight is:

[0066]

[0067]

[0068] And the sample size balance positioning loss can be represented as:

[0069]

[0070] After the calculation of the classification loss and the positioning loss is completed, the loss function L can be represented as follows:

[0071] L = L cls + λ2L loc #(5)

[0072] In this embodiment, MAE (Mean Absolute Error) and MSE (Mean Squared Error) are used as indicators for evaluating the accuracy of the model. After the parameter update is completed, the model is tested on the test set and the MAE and MSE are calculated. If the current accuracy indicator is better than the historical optimal indicator, the historical optimal indicator is updated and the current model weight is saved.

[0073] In this embodiment, the following processes are mainly included: image feature extraction, candidate point coordinate and confidence prediction, candidate point label point matching, sample scale balance weight calculation, loss function calculation, weight update, and test evaluation. Next, the algorithm process will be described from the input, output, and execution steps of the algorithm.

[0074] In the model training, the input of the algorithm includes: crowd counting dataset: contains images and corresponding point label information in the training set and test set. The hyperparameters required in the model training process include λ1, λ2 in the loss function, the number of nearest points k required for calculating the sample scale balance weight, the learning rate lr of the model, the batch size batch_size of the training, and the iteration number epoch of the model. The output of the algorithm includes: the model accuracy evaluation indicator of each training round. The predicted target point coordinate position. The model weight file with the highest accuracy and the last updated model weight file.

[0075] In this embodiment, the specific training algorithm process of the crowd counting model includes five steps: feature extraction, predicted point coordinate and confidence calculation, loss function calculation, model parameter update, and model accuracy evaluation.

[0076] Feature extraction includes: step 1-1: input the image into the backbone network to extract the feature map; step 1-2: upsample the feature map of the last layer of the backbone network; step 1-3: add the results bit by bit with the second to last layer, and get the final feature map after one convolution.

[0077] Predicted point coordinate and confidence calculation includes: step 2-1: input the feature map into the three-layer convolution network of the coordinate prediction head and the confidence calculation head respectively; step 2-2: perform ReLU on the output results of the previous step to get the predicted point coordinates and the confidence respectively; step 2-3: generate M predicted point coordinates as the final result according to the anchor point number, and select appropriate anchor point number to ensure that M is greater than the point label number N.

[0078] The loss function calculation includes:

[0079] Step 3-1: Obtain the scale balance weight of the point annotation;

[0080] Step 3-1-1: For each point, label p i , calculate its distance information to all other points;

[0081] Step 3-1-2: Select the k points with the smallest distance to the point and calculate the sum of the distances d of these k points i , the calculation method is as shown in formula (2);

[0082]

[0083] Step 3-1-3: Set the distances of all points {d i |i=1,2,…,N} is normalized to obtain the scale balance weight {w i |i=1,2,…,N}, the calculation method is expressed by formula (3);

[0084]

[0085] Step 3-2: Use the Hungarian algorithm to perform a one-to-one match between the predicted points and the point annotations. Assume that after matching, the point annotation with the subscript i corresponds to the predicted point with the subscript ξ(i).

[0086] Step 3-3: Calculate the classification loss L using the cross entropy function cls , the calculation method is expressed by formula (1);

[0087]

[0088] Step 3-4: Use Euclidean distance to represent the sample scale and balance the positioning loss L loc , the calculation method is shown in formula (4);

[0089]

[0090] Step 3-5: Calculate the overall loss function L. The calculation method is shown in formula (5).

[0091] L=L cls +λ2L loc #(5)

[0092] Model weight updates include:

[0093] Step 4-1: Use the automatic derivation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the loss value obtained in steps 3-5;

[0094] Step 4-2: Update the model parameters using the Adam optimization algorithm based on the gradient obtained in step 4-1;

[0095] Model accuracy evaluation includes:

[0096] Step 5-1: Read the input images in the test set. Assuming the total number of images in the test set is n, perform the inference process with the current parameters to obtain the count results for each image.

[0097] Step 5-2: Calculate the prediction result {M i |i=1,2,…,n} and the true result {N i Mean absolute error and mean square error of |i=1,2,…,n};

[0098] Step 5-3: Compare the accuracy of the current model with the historical best accuracy. If the current model is better, save the current weight and update the historical best accuracy value.

[0099] Step 5-4: If the maximum number of rounds is not reached, return to step 1; otherwise, end the training.

[0100] Example 3

[0101] This embodiment 3 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the above-mentioned method for counting people based on the point-to-point framework is implemented. The method includes:

[0102] Obtaining an image to be recognized;

[0103] The acquired images are processed using a pre-trained crowd counting and recognition model to obtain crowd counting and recognition results. Training the crowd counting and recognition model includes: inputting the training set images into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head to obtain the predicted point coordinates and confidence levels, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0104] Example 4

[0105] This embodiment 4 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the above-mentioned crowd counting method based on the point-to-point framework, the method comprising:

[0106] Obtaining an image to be recognized;

[0107] The acquired images are processed using a pre-trained crowd counting and recognition model to obtain crowd counting and recognition results. Training the crowd counting and recognition model includes: inputting the training set images into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head to obtain the predicted point coordinates and confidence levels, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0108] Example 5

[0109] This embodiment 5 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the above-mentioned method for counting people based on a point-to-point framework, the method including:

[0110] Obtaining an image to be recognized;

[0111] The acquired images are processed using a pre-trained crowd counting and recognition model to obtain crowd counting and recognition results. Training the crowd counting and recognition model includes: inputting the training set images into the backbone network to extract feature maps; inputting the feature maps into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head to obtain the predicted point coordinates and confidence levels, respectively; and calculating the loss function using the automatic differentiation mechanism of the PyTorch framework to calculate the gradient direction of the model parameters based on the obtained loss value and update the model parameters.

[0112] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0116] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A crowd counting method based on a point-to-point framework, characterized in that: include: Obtaining an image to be recognized; The acquired image is processed using a pre-trained crowd counting recognition model to obtain a crowd counting recognition result; wherein, training the crowd counting recognition model includes: inputting the training set image into the backbone network to extract the feature map; inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively to obtain the predicted point coordinates and confidence respectively; loss function calculation, using the automatic derivation mechanism of the PyTorch framework, calculates the gradient direction of the model parameters according to the obtained loss value, and updates the model parameters; predicted point coordinates and confidence calculation include: inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively, and performing ReLU on the output results to obtain the predicted point coordinates and confidence respectively; generating M predicted point coordinates as the final result according to the number of anchor points, and selecting an appropriate number of anchor points to ensure that M is greater than the number of point annotations N; loss function calculation includes: obtaining the scale balance weight of the point annotation; using the Hungarian algorithm, one-to-one matching of the predicted point with the point annotation with the subscript i after matching, assuming that the subscript of the predicted point corresponding to the point annotation with the subscript i is ξ(i); using the cross entropy function to calculate the classification loss L cls , using Euclidean distance to represent the sample scale balanced positioning loss L loc , the overall loss function L is: L = L cls +λ2L loc .

2. The method for counting people based on a point-to-point framework according to claim 1, characterized in that: Training the crowd counting and recognition model also includes evaluating the model accuracy: reading input images from a test set, assuming that the total number of images in the test set is n, performing an inference process with current parameters to obtain a count result for each image; According to the annotation information, the prediction results {M i |i=1,2,…,n} and the true result {N i The mean absolute error and mean square error of |i=1,2,…,n} are calculated. The accuracy of the model in this round is compared with the historical best accuracy. If the current accuracy is better, the current weight is saved and the historical best accuracy value is updated.

3. The crowd counting method based on a point-to-point framework according to claim 1, characterized in that: The extraction of feature maps includes: inputting the image into the backbone network to extract the feature map; upsampling the feature map of the last layer of the backbone network; adding it bit by bit with the result of the second-to-last layer, and then performing a convolution to obtain the final feature map.

4. The method for counting people based on a point-to-point framework according to claim 1, characterized in that: Get the scale balance weight of the point annotation, including: for each point annotation p i , calculate the distance information between it and all other points; select the k points with the smallest distance to the point, and calculate the sum of the distances d of these k points i , the distances of all points {d i |i=1,2,…,N} is normalized to obtain the scale balance weight {w i |i=1,2,…,N}.

5. A crowd counting system based on a point-to-point framework, characterized in that, include: An acquisition module, used for acquiring an image to be identified; The recognition module is used to process the acquired image using a pre-trained crowd counting recognition model to obtain the crowd counting recognition result; wherein, the training of the crowd counting recognition model includes: inputting the training set image into the backbone network to extract the feature map; inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively to obtain the predicted point coordinates and confidence respectively; calculating the loss function, using the automatic derivation mechanism of the PyTorch framework, calculating the gradient direction of the model parameters according to the obtained loss value, and updating the model parameters; predicting the point coordinates and confidence The calculation includes: inputting the feature map into the three-layer convolutional network of the coordinate prediction head and the confidence calculation head respectively, and performing ReLU on the output results to obtain the predicted point coordinates and confidence respectively; generating M predicted point coordinates as the final result according to the number of anchor points, and selecting the appropriate number of anchor points to ensure that M is greater than the number of point annotations N; the loss function calculation includes: obtaining the scale balance weight of the point annotation; using the Hungarian algorithm, one-to-one matching of the predicted point and the point annotation, assuming that the point annotation with the subscript i after matching corresponds to the subscript ξ(i) of the predicted point; using the cross entropy function to calculate the classification loss L cls , using Euclidean distance to represent the sample scale balanced positioning loss L loc , the overall loss function L is: L = L cls +λ2L loc .

6. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the crowd counting method based on the point-to-point framework according to any one of claims 1 to 4 is implemented.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the crowd counting method based on the point-to-point framework as described in any one of claims 1 to 4.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the crowd counting method based on the point-to-point framework as described in any one of claims 1 to 4.

Images