A crowd counting method and device based on online knowledge distillation network

Abstract

The present invention discloses a crowd counting method and device based on an online knowledge distillation network. First, a crowd counting image is obtained and a corresponding label is generated, and the image and the label are preprocessed. Then, an online knowledge distillation network is constructed, including a shared shallow module, a teacher strong semantic branch and a student refinement branch. The shared shallow module is used to extract the base features of the image. The teacher strong semantic branch captures the robust semantic features of the image based on pre-trained parameters. The student refinement branch reduces the parameter amount by channel reduction to improve the reasoning speed of the network. Then the image and the label are input into the network for training. Finally, the image is sent to the trained student refinement branch to output the number of people. Compared with the traditional feature learning strategy, the present invention can realize single-stage network learning, that is, the training of the student network can be completed in a single time, which greatly reduces the training cost of the network. It has important application prospects in many fields such as transportation, agriculture and forestry.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to a crowd counting method and device based on an online knowledge distillation network. Background Art

[0002] Crowd counting aims to count dense crowds in complex scenes, which is a very important visual task in the field of artificial intelligence. This can help managers of public places to timely analyze the density distribution of crowds in scenes. When the crowd is too dense, early warning information can be issued through inference results to avoid congestion. It has a wide range of applications in transportation, medicine, forestry and other fields.

[0003] In real application scenarios, computing resources are usually limited, such as mobile phones, development boards and other terminals. This makes it difficult to directly apply networks with higher accuracy. Knowledge distillation is an effective lightweight strategy, but currently a two-stage distillation method is usually adopted, that is, first learning a teacher network with large parameters and good performance, and then using it to distill a student network with small parameters. When faced with a large amount of data input, it consumes a lot of training time, which is obviously not conducive to rapid deployment and also causes a waste of resources.

[0004] To this end, this paper proposes a single-stage crowd counting distillation method, which effectively reduces the training time by constructing a teacher strong semantic branch and a student refined branch to approximate the teacher and student networks. At the same time, a new knowledge distillation strategy between features is designed to improve the efficiency of knowledge transfer. Experimental results show that this method is superior to the traditional two-stage method in terms of both time and effect.

[0005] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides an efficient crowd counting method and device based on an online knowledge distillation network.

[0006] Technical solution: The crowd counting method based on online knowledge distillation network described in the present invention includes:

[0007] S1: Obtain crowd counting images and generate corresponding labels;

[0008] S2: Preprocess images and labels;

[0009] S3: Build an online knowledge distillation network, including a shared shallow module, a teacher strong semantic branch, and a student refined branch. The shared shallow module is used to extract the basic features of the image. The teacher strong semantic branch captures the robust semantic features of the image based on pre-trained parameters. The student refined branch reduces the number of parameters through channel reduction to improve the reasoning speed of the network.

[0010] S4: Construct knowledge distillation strategy, including feature conversion layer, response distillation, intra-feature distillation and inter-feature distillation;

[0011] S5: Input images and labels into the network for training;

[0012] S6: Send the image to the trained student refinement branch and output the number of people.

[0013] Furthermore, the S2 specifically includes:

[0014] S2-0: randomly crop the image;

[0015] S2-1: Randomly add noise to the image based on the predicted probability;

[0016] S2-2: Randomly flip the image horizontally with a preset probability.

[0017] Furthermore, the shared shallow modules in S3 specifically include:

[0018] A shared shallow feature extractor, formed by stacking several convolutional layers, is used to extract basic features of the input image;

[0019] The maximum pooling layer is used for spatial downsampling of the base features.

[0020] Furthermore, the teacher's strong semantic branch in S3 specifically includes:

[0021] A first convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the base feature;

[0022] A first maximum pooling layer for spatial downsampling of features of the first convolutional block feature extractor;

[0023] A second convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the output features of the first maximum pooling layer;

[0024] A second maximum pooling layer for spatial downsampling of features of the second convolutional block feature extractor;

[0025] A third convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the output features of the second maximum pooling layer;

[0026] Furthermore, the student refining branch in S3 specifically includes:

[0027] The student refining branch convolutional feature extractor has the same structure as the teacher strong semantic branch extractor, with the number of channels reduced to 1 / n (n is a positive integer), which is used for the student refining branch to extract basic convolutional features.

[0028] Furthermore, the feature conversion layer in S4 specifically includes:

[0029] A feature conversion layer, composed of a convolution layer with a convolution kernel of 1×1, is used to convert the channels of the output features of the teacher strong semantic branch in S3 at different stages to generate the same dimension as the output feature channels of the corresponding stage of the student refinement branch in S3;

[0030] Furthermore, the response distillation in S4 specifically includes:

[0031] L s =1-SSIM(t N ,s′ N ),

[0032] in

[0033]

[0034] In the formula, and Respectively represent the input t N and s′ N The average value of all pixels, and Respectively represent the input t N , s′ N The variance of Represents two input t N and s′ N The variance of , c1 and c2 represent constants.

[0035] Furthermore, the characteristic internal distillation in S4 specifically includes:

[0036]

[0037] where p(s′ i ) and p(t i ) respectively represent the output features of the teacher's strong semantic branch and the student's refined branch at different stages.

[0038] Furthermore, the feature-to-feature distillation in S4 specifically includes:

[0039]

[0040] in

[0041]

[0042] In the formula represents the element-wise product, N represents the number of outputs of the teacher’s strong semantic branching stage, and t i ∈R c×h×wrepresents the output of the network features of the teacher network at the i-th stage, c, h, w represent the number of channels, height and width of the features respectively, and t i After dimension conversion, we get σ(·) represents the sigmoid operation, Calculation and The same is obtained, and then the relationship matrix of the student network at different stages is obtained.

[0043] Furthermore, the S5 specifically includes:

[0044] S5-0: Divide the images in S1 into a training set and a validation set;

[0045] S5-1: Initialize the learning rate of the network;

[0046] S5-2: Input the training image into the online knowledge distillation network and calculate the network loss through the following loss,

[0047]

[0048] in and They represent the mean square error loss functions of the teacher's strong semantic branch and the student's refined branch, and α, β, and δ represent hyperparameters.

[0049] S5-3: Use the Adam optimizer to optimize the network parameters, save the network parameters with the smallest loss in the validation set, and complete the network training;

[0050] Furthermore, the different stages of the teacher's strong semantic branch specifically include:

[0051] The different stages of the teacher’s strong semantic branch represent the features of the first max pooling layer, the second max pooling layer, and the third convolutional block feature extractor, respectively.

[0052] The crowd counting device based on online knowledge distillation of the present invention comprises a processor and a computer program stored in a memory and executable on the processor, and the above method is implemented when the processor executes the program.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention designs a teacher strong semantic branch and a student refinement branch. The teacher strong semantic branch loads a pre-trained model to better capture the semantic information of the image, and the student branch greatly reduces the number of network parameters by reducing the number of channels. This design avoids the training of the traditional two-stage distillation network and greatly reduces the computational overhead.

[0055] 2. The present invention designs a new inter-feature distillation loss, which utilizes the correlation between features to better capture the relationship between features at different stages and improve the efficiency of knowledge transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of a crowd counting method based on online knowledge distillation network in the present invention;

[0057] Figure 2 It is a framework diagram of the deep network model in the present invention. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Embodiment 1: This embodiment specifically provides a crowd counting method based on an online knowledge distillation network, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0060] S1: Obtain crowd counting images and generate corresponding labels.

[0061] S2: Preprocess images and labels.

[0062] The pre-processing methods include:

[0063] S2-0: randomly crop the image;

[0064] S2-1: Randomly add noise to the image based on the predicted probability;

[0065] S2-2: Randomly flip the image horizontally with a preset probability.

[0066] S3: Construct an online knowledge distillation network, including a shared shallow module, a teacher strong semantic branch, and a student refinement branch. The shared shallow module is used to extract the basic features of the image, which includes

[0067] A shared shallow feature extractor, formed by stacking several convolutional layers, is used to extract basic features of the input image;

[0068] The maximum pooling layer is used for spatial downsampling of the base features.

[0069] The teacher strong semantic branch captures the robust semantic features of the image based on pre-trained parameters.

[0070] A first convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the base feature;

[0071] A first maximum pooling layer for spatial downsampling of features of the first convolutional block feature extractor;

[0072] A second convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the output features of the first maximum pooling layer;

[0073] A second maximum pooling layer for spatial downsampling of features of the second convolutional block feature extractor;

[0074] The third convolutional block feature extractor is formed by stacking a plurality of convolutional layers and is used to extract basic convolutional features from the output features of the second maximum pooling layer.

[0075] The student refinement branch reduces the number of parameters by decreasing the number of channels, thereby improving the reasoning speed of the network.

[0076] The student refining branch convolutional feature extractor has the same structure as the teacher strong semantic branch extractor, with the number of channels reduced to 1 / n (n is a positive integer), which is used for the student refining branch to extract basic convolutional features.

[0077] S4: Construct a knowledge distillation strategy, including feature conversion layer, response distillation, intra-feature distillation, and inter-feature distillation.

[0078] The feature conversion layer is composed of a convolution layer with a convolution kernel of 1×1, and is used to convert the channels of the output features of the teacher strong semantic branch in S3 at different stages to generate the same dimension as the output feature channels of the corresponding stage of the student refinement branch in S3;

[0079] Among them, response distillation includes

[0080] L s =1-SSIM(t N ,s′ N ),

[0081] in

[0082]

[0083] In the formula, and Respectively represent the input t N and s′ N The average value of all pixels, and Respectively represent the input t N , s′ N The variance of Represents two input t N and s′ N The variance of , c1 and c2 represent constants.

[0084] Among them, characteristic internal distillation specifically includes:

[0085]

[0086] where p(s′ i ) and p(t i ) respectively represent the output features of the teacher's strong semantic branch and the student's refined branch at different stages.

[0087] Furthermore, the feature-to-feature distillation in S4 specifically includes:

[0088]

[0089] in

[0090]

[0091] In the formula represents the element-wise product, N represents the number of outputs of the teacher’s strong semantic branching stage, and t i ∈R c×h×w represents the output of the network features of the teacher network at the i-th stage, c, h, w represent the number of channels, height and width of the features respectively, and t i After dimension conversion, we get σ(·) represents the sigmoid operation, Calculation and The same is obtained, and then the relationship matrix of the student network at different stages is obtained.

[0092] S5: Input images and labels into the network for training.

[0093] The training steps are as follows:

[0094] S5-0: Divide the images in S1 into a training set and a validation set;

[0095] S5-1: Initialize the learning rate of the network;

[0096] S5-2: Input the training image into the online knowledge distillation network and calculate the network loss through the following loss,

[0097]

[0098] in and They represent the mean square error loss functions of the teacher's strong semantic branch and the student's refined branch, and α, β, and δ represent hyperparameters.

[0099] S5-3: Use the Adam optimizer to optimize the network parameters, save the network parameters with the smallest loss in the validation set, and complete the network training.

[0100] S6: Send the image to the trained student refinement branch and output the number of people.

[0101] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A crowd counting method based on online knowledge distillation network, characterized in that: The following steps are involved: S1: Obtain crowd counting images and generate corresponding labels; S2: Preprocess images and labels; S3: Build an online knowledge distillation network, including a shared shallow module, a teacher strong semantic branch, and a student refined branch. The shared shallow module is used to extract the basic features of the image. The teacher strong semantic branch captures the robust semantic features of the image based on pre-trained parameters. The student refined branch reduces the number of parameters through channel reduction to improve the reasoning speed of the network. S4: Construct knowledge distillation strategy, including feature conversion layer, response distillation, intra-feature distillation and inter-feature distillation; S5: Input images and labels into the network for training; S6: Send the image to the trained student refinement branch and output the number of people; The S2 specifically includes: S2-0: randomly crop the image; S2-1: Randomly add noise to the image based on the predicted probability; S2-2: Randomly flip the image horizontally with a preset probability; The shared shallow modules in S3 specifically include: A shared shallow feature extractor, formed by stacking several convolutional layers, is used to extract basic features of the input image; A maximum pooling layer for spatial downsampling of the base features; The teacher's strong semantic branch in S3 specifically includes: The first convolutional block feature extractor is formed by stacking several convolutional layers and is used for base feature extraction of basic convolutional features; A first maximum pooling layer for spatial downsampling of features of the first convolutional block feature extractor; A second convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the output features of the first maximum pooling layer; A second maximum pooling layer for spatial downsampling of features of the second convolutional block feature extractor; A third convolutional block feature extractor, formed by stacking a plurality of convolutional layers, for extracting basic convolutional features from the output features of the second maximum pooling layer; The student refining branches in S3 specifically include: The student refining branch convolution feature extractor has the same structure as the teacher strong semantic branch extractor, but the number of channels is reduced to 1 / n, where n is a positive integer, and is used for extracting basic convolution features from the student refining branch; The feature conversion layer in S4 specifically includes: A feature conversion layer, composed of a convolution layer with a convolution kernel of 1×1, is used to convert the channels of the output features of the teacher strong semantic branch in S3 at different stages to generate the same dimension as the output feature channels of the corresponding stage of the student refinement branch in S3; The response distillation in S4 specifically includes: L s =1-SSIM(t N ,s′ N ), in In the formula, and Respectively represent the input t N and s′ N The average value of all pixels, and Respectively represent the input t N , s′ N The variance of Represents two input t N and s′ N The variance of , c1 and c2 represent constants; The characteristic internal distillation in S4 specifically includes: where p(s′ i ) and p(t i ) respectively represent the output features of the teacher's strong semantic branch and the student's refined branch at different stages; The inter-feature distillation in S4 specifically includes: in In the formula represents the element-wise product, N represents the number of outputs of the teacher’s strong semantic branching stage, and t i ∈R c×h×w represents the output of the network features of the teacher network at the i-th stage, c, h, w represent the number of channels, height and width of the features respectively, and t i After dimension conversion, we get σ(·) represents the sigmoid operation, Calculation and The same is obtained, and then the relationship matrix of the student network at different stages is obtained.

2. The crowd counting method based on online knowledge distillation network according to claim 1 is characterized in that: The S5 specifically includes: S5-0: Divide the images in S1 into a training set and a validation set; S5-1: Initialize the learning rate of the network; S5-2: Input the training image into the online knowledge distillation network and calculate the network loss through the following loss, in and They represent the mean square error loss functions of the teacher's strong semantic branch and the student's refined branch, α, β, δ represent hyperparameters, S5-3: Use the Adam optimizer to optimize the network parameters, save the network parameters with the smallest loss in the validation set, and complete the network training.

3. The crowd counting method based on online knowledge distillation network according to claim 2 is characterized in that: The different stages of the teacher's strong semantic branch specifically include: The different stages of the teacher’s strong semantic branch represent the features of the first max pooling layer, the second max pooling layer, and the third convolutional block feature extractor, respectively.

4. An online knowledge distillation crowd counting device, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images