Target counting-oriented incremental learning network modeling method and device

Abstract

The present invention discloses a method and apparatus for modeling a class-incremental learning network for target counting, comprising: S1: obtaining a dataset of samples of different classes; S2: online augmenting the dataset samples; S3: constructing a class-incremental learning network for target counting, the class-incremental learning network comprising a feature extractor, a class-independent module, and a class-incremental module; and S4: training the class-incremental learning network using the augmented dataset samples to obtain a trained class-incremental learning network. The present invention enables online learning and prediction of density maps of targets of different classes, improving the practicality and applicability of the target counting network.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a method and device for modeling a class incremental learning network for target counting. Background Art

[0002] Object counting is a key application of computer vision, thanks to its widespread use in video surveillance, such as traffic flow management, pedestrian flow control, and public safety management. Crowd counting can count the total number of people within the camera's field of view to prevent crowds and minimize the spread of the virus.

[0003] Existing object counting models can only be trained under the assumption that all classes are fixed and the samples for all classes are known in advance. This means that existing models can only count or estimate the density of a fixed number of classes, such as pedestrians and cars. When new classes emerge, existing models are unable to learn, identify, and count them. To address this issue, previous approaches have designed a separate model for each class, but this leads to high deployment costs and waste of resources.

[0004] Incremental learning (also known as continuous learning) can imitate the ability of humans to continuously process continuous information flows in the real world, retaining or even integrating and optimizing old knowledge while absorbing new knowledge. Therefore, combining target counting with incremental learning can greatly improve the versatility of target counting in practical applications, and has a very broad application prospect. However, existing incremental learning methods are all aimed at category increments, such as image classification tasks and semantic segmentation tasks. So far, no incremental learning task of density regression has been found, that is, to increase the categories of density regression online and distinguish their categories at the same time. To this end, the present invention proposes a class incremental learning network modeling method for target counting. Summary of the Invention

[0005] Purpose of the invention: To address the problems existing in the prior art, the present invention provides a class incremental learning network modeling method for target counting, which completes the online learning and prediction of density maps of different classes of targets.

[0006] Technical solution: The target counting-oriented incremental learning network modeling method of the present invention includes:

[0007] S1: Obtain a dataset of different types of samples;

[0008] S2: Perform online augmentation on dataset samples;

[0009] S3: Constructing a class-incremental learning network for target counting, the class-incremental learning network includes a feature extractor, a class-independent module, and a class-incremental module, the feature extractor is connected to the class-independent module and the class-incremental module, respectively, the class-independent module is connected to the class-incremental module, the feature extractor is used to extract a basic feature vector, the class-independent module is used to learn based on the basic feature vector to obtain a consistent density map, the class-incremental module is used to perform classification based on the basic feature vector, and perform regression counting based on the classification result and the density map;

[0010] S4: Using the augmented data set samples to train the class incremental learning network to obtain a trained class incremental learning network.

[0011] Furthermore, the S2 specifically includes:

[0012] S2-1: Randomly scale the samples;

[0013] S2-2: Crop the samples at random positions;

[0014] S2-3: Randomly flip the sample;

[0015] S2-4: gamma correct the sample;

[0016] S2-5: Convert the sample into grayscale image.

[0017] Furthermore, the feature extractor in S3 is specifically the first 10 layers of a pre-trained VGG16 network.

[0018] Furthermore, the class-independent modules specifically include:

[0019] The first convolutional layer includes two 3×3 convolutional layers;

[0020] The mask layer includes four sequentially connected single convolutional layers and an upsampling layer. The four single convolutional layers are used to obtain the mask from the output of the first convolutional layer. The upsampling layer is used to perform bilinear interpolation upsampling on the mask to obtain a standard mask of the same size as the input sample.

[0021] The activation layer is used to first activate the standard mask through the Sigmoid function, then perform a Ones-like operation and subtract the activation feature output by the Sigmoid function to obtain the activation feature;

[0022] The attention layer includes a 3×3 convolutional layer, a ReLu function, and a 3×3 convolutional layer connected in sequence, which is used to obtain attention features based on activation features;

[0023] The second convolutional layer includes a splicing unit and a 3×3 convolutional layer. The splicing unit is used to splice the output of the first convolutional layer and the attention feature. The 3×3 convolutional layer is used to convolve the spliced ​​features to obtain a density map.

[0024] Furthermore, the class increment module includes:

[0025] A global average pooling layer, used to perform global average pooling on the basic feature vector;

[0026] The classifier is used to classify the input sample based on the output of the global average pooling layer;

[0027] The regression layer is used to regress the counts based on the classifier output and the density map to obtain the counts of different classes.

[0028] Furthermore, the S4 specifically includes:

[0029] S4-1: Divide the dataset of each class into training dataset and validation dataset;

[0030] S4-2: Randomly select a class of data from all classes of data sets for the following training;

[0031] S4-3: Set training parameters. If the current stage is the base training stage, randomly initialize the class incremental learning network. If the current stage is the incremental training stage, directly use the network parameters obtained from the previous training for initialization, and expand the dimension of the class incremental module so that its dimension can match the total number of classes of the previously trained classes and the current class. After expansion, the parameters of the new dimension are randomly initialized. The training for the first class is the base training stage, and the training for subsequent classes is the incremental training stage.

[0032] S4-4: Use the example set and the training dataset of the current class to train the class incremental learning network, and optimize the network by calculating the total loss;

[0033] S4-5: Testing the trained class incremental learning network using samples from a validation dataset, and saving the network parameters with the minimum error as the network parameters obtained from training the current class. The validation dataset includes validation datasets for the current class and previously trained classes.

[0034] S4-6: Select several samples that meet the preset conditions from the data set of the current class and add them to the example set;

[0035] S4-7: Return to step S4-2 and continue until the training of all class data sets is completed, and the final class incremental learning network with the target count is obtained.

[0036] Furthermore, the step of calculating the total loss in S4-4 includes:

[0037] S4-4-1: Calculate the mask loss L1 of the class-independent module:

[0038]

[0039] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the semantic segmentation map, 0 represents the background area, 1 represents the target area, and p i represents the pixel-level prediction of the mask layer of the class-independent module, σ( ) represents the Sigmoid activation function;

[0040] S4-4-2: Calculate the classification loss L2 in the class increment module:

[0041]

[0042] q i =σ(F cls )

[0043] F cls =FC(F1 avg )

[0044] Among them, FC represents the fully connected layer, F1 avg represents the output of the global average pooling layer of the incremental module, F cls represents the predicted output of the class increment module classifier, y i Represents the category labels for different categories, q i Represents the confidence of the classifier's classification prediction;

[0045] S4-4-3: Calculate the density map estimation loss L3 of the incremental module:

[0046]

[0047] Among them, B represents the batch size, Z i represents the true label of the density map, represents the prediction of the density map;

[0048] S4-4-4: Calculate the distillation loss L4 of the sample density map prediction of the previously trained class:

[0049]

[0050] in, Represent the density map of the trained class and the density map prediction of the current class respectively;

[0051] S4-4-5: Calculate the weighted sum of each loss as the total loss:

[0052] L=L1+L2+(1-λ)*L3+λ*L4

[0053] Here, λ represents a hyperparameter.

[0054] Furthermore, the S4-6 specifically includes:

[0055] S4-6-1: Based on the pre-set example set memory K, calculate the average number K / I allocated to the trained class i , I i Indicates the total number of trained classes;

[0056] S4-6-2: Remove the samples at the end of each class in the example set, that is, remove K / (I i -1)-K / I i samples;

[0057] S4-6-3: According to the herding algorithm, select the most representative K / I in the current class i samples are added to the example set.

[0058] Furthermore, after dividing the training data set and the validation data set in S4-1, negative samples are added to the training data set and the validation data set.

[0059] The target counting-oriented incremental learning network modeling device described in the present invention includes a processor and a computer program stored in a memory and runnable on the processor, and the processor implements the above method when executing the program.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0061] 1) This paper designs a class-incremental crowd counting network that can learn and predict density maps of different categories online. This can be applied to different fields, such as traffic pedestrian and vehicle statistics, yield prediction in agriculture, pest prediction, cell statistics in medicine, etc.

[0062] 2) The present invention designs a class-independent module that can capture consistent semantic information and use it as an auxiliary information to guide the learning of the class incremental module, thereby enhancing the feature expression ability of the network.

[0063] 3) The present invention designs a class increment module that can dynamically increment the number of categories and has the functions of category prediction and density regression. At the same time, the category serves as a guiding module to assist in the implementation of density regression. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1Schematic diagram of the process of the target counting-oriented incremental learning network modeling method provided by the present invention;

[0065] Figure 2 It is a structural diagram of the quasi-incremental learning network of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0067] This embodiment provides a target counting-oriented incremental learning network modeling method, such as Figure 1 As shown, the following steps are included:

[0068] S1: Obtain a dataset of samples of different classes.

[0069] The dataset may be a traffic camera video frame, a plant distribution map, a cell distribution map, or other maps that need to be counted. If it is a traffic camera video frame, the class here refers to the categories of vehicles, pedestrians, etc. in the map.

[0070] S2: Perform online augmentation on dataset samples.

[0071] Specifically include:

[0072] S2-1: Randomly scale the samples;

[0073] S2-2: Cropping the sample at random positions to a size of 400*400;

[0074] S2-3: randomly flip the sample with a probability of 0.5;

[0075] S2-4: Perform gamma correction on the sample with a correction coefficient range of 0.5-1.5 and a probability of 0.3;

[0076] S2-5: Convert the sample to grayscale with a probability of 0.1.

[0077] S3: Constructing a class-incremental learning network for object counting.

[0078] Among them, such as Figure 2 As shown, the class-incremental learning network includes a feature extractor, a class-independent module, and a class-incremental module. The feature extractor is connected to the class-independent module and the class-incremental module, respectively. The class-independent module is connected to the class-incremental module. The feature extractor is used to extract basic feature vectors. The class-independent module is used to learn based on the basic feature vectors to obtain a consistent density map. The class-incremental module is used to classify based on the basic feature vectors and perform regression counting based on the classification results and the density map. Each module is introduced in detail below.

[0079] The feature extractor is specifically the first 10 layers of the pre-trained VGG16 network, which is used to extract features from the input sample x and obtain the basic feature vector F1:

[0080] F1=F m (x)

[0081] Among them, F m represents the feature extractor function. The dimension of the input sample x is C×H×W, where C, H, and W represent the input channel, height, and width, respectively. The dimension of F1 is B×C1×(H / 8)×(W / 8), where B represents the batch size, C1 represents the output channel dimension, and H / 8 and W / 8 represent the height and width of the output feature.

[0082] Class-independent modules specifically include:

[0083] The first convolutional layer, including two 3×3 convolutional layers, is used to obtain the feature F2:

[0084] F2=Conv1(F1)

[0085] Among them, Conv1 represents the first convolutional layer function, and F2 dimension is B×C1×(H / 8)×(W / 8);

[0086] The mask layer includes four single convolutional layers and an upsampling layer connected in sequence. The four single convolutional layers are used to obtain the mask from the output of the first convolutional layer. The upsampling layer is used to perform bilinear difference upsampling on the mask to obtain a standard mask F with the same size as the input sample x. mask ;

[0087] M1=Mask1(F2), F mask =Up(M1)

[0088] Among them, Mask1 represents the function of 4 single convolutional layers, M1 represents its output, and the output dimension is B×1×(H / 8)×(W / 8). Up represents the upsampling layer, which upsamples M1 by 8 times through bilinear interpolation to obtain F mask , the output dimension is B×1×H×W;

[0089] The activation layer is used to first activate the standard mask through the Sigmoid function, then perform a Ones-like operation and subtract the activation feature output by the Sigmoid function to obtain the activation feature F3;

[0090]

[0091] Among them, σ(x) represents the Sigmoid activation function, Represents a Ones-like operation, that is, a tensor filled with all 1s of the same scalar value as the input x dimension.

[0092] The attention layer includes a 3×3 convolutional layer, a ReLu function, and a 3×3 convolutional layer connected in sequence, which is used to obtain the attention feature F4 based on the activation feature;

[0093] F4=conv2(θ(conv1(F3)))

[0094] Among them, conv1 represents the first 3×3 convolutional layer, θ(x) represents the ReLu function, and conv2 represents the second 3×3 convolutional layer;

[0095] The second convolutional layer includes a splicing unit and a 3×3 convolutional layer. The splicing unit is used to splice the output of the first convolutional layer and the attention feature. The 3×3 convolutional layer is used to convolve the spliced ​​features to obtain a density map F. out :

[0096] F out =Conv3(concat(F2,F4))

[0097] Among them, Conv3 represents the convolution layer, concat(,) represents the concatenation unit to concatenate the two along the channel dimension, and then F out It is sent to the regression layer of the incremental class module for density regression.

[0098] The class increment module specifically includes:

[0099] The global average pooling layer is used to perform global average pooling on the basic feature vector to obtain F1 avg ;

[0100] The classifier is used to classify the input sample based on the output of the global average pooling layer;

[0101] The regression layer is used to regress the counts based on the output of the classifier and the density map to obtain the counts E of different classes. i :

[0102] E i =F reg (argmax(F1 avg ),F out )

[0103] Among them, F reg Represents the regression layer function, which is used to regress the density map. The argmax() function can be used to regress the input vector F1 avg Determine the index of the maximum value, F reg Indicates global average pooling of F1, F out Density map representing the output of the class-agnostic module.

[0104] S4: Using the augmented data set samples to train the class incremental learning network to obtain a trained class incremental learning network.

[0105] This step specifically includes:

[0106] S4-1: Divide the dataset for each class into a training dataset and a validation dataset. Then, add negative samples to the training and validation datasets. A test dataset can also be created to better test the network's performance.

[0107] S4-2: Randomly select a class of data from all classes of data sets for the following training.

[0108] S4-3: Set the training parameters. If the current stage is the base training stage, randomly initialize the class incremental learning network. If it is the incremental training stage, directly use the network parameters obtained from the previous training for initialization, and expand the dimension of the class incremental module so that its dimension can match the total number of classes of the previously trained classes and the current class. After the expansion, the parameters of the new dimension are randomly initialized. Among them, the training for the first class is the base training stage, and the training for other subsequent classes is the incremental training stage.

[0109] S4-4: Use the example set and the training dataset of the current class to train the class incremental learning network, and optimize the network by calculating the total loss.

[0110] The calculation steps of total loss include:

[0111] S4-4-1: Calculate the mask loss L1 of the class-independent module:

[0112]

[0113] Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the semantic segmentation map, 0 represents the background area, 1 represents the target area, and p i Represents the pixel-level prediction of the mask layer of the class-independent module;

[0114] S4-4-2: Calculate the classification loss L2 in the class increment module:

[0115]

[0116] q i =σ(F cls )

[0117] F cls =FC(F1 avg )

[0118] Among them, FC represents the fully connected layer, F1 avg represents the output of the global average pooling layer of the incremental module, F cls represents the predicted output of the class increment module classifier, y i Represents the category labels for different categories, q i Represents the confidence of the classifier's classification prediction;

[0119] S4-4-3: Calculate the density map estimation loss L3 of the incremental module:

[0120]

[0121] Among them, B represents the batch size, Z i represents the true label of the density map, represents the prediction of the density map;

[0122] S4-4-4: Calculate the distillation loss L4 of the sample density map prediction of the previously trained class:

[0123]

[0124] in, Represent the density map of the trained class and the density map prediction of the current class respectively;

[0125] S4-4-5: Calculate the weighted sum of each loss as the total loss:

[0126] L=L1+L2+(1-λ)*L3+λ*L4

[0127] Here, λ represents a hyperparameter.

[0128] S4-5: Testing the trained class incremental learning network using samples from a validation dataset, and saving the network parameters with the minimum error as the network parameters obtained from training the current class. The validation dataset includes validation datasets for the current class and previously trained classes.

[0129] S4-6: Select several samples that meet the preset conditions from the data set of the current class and add them to the example set.

[0130] This step specifically includes:

[0131] S4-6-1: Based on the pre-set example set memory K, calculate the average number K / I allocated to the trained class i , I i Indicates the total number of trained classes;

[0132] S4-6-2: Remove the samples at the end of each class in the example set, that is, remove K / (I i -1)-K / I isamples;

[0133] S4-6-3: According to the herding algorithm, select the most representative K / I in the current class i samples are added to the example set.

[0134] S4-7: Return to step S4-2 and continue until the training of all class data sets is completed, and the final class incremental learning network with the target count is obtained.

[0135] The target counting-oriented incremental learning network modeling device described in S4-1 of the present invention includes a processor and a computer program stored in a memory and runnable on the processor, and the above method is implemented when the processor executes the program.

[0136] In order to test the performance of the network, a test data set can also be used to test the final network model to obtain the final average error.

[0137] This embodiment further provides an exposure difference enhancement device for low-light images, comprising a processor and a computer program stored in a memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0138] The above disclosure is only a preferred embodiment of the present invention and cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A target counting-oriented incremental learning network modeling method, characterized by include: S1: Obtain a data set of samples of different classes, wherein the data set is a collection of graphs that need to be counted; S2: Perform online augmentation on dataset samples; S3: Constructing a class-incremental learning network for target counting, the class-incremental learning network includes a feature extractor, a class-independent module, and a class-incremental module, the feature extractor is connected to the class-independent module and the class-incremental module, respectively, the class-independent module is connected to the class-incremental module, the feature extractor is used to extract a basic feature vector, the class-independent module is used to learn based on the basic feature vector to obtain a consistent density map, the class-incremental module is used to perform classification based on the basic feature vector, and perform regression counting based on the classification result and the density map; S4: using the augmented data set samples to train the class incremental learning network to obtain a trained class incremental learning network; The class-independent modules specifically include: The first convolutional layer includes two 3×3 convolutional layers; The mask layer includes four sequentially connected single convolutional layers and an upsampling layer. The four single convolutional layers are used to obtain the mask from the output of the first convolutional layer. The upsampling layer is used to perform bilinear interpolation upsampling on the mask to obtain a standard mask of the same size as the input sample. The activation layer is used to first activate the standard mask through the Sigmoid function, then perform a Ones-like operation and subtract the activation feature output by the Sigmoid function to obtain the activation feature; The attention layer includes a 3×3 convolutional layer, a ReLu function, and a 3×3 convolutional layer connected in sequence, which is used to obtain attention features based on activation features; The second convolutional layer includes a splicing unit and a 3×3 convolutional layer. The splicing unit is used to splice the output of the first convolutional layer and the attention feature. The 3×3 convolutional layer is used to convolve the spliced ​​features to obtain a density map. The class increment module includes: A global average pooling layer, used to perform global average pooling on the basic feature vector; The classifier is used to classify the input sample based on the output of the global average pooling layer; The regression layer is used to regress the counts based on the classifier output and the density map to obtain the counts of different classes.

2. The target counting-oriented incremental learning network modeling method according to claim 1 is characterized in that: The S2 specifically includes: S2-1: Randomly scale the samples; S2-2: Crop the samples at random positions; S2-3: Randomly flip the sample; S2-4: gamma correct the sample; S2-5: Convert the sample into grayscale image.

3. The target counting-oriented incremental learning network modeling method according to claim 1 is characterized in that: The feature extractor in S3 is specifically the first 10 layers of the pre-trained VGG16 network.

4. The target counting-oriented incremental learning network modeling method according to claim 1, characterized in that: The S4 specifically includes: S4-1: Divide the dataset of each class into training dataset and validation dataset; S4-2: Randomly select a class of data from all classes of data sets for the following training; S4-3: Set training parameters. If the current stage is the base training stage, randomly initialize the class incremental learning network. If the current stage is the incremental training stage, directly use the network parameters obtained from the previous training for initialization, and expand the dimension of the class incremental module so that its dimension can match the total number of classes of the previously trained classes and the current class. After expansion, the parameters of the new dimension are randomly initialized. The training for the first class is the base training stage, and the training for subsequent classes is the incremental training stage. S4-4: Use the example set and the training dataset of the current class to train the class incremental learning network, and optimize the network by calculating the total loss; S4-5: Testing the trained class incremental learning network using samples from a validation dataset, and saving the network parameters with the minimum error as the network parameters obtained from training the current class. The validation dataset includes validation datasets for the current class and previously trained classes. S4-6: Select several samples that meet the preset conditions from the data set of the current class and add them to the example set; S4-7: Return to step S4-2 and continue until the training of all class data sets is completed, and the final class incremental learning network with the target count is obtained.

5. The target counting-oriented incremental learning network modeling method according to claim 4 is characterized in that: The steps for calculating the total loss in S4-4 include: S4-4-1: Calculate the mask loss L1 of the class-independent module: Among them, X i Represents the data loaded in each batch, a i ∈{0,1} is the true label in the semantic segmentation map, 0 represents the background area, 1 represents the target area, and p i represents the pixel-level prediction of the mask layer of the class-independent module, σ( ) represents the Sigmoid activation function; S4-4-2: Calculate the classification loss L2 in the class increment module: what i =σ(F cls ) F cls =FC(F1 avg ) Among them, FC represents the fully connected layer, F1 avg represents the output of the global average pooling layer of the incremental module, F cls represents the predicted output of the class increment module classifier, y i Represents the category labels for different categories, q i Represents the confidence of the classifier's classification prediction; S4-4-3: Calculate the density map estimation loss L3 of the incremental module: Among them, B represents the batch size, Z i represents the true label of the density map, represents the prediction of the density map; S4-4-4: Calculate the distillation loss L4 of the sample density map prediction of the previously trained class: in, Represent the density map of the trained class and the density map prediction of the current class respectively; S4-4-5: Calculate the weighted sum of each loss as the total loss: L=L1+L2+(1-λ)*L3+λ*L4 Here, λ represents a hyperparameter.

6. The target counting-oriented incremental learning network modeling method according to claim 4, characterized in that: The S4-6 specifically includes: S4-6-1: Based on the pre-set example set memory K, calculate the average number K / I allocated to the trained class i , I i Indicates the total number of trained classes; S4-6-2: Remove the samples at the end of each class in the example set, that is, remove K / (I i -1)-K / I i samples; S4-6-3: According to the herding algorithm, select the most representative K / I in the current class i samples are added to the example set.

7. The target counting-oriented incremental learning network modeling method according to claim 4, characterized in that: After the training data set and the validation data set are divided in S4-1, negative samples are added to the training data set and the validation data set.

8. A target counting-oriented incremental learning network modeling device, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images