Urban garbage incremental learning detection method and system supporting detection of unknown classes

Abstract

The application provides a kind of urban garbage incremental learning detection method and system supporting detection unknown class, on the basis of Faster-RCNN detection model, optimization feature extraction and feature utilization, including setting pyramid feature structure to obtain multi-scale features, making channel attention correction to features, by highlighting the key areas of image features, suppresses unobvious features, obtains garbage detection base model;On the basis of garbage detection base model, a sustainable incremental registration learning model is established;On the basis of sustainable incremental registration learning model, set up network structure supporting detection unknown class, to stably identify unknown class of urban garbage;Support unknown output results as supplement, incremental registration learning is carried out, closed loop is realized.The application improves detection accuracy;Realize model sustainable registration incremental learning, use new data to register new detection class, without touching old data while not producing knowledge forgetting;Unknown class can be detected, and effective detection is output for the class that does not exist in training data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent detection, and particularly relates to a new technology of urban garbage detection and classification capable of detecting unknown classes through continuous incremental registration learning. BACKGROUND

[0002] It is common and mature to apply a neural network target detection method to monitor the camera, and the principle is to drive the data, supplement a large number of category data to the model for training, that is, thousands of category instances are needed to train, so that the model can "recognize" new instances of the category.

[0003] It is well known that the large data dependence of the existing deep learning method is the defect of deep learning. In specific problems, this defect is not obvious for identifying features of uniform objects. However, when applied to particularly diverse and complex detection events, such as the garbage detection in patent CN 109598303 (a garbage detection method based on urban scenes), it is well known that there are hundreds of garbage categories, and the same garbage is different due to factors such as nature, size, and color. The similar problem also exists in patent CN 113963251 (a marine organism detection method). It is not difficult to develop a target detection method for detecting marine organisms, and the difficulty lies in how to collect enough marine organism data of each category, and each category has a large number of samples to cover the detection requirements, but this is obviously impossible due to high cost. Therefore, the above-mentioned inventions have obvious defects:

[0004] (1) There are many undetectable situations. For rarely labeled or unlabeled garbage categories, the model cannot detect. For scene features, the model cannot detect when the garbage is severely occluded by the scene.

[0005] (2) It is almost impossible to realize garbage detection data set by current labeling technology, and the cost is too high. For labeled data, a large amount (thousands or tens of thousands) of data is needed to support the model. There is no mature garbage detection data set on the market, and even if there is data, the amount is very small.

[0006] The two serious defects of patent CN 109598303 prompted the present application. The root cause of the above problems is that the garbage category contains too many types, and deep learning is a data-driven method. Therefore, the present application is designed to solve the above two problems, and a new method of continuous incremental learning capable of detecting unknown urban garbage is designed.

[0007] It can be seen that the existing technology still lacks a solution that meets the demand for implementation effect, and the present application has important practical significance. SUMMARY

[0008] The technical problem solved by the present application is to provide a new method for urban garbage detection that can continuously incrementally learn and detect unknown classes, to solve the shortcomings in the background art.

[0009] The present application provides a kind of urban garbage incrementally learning detection method that supports detecting unknown class, carries out the following processing,

[0010] On the basis of Faster-RCNN detection model, the feature extraction and feature utilization are optimized, including setting pyramid feature structure to obtain multi-scale features, then making channel attention correction to the features, highlighting the key areas of image features by correction, suppressing the features that are not obvious, and obtaining a garbage detection base model;

[0011] On the basis of the garbage detection base model, a sustainable incremental registration learning model is established, including gradient processing based on meta-learning, setting new gradient processing loss and distillation loss, establishing parameter update strategy, so that the model training supports using only new data without touching old data, completing new class registration, and at the same time not forgetting old knowledge;

[0012] On the basis of the sustainable incremental registration learning model, a network structure is set to support detection of unknown classes, including establishing a clustering loss during training, fitting an energy function, and using the energy function and energy value in combination with the parameter matrix classification head to generate classification results during prediction, to stably identify unknown classes of urban garbage; The unknown output results are supported as supplements for incremental registration learning, to realize closed loop.

[0013] Moreover, the implementation of optimizing feature extraction and feature utilization is as follows,

[0014] Convolution operation is performed on the input image, five groups of scale features are extracted from bottom to top, which are reflected in the reduction of width and height and the increase of channel number, then feature fusion is completed from top to bottom for each group of features, which is reflected in the up-sampling and addition operation, the width and height are restored and fused with the previous scale features;

[0015] The extracted features are corrected by channel attention map, global mean and extreme value pooling operations are performed on the features, shared MLP multi-layer perceptron is used to connect different dimension information, attention map is generated by addition operation and activation function, and finally the original features are combined for weighted correction.

[0016] Moreover, the implementation of establishing a sustainable incremental registration learning model is as follows,

[0017] The parameter matrix gradient descent with controllable gradient is used to replace the regular gradient descent of chain rule derivation, and the parameter constraint in gradient descent will control the gradient descent direction;

[0018] The parameter constraint module is modularized, a gradient processing layer is set, a feature pool storing the number of features corresponding to each category of the data set is maintained, and a gradient loss function is set to guide the model gradient descent to consider all categories;

[0019] The knowledge distillation strategy takes the past time task feature extraction as the teacher network, adds a new detection category as the current task, sets a distillation loss around the feature difference before and after, and ensures that the old category data is not forgotten when registering new category data.

[0020] Moreover, the implementation of the model recognizing the unknown category is as follows,

[0021] In the training process, a contrast clustering loss function is constructed for the final to-be-detected output feature, the distance between the to-be-detected feature and different categories is compared by maintaining the feature vector center of each category, the feature vector distance between different categories is increased, and the feature vectors of the same category are gathered;

[0022] The learning energy function converts the multi-dimensional feature vector into a scalar, learns the matching degree between the scalar and the belonging category, and the learned model mapping judges whether an unknown feature vector belongs to an unknown category;

[0023] The unknown classification head using the parameter matrix in parallel normalizes the feature, sets a bounded sparse parameter mask matrix, limits the output distribution, and ensures the stability of the output category.

[0024] Moreover, when the basic category garbage is recognized, the machine reminder mode is adopted by transmitting a signal to the camera to stop the garbage generation behavior; when the unknown category is recognized, the snapshot is taken and the manual processing is prompted, the measures are decided according to the user's judgment, and the unknown category is included in the new round of training data of the application.

[0025] On the other hand, the application provides a city garbage incremental learning detection system supporting detection of unknown categories, which is used to implement the city garbage incremental learning detection method supporting detection of unknown categories as described above.

[0026] Moreover, the application includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the city garbage incremental learning detection method supporting detection of unknown categories as described above.

[0027] Alternatively, the application includes a readable storage medium, the readable storage medium stores a computer program, and the computer program is executed to implement the city garbage incremental learning detection method supporting detection of unknown categories as described above.

[0028] Overall, compared with the prior art, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0029] (1) Significantly reduce undetectable cases and achieve comprehensive detection of garbage: The model tries to identify samples that do not exist in the dataset, significantly reducing undetectable cases. The model can also identify "unseen garbage" as an unknown class.

[0030] (2) Significantly reduce data labeling requirements: Only a certain amount of data is needed to complete the initial model training. Subsequently, the detection results and unknown identifications can be used as model data to continuously register new detection categories to the model, thereby achieving "incremental learning" of the model. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the use of features improved by pyramid features and channel attention mechanism in the embodiment of the present invention (1).

[0032] Figure 2 This is a schematic diagram of adding an incremental learning module to the detection model based on meta-learning, which is part of the embodiment of the present invention (2).

[0033] Figure 3 This is a schematic diagram of setting up clustering and energy function calculation to add an unknown identification module to the detection model in the embodiment of the present invention (3).

[0034] Figure 4 This is a flowchart illustrating the detection process in an embodiment of the present invention.

[0035] Figure 5 This is a schematic diagram illustrating data annotation and collection during the application of an embodiment of the present invention.

[0036] Figure 6 This is a schematic diagram of the model architecture of an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0038] This invention addresses the problem of limited practicality in object detection and garbage detection-related inventions by optimizing feature extraction and utilization in the detection model to improve detection accuracy; it enables the model to continuously register incremental learning, using new data to register new detection categories without touching old data or experiencing knowledge forgetting; and it enables the model to detect unknown categories and produce effective detections for categories that do not exist in the training data.

[0039] This invention mainly proposes the following three technical improvements:

[0040] (1) Improved two-stage detection method model based on pyramid features and channel attention mechanism: From the perspective of feature extraction and feature utilization, the basic two-stage detection method model is improved. This is an important design module that enables the present invention to detect difficult instances.

[0041] (2) Based on meta learning, an incremental learning module is added to the detection model: through gradient preprocessing based on meta learning, gradient processing loss and distillation loss are set, and the model can learn new classes without forgetting the knowledge of old classes. This is an important design module that enables the application to continuously learn incrementally.

[0042] (3) An unknown recognition module is added to the detection model: a clustering loss is set, the model output features are clustered, and an energy function is set to provide a basis for the model to judge the unknown. In combination with the parameter matrix classification, the input features are classified again to improve the stability of unknown class recognition. This is an important design module that enables the application to recognize unknown classes.

[0043] The application proposes new modules that affect the entire detection process and achieve new technical effects.

[0044] Referring to Figure 4 , the application embodiment proposes a city garbage incremental learning detection method supporting unknown class detection, including the following steps:

[0045] Step 1) Multi-scale features are generated using a feature pyramid, and the extracted feature map is corrected using a channel attention mechanism.

[0046] The application proposes to optimize feature extraction and feature utilization based on the Faster-RCNN detection model, including setting a pyramid feature structure to obtain multi-scale features, and then correcting the features using a channel attention mechanism. By highlighting the key areas of image features and suppressing less obvious features, a garbage detection base model is obtained.

[0047] Further, the implementation of optimizing feature extraction and feature utilization is as follows,

[0048] Convolution is performed on the input image, and five groups of scale features are extracted from bottom to top, which are reflected in the reduction of width and height and the increase of channel number. Then, feature fusion is completed from top to bottom for each group of features, which is reflected in the upsampling and addition operation, which restores the width and height and fuses with the previous scale feature;

[0049] The extracted features are corrected using a channel attention map, global mean and maximum pooling operations are performed on the features, shared MLP multi-layer perceptron is used to connect different dimension information, attention map is generated through addition operation and activation function, and finally the original features are weighted and corrected.

[0050] Compared with the conventional Faster-RCNN target detection model: the backbone network extracts a single feature, which is sent to the RPN network to generate region proposals (regions that may contain targets), the region proposals are sent to the ROI (Region of Interest) to generate features of uniform size, and finally used for classification (detecting what the object is) and regression (detecting where the object is).

[0051] Referring to Figure 1 , the present application is to detect difficult instances in complex scenes, which is different from the above-mentioned general model. Before extracting features and sending them into the RPN network, the present application extracts more rich and multi-scale features from the image to be extracted, and generates multi-scale features through feature pyramid. At the same time, it ensures efficient use of features, and the channel attention mechanism corrects the extracted feature map, so that the subsequent module can be used. Specifically:

[0052] Referring to Figure 1 The left upper part: in the embodiment of the present application, the feature pyramid is divided into bottom-up feature extraction, which obtains feature maps with gradually reduced width and height and improved channel number by continuously convolving the input image; top-down feature fusion, which obtains features in multiple dimensions through upsampling and fusion. The formula for the fusion process is: where P represents the new fused feature map, the subscript represents the first group of features, the higher the value represents the higher the feature dimension, C represents the different scale features extracted by ResNet50 to be fused, conv 1×1 represents the convolution operation of adjusting the channel, 1x1 represents the size of the convolution kernel, and up represents the upsampling operation.

[0053] P5=conv 1×1 (C5)

[0054] P4=conv 1×1 (C4)+up(P5)

[0055] P3=conv 1×1 (C3)+up(P4)

[0056] P2=conv 1×1 (C2)+up(P2)

[0057] Finally, each group of features is again conv 3×3 convoluted to improve the receptive field. The feature map sent to the channel attention is obtained:

[0058] P5=conv 3×3 (P5), P4, P3, P2…

[0059] Referring to Figure 1 The right upper part: in the embodiment of the present application, the channel attention mechanism performs maximum pooling and mean pooling on the multi-scale features extracted in the foregoing along the global, sends them into the shared MLP multi-layer perceptron, outputs two copies of the same channel number of the pooled output, and infers the attention map M c through element addition operation and activation function. c The attention map M

[0060] The above process can be expressed in the form of mathematical formula:

[0061] M c (F) = sigma(MLP(AvgPool(F)) + MLP(MaxPool(F)))

[0062] Wherein, F represents input features, AvgPool(F) and MaxPool(F) refer to mean pooling and maximum pooling of input features F, MLP represents a multilayer perceptron, and sigma represents a sigmoid activation function. In the output feature map, important features are strengthened, and unobvious features are inhibited.

[0063] Step 2) On the basis of the garbage detection basic model, a sustainable incremental registration learning model is established, including gradient processing based on meta-learning, setting a new gradient processing loss and a distillation loss, establishing a parameter updating strategy, so that the model training supports only using new data without accessing old data, completing new class registration, and at the same time not forgetting old knowledge.

[0064] Further, the implementation mode of establishing a sustainable incremental registration learning model is as follows,

[0065] The parameter matrix gradient descent with controllable gradient is adopted to replace the conventional gradient descent of chain rule derivation, and the parameter constraint in the gradient descent controls the gradient descent direction;

[0066] The parameter constraint is modularized, a gradient processing layer is set, a feature pool storing the number of features corresponding to each class of the data set is maintained, and a gradient loss function is set to guide the model gradient descent to consider all classes;

[0067] The knowledge distillation strategy takes the past task feature extraction as the teacher network, and the newly added detection class as the current task, sets a distillation loss around the feature difference before and after, and ensures that the old class data is not forgotten when registering the new class data.

[0068] Referring to Figure 2 , the incremental learning module of the application is defined: first, the incremental learning problem in the application is defined, which is different from the conventional method model process: obtaining data-training model-performing detection, when the model detection class does not meet the requirements: supplementing data-retraining the model-performing detection. This will bring huge additional computational overhead in the actual detection process. The incremental learning of the application refers to: the garbage detection model will only need to access new garbage data, without accessing old data for knowledge learning. This greatly reduces the computational overhead of training the model. This requires adding corresponding modules and designing a new model structure to realize, and the embodiment further provides a preferred implementation mode as follows:

[0069] (2-1) The parameter updating of the model in the application is changed from the gradient descent of chain rule derivation to the parameter matrix gradient descent with controllable gradient. Let is the loss function, μ is the learning rate, θ is the parameter set of the target detector, is the argument of the loss function , P is the parameter matrix, φ is the parameter in the matrix, is the derivative of the current target detector loss function. P(θ; φ) then represents the parameter constraint generated by . θ' is the new target detector parameter after a round of gradient descent.

[0070]

[0071] (2-2) See Figure 2 Flow: The gradient preprocessing layer is a series of three convolutional layers and gradient warp layers, with residual connections before and after each convolutional and gradient warp layer. The specific calculation method of the gradient preprocessing layer in the present application is as follows: maintain a feature pool F Store to store N features of each class in the data set, and N is preferably 20 in the embodiment. Store the uniform size features obtained after F RoI pooling of each class through a queue structure. At the same time, use the current model parameters θt to process the uniform size features after F RoI pooling in the feature pool, and output the corresponding classification information and regression information. Then calculate the loss as follows according to the classification information and regression information:

[0072] Let f be a feature in the feature pool that has been F RoI pooled by the current model parameters θt, p * and l * represent the real class information and the parameterized coordinates of the detection frame corresponding to the feature. f will be sent to the classification and regression head F RoI Head to generate the corresponding class prediction p and detection frame prediction l, that is, (p, l) = F RoI Head (f). and is a function that calculates the difference between the predicted value and the label value. By taking out each (f, p Store , l * ) in F * , the difference from the true value is calculated and accumulated. The above calculation process is denoted as the loss function

[0073]

[0074] Through the constraint of the loss function, it is ensured that the parameters of the network classification and regression do not produce catastrophic forgetting when facing new tasks. Through the construction of the feature pool of each class, it is ensured that the gradient preprocessing of meta-learning pays equal attention to all classes.

[0075] (2-3) In order to enable the current task to generalize to the detection performance of the past time task, and not to completely contact the past training data, the application introduces a knowledge distillation strategy. The knowledge distillation operation uses a set of parameters θt suitable for the detection task to parameterize the currently used model, and also needs to use the parameters of the past time task model, the purpose is to distill the knowledge from the backbone network feature extraction and classification regression head of the teacher network (past time task) to the student network (current task, adding new detection categories).

[0076] Each training image will be transmitted into the current task and past task model to obtain a set of extracted scale feature maps F t and F t-1 . The two sets of features are respectively input into the region proposal network of the current time model to generate object information, and the output is input into the F RoI and F RoI Head classifier and regressor. The final output of the two sets of classification and positioning information {p t ,p t-1} and {p t-1 ,l t-1}. The two sets of information and the previously output feature information will be used as the loss calculation of knowledge distillation:

[0077]

[0078] Where is the general L2 loss calculation, is the KL divergence calculation between the two sets of classification probabilities calculated by the model before and the current model, and at the same time, the calculation is only calculated for the registered detection categories.

[0079] The above steps ensure that during the new data training process, the network model can register new category data while not forgetting the old category data.

[0080] Step 3) On the basis of sustainable incremental registration learning model, set up support to detect unknown category network structure, including training to establish clustering loss, fit energy function, use energy function and energy value combined with parameter matrix classification head to produce classification results when predicting, to stably identify unknown categories of urban garbage (having garbage visual features and not existing in training data categories, not conforming to known garbage visual features); Support unknown output results as a supplement to incremental registration learning, realize closed loop, solve the problem that garbage detection task cannot be defined once for all.

[0081] Further, the implementation of the model to identify unknown categories is as follows,

[0082] The final to-be-detected output feature is used to construct a contrast clustering loss function during the training process, the distance between the to-be-detected feature and different categories is compared by maintaining the feature vector center of each category, the feature vector distance between different categories is increased, and the feature vectors of the same category are gathered;

[0083] The learning energy function converts the multi-dimensional feature vector into a scalar, learns the matching degree between the scalar and the category to which it belongs, and the mapping of the learned model judges whether an unknown feature vector belongs to an unknown category;

[0084] The unknown classification head using the parameter matrix in parallel normalizes the features, sets a bounded sparse parameter mask matrix, limits the output distribution, and ensures the stability of the output category.

[0085] Referring to Figure 3 , a classifier acting on the weight parameters and a new unknown class learner are inserted. The model structure realizes the recognition of unknown categories in combination with the following steps. The definition of unknown garbage of the present application is as follows: unlike the garbage data in the training set data (having a determined label and uniform visual features), the unknown garbage meets the following characteristics: not meeting the visual features of the training basic garbage data; having garbage object features. (The data category in the training set is recommended to be not less than 10 categories, and the total data is recommended to be not less than 10,000 images.) The basic class is the common garbage category defined in the training data set, including but not limited to metal cans, plastic bottles, paper boxes, plastic packaging, cigarette butts, cigarette boxes, masks, etc.

[0086] The unknown class learner has the following steps:

[0087] (3-1) In the final F RoI When outputting a 2048-dimensional feature for each detection frame, the feature vector f c of the feature output layer in the detector is obtained. i For each known class, a feature vector center p c is maintained. Therefore, a contrast clustering loss function expressed as follows can be constructed:

[0088] Wherein

[0089]

[0090] Wherein, is the clustering loss value calculated by the feature, by accumulating l(f c ,p i ), f c is the feature vector obtained in the foregoing, and the feature vector center p iis the representative of class i in the training data, is computed by averaging the features of all class i data, and C represents all classes. l is a piecewise function that computes the clustering loss, and the sought feature f c is the feature vector center p of the same class c The distance is then computed, is the feature distance computation function, which here uses the regular Euclidean distance; the sought feature f c is the distance to other class feature vector centers, and the distance is computed whether it exceeds a constant Δ, which is used to define the maximum distance of feature vectors within a cluster. We want to be able to exceed the constant Δ, indicating that it does not belong to the class and is relatively far away. When it exceeds, its loss is 0. By continuously reducing the loss, we can ensure that the feature vectors in the latent space are gathered together, and the features of different classes are repelled from each other. During network training, different classes of detection examples will maintain a relatively far distance, and when an unknown class appears, the corresponding feature vector will maintain the distance.

[0091] (3-2) Referring to Figure 3 The first half: considering that the automatically labeled unknown instance features in the feature latent space are inherently separated from the known classes (they come from the background and foreground respectively during training, and their visual features are also inherently separated), and considering the characteristics of containing unknown classes, the traditional softmax classifier may give uncontrollable results.

[0092] Given a feature f in the latent space F and its corresponding label l e all label classes L, we try to find a learning energy function E(f, l). The energy model learning function E(f) outputs a single scalar, i.e. E(f): R d into a real number R in the real number domain. This can estimate the matching degree between the observed variable f and the possible output variable l. The scalar value E(f) is called energy, and the model assigns low energy values to data within the known distribution and high energy values to data outside the known distribution. The energy value is used to measure whether a sample comes from an unknown class. As shown in the following formula, the Gibbs free energy formula is used to calculate the sum of all label results E(f):

[0093]

[0094] where l' is the predicted class of feature f, T is the temperature parameter, and exp is the exponential function. After integrating and taking the logarithm, the energy value is obtained. In order to convert it into a structure that can be used in the model, similar to softmax, the Gibbs distribution can convert the energy of each label into the following formula:

[0095]

[0096] Where p(l|f) is the probability density function of label l, g i (f) is the classification head g i The l th There are several classification units (similar to the output nodes of softmax for each class). Therefore, the classification result g can be... i (f) is equivalently replaced by E(f). Therefore, according to the correspondence in the previous formula, for an input feature, the classification model can use the following free energy formula:

[0097]

[0098] The scalar value output by this free energy formula is called the energy E(f,g), where each conforms to the same rules as described above. Naturally, the traditional softmax classifier is replaced with a classification head based on the energy function. Furthermore, due to the clustering loss mentioned earlier, the energy values ​​of known and unknown classes should have a significant difference. Calculating the energy values ​​of known and unknown classes facilitates network partitioning; thus, this invention completes the classification of the unknown classes.

[0099] (3-3) See Figure 3 The second part: To make the classification more stable, a weight parameter layer is introduced in the above steps. The weight w is normalized, i.e., w = w / ||w||. At the same time, the feature X is also normalized. Where ||·|| represents the L2 norm. This step confines unbounded values ​​to a bounded region, making the identification of unknown class detection examples more stable and preventing abnormally large or small values ​​in feature extraction.

[0100] `w` is a D×C matrix representing the weights of the last linear layer mapped to classes C, where D represents the weight dimension of each class. The magnitude of the weights can be expressed as `R*w / ||w||`, where the range of `R` is {0,1}, and `R` represents a parameter mask; the dimension is D×C, which follows a Bernoulli distribution with probability p, and `*` denotes element-wise product. The total number of 1s in `R` is `n`, where a value of 1 indicates that a weight is retained, and a value of 0 indicates that a weight is discarded. This constructs a bounded and sparse weight matrix.

[0101] The final output can be expressed as the following formula:

[0102]

[0103] Wherein, a is a regularization factor, through the above steps, the output expected value range is [-p a, p a], without the above steps, the range is [-a, a], therefore, the range of the expected difference caused by the classifier acting on the weight parameter is [- (1 + p) a, (1 + p) a]. For the traditional output, the expected difference is (-∞, +∞). This means that the output distribution of the model is more consistent under such processing. The output l c Is the predicted category.

[0104] Referring to Figure 3 , the step is parallel to the (3-2) step, and finally the output results are judged logically: when both results are output as unknown categories, the model outputs unknown, otherwise, the classification output result of (3-3) is taken as the category prediction result. In this way, the detection of unknown categories is more stable, and the error division of known categories is reduced.

[0105] Step 4) Refer to Figure 5 Data labeling and collection. The data flow combines Figure 5 Described as follows: according to known common categories, collect garbage data, the collection method includes network crawler and self-sampling, use labelimg open source software to label the self-sampling data set, including detailed category information and detection frame information, the labeled data set is used for model training in this paper, after training, the model can be used for actual prediction, and a large number of categories and detection frames can be predicted for the to-be-detected data in the real scene. Unknown categories can be selected as supplementary data, and labelimg is used again for data fine-tuning. The supplementary data can be registered again to the same model, expand the model detection category, and improve the model detection capability.

[0106] Step 5) Use the labeled data to train the model for the first round, and the important training parameters are set as follows: the total training iteration epochs is set to 100, the batch image is set to 32, and the input network image size is fixed to 640*640 pixels. The optimizer adopts SGD random gradient descent, the initial learning rate is set to 1E-4, and the weight decay coefficient is set to 0.0005. The cuda random seed deterministic value is set to true, which guarantees the stability of the input and output relationship. The workers multi-thread data loading is set to the default value 8.

[0107] The above trained model can be used for detection, and the model is referred to in the following.

[0108] Step 6) Apply the whole model of the application to make prediction. Refer to Figure 6The overall model of the present application is shown. After training, the network architecture with fixed parameters of each part will be obtained, that is, the specific parameters of the rectangular part in the figure. At this time, the model can sequentially pass the input image through the modules in the figure, and finally output the target information in the image. Specifically, in the prediction process, the input image will be sent into Figure 6 In the model network of the present application, first, the features are extracted according to the pyramid feature structure, and the output features are refined by using channel attention correction. Then, the region proposal network generates region proposals, and the output results (several region proposals) are sent into the conventional ROI feature extraction together with the original feature map output after attention. The ROI feature extraction will use the above two parts to generate a feature vector of uniform dimension for each region. Then, it is sent into the gradient processing layer. In the prediction process, the gradient preprocessing layer will be fixed and can be regarded as several times of convolution. In the training process, the gradient processing layer will update the gradient separately as described above, and at the same time, this part will set the contrast clustering loss function of the present application in the training process, so that the feature extraction is more consistent with the specific category. The output results will be used as the basis for final classification and positioning, and will be sent into two paths for processing: one path uses the energy function-based classifier in the present application and the weight parameter classification output category information, including unknown categories not seen in the training data set, and the other path refers to the conventional box regression to predict the specific pixel position.

[0109] Step 7) refer to Figure 4 The lower right flow. In actual application, when a certain number of unknown instances are detected, the method of the present application can use them as new training data, organize them in the form of data and labels, and then train again based on the original model. According to the gradient preprocessing setting based on meta-learning and the distillation loss setting in the present application, the model can complete the training of new data without touching a large amount of original data. This greatly saves the computing cost and improves the upper limit of the detection capability of the model in the present application.

[0110] Step 8) refer to Figure 4 The flow directly below. In actual application, the prediction result information is output. The output information of the present application includes: the pixel position information of the detected garbage target, the confidence of the detection example (the higher the confidence, the more likely it is a correct recognition), and the category information of the detected garbage target (the category in the training data, unknown class). The present application suggests that different processing should be done for the output results of this method. When the basic class garbage is recognized, the machine reminds the user to stop the garbage producing behavior by transmitting a signal to the camera. When the unknown class is recognized, it is snapped and handed over to manual processing, and the user decides the measures according to the judgment, and at the same time, it is included in the new round of training data of the present application.

[0111] By now, the present application is significantly different from the related patent CN 109598303:

[0112] (I) Method design level, patent CN 109598303 does not carry out method design, and the invention idea is mainly reflected in the application of FasterRCNN in the specific problem: city garbage detection level. The invention focuses on model architecture design, bringing a new model effect: capable of incremental continuous learning. At the same time, it can detect unknown conditions.

[0113] (II) Problem solving level, the problem solved by patent CN 109598303 is how to apply target detection to application problems, that is: how to use target detection to detect urban garbage. The invention aims at: how to solve the problem of few data and difficult data acquisition in target detection related inventions-how to use for a small number of samples, and how the model handles data that has not been obtained. By solving the above problems, the practical application of the garbage detection invention is enabled. It realizes that only by using the existing garbage data set, the detection effect that the previous model has not obtained can be achieved: neither a large number of sample labeling is needed, nor the data types not contained in the data are tried to detect. And subsequent new detection categories can be continuously registered.

[0114] (III) Invention content level, patent CN 109598303 includes each step in the problem solving process, most of which are standard target detection conventional processing means applied to specific problems, and the invention content only targets unconventional processing means in the steps. Without departing from the spirit and scope of the invention, the invention can be changed and improved.

[0115] The invention focuses on the design and processing of the method level. Through the ingenious design of the module design and training part, it realizes the effect that similar patent CN 109598303 cannot realize for completely different problems, that is, two advantages: a powerful detection model can be completed without collecting a large amount of data, reducing costs, and the model can be continuously upgraded to register more detection categories. The model can still try to identify data that has not been collected, turning the problem from infinite to finite. From the perspective of data, it solves the application pain points of some target detection models, fundamentally solves the problem of insufficient data and poor model effect, and makes an invention in the practical application level of target detection model design.

[0116] In specific implementation, the method proposed by the technical solution of the invention can be automatically run by a person skilled in the art using computer software technology. The system device for running the method, such as a computer readable storage medium storing the corresponding computer program of the technical solution of the invention and a computer device including a computer device running the corresponding computer program, should also be within the protection scope of the invention.

[0117] In some possible embodiments, a garbage increment learning detection system for detecting unknown classes in a city is provided, comprising a processor and a memory, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions stored in the memory to execute the garbage increment learning detection method for detecting unknown classes in a city.

[0118] In some possible embodiments, a garbage increment learning detection system for detecting unknown classes in a city is provided, comprising a readable storage medium, and a computer program is stored on the readable storage medium, and the computer program is configured to implement the garbage increment learning detection method for detecting unknown classes in a city when executed.

[0119] The above merely provides preferred embodiments of the present application but not for limiting the present application. For those skilled in the art, the present application can have various modifications and changes. Any modified, equivalent replaced, improved, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A city garbage incremental learning detection method supporting detection of unknown classes, characterized in that, The following processing is performed, On the basis of the Faster-RCNN detection model, the feature extraction and feature utilization are optimized, including setting a pyramid feature structure to obtain multi-scale features, and then performing channel attention correction on the features, highlighting the key areas of the image features by correction, suppressing the non-obvious features, and obtaining a garbage detection basic model; On the basis of the garbage detection basic model, a sustainable incremental registration learning model is established, including gradient processing based on meta-learning, setting a new gradient processing loss and a distillation loss, establishing a parameter update strategy, so that the model training supports only using new data without touching old data, completing new class registration, and without forgetting old knowledge; On the basis of the sustainable incremental registration learning model, a network structure supporting unknown class detection is set, including establishing a clustering loss during training, fitting an energy function, and using the energy function and energy value in combination with the parameter matrix classification head to generate classification results to stably identify unknown classes of urban garbage; the unknown output results are supported to be supplemented for incremental registration learning, realizing a closed loop; The implementation of the model to identify unknown classes is as follows, During the training process, a contrast clustering loss function is constructed for the final detection output features, the distance between the detection features and different classes is compared by maintaining the feature vector center of each class, the feature vector distance between different classes is increased, and the same class feature vectors are gathered; The energy function is learned to convert the multi-dimensional feature vector into a scalar, and the matching degree between the learned scalar and the belonging class is learned. The mapping of the learned model will judge whether an unknown feature vector belongs to an unknown class; The unknown classification head of the parameter matrix is used in parallel, the features are normalized, a bounded sparse parameter mask matrix is set, the output distribution is limited, and the output class is stable; When the basic class garbage is identified, the machine reminds the user to stop the garbage producing behavior by transmitting a signal to the camera; When the unknown class is identified, the user is prompted to handle it manually, and the user decides the measures according to the judgment, and the unknown class is included in the next round of training data.

2. The urban garbage incremental learning detection method of claim 1, wherein: The implementation of the optimized feature extraction and feature utilization is as follows, Convolution operation is performed on the input image, five groups of scale features are extracted from bottom to top, which are reflected in the reduction of width and height and the increase of channel number, and then feature fusion is completed from top to bottom, which is reflected in the upsampling and addition operation, the width and height are restored and fused with the previous scale feature; The extracted features are modified by channel attention map, global mean and maximum pooling operations are performed on the features, shared MLP multi-layer perceptron is used to connect different dimension information, attention map is generated by addition operation and activation function, and finally the original feature is weighted and corrected.

3. The urban garbage incremental learning detection method of claim 1, wherein: The implementation of the sustainable incremental registration learning model is as follows, The controllable gradient parameter matrix gradient descent is used to replace the regular gradient descent of chain rule, and the parameter constraint in the gradient descent controls the gradient descent direction; The parameter constraint is modularized, a gradient processing layer is set, a feature pool storing the number of features corresponding to each class of the data set is maintained, and a gradient loss function is set to guide the model gradient descent to consider all classes. The knowledge distillation strategy takes the past time task feature extraction as a teacher network, and adds a new detection category as a current task, sets a distillation loss around the feature difference before and after, and ensures that the old category data is not forgotten when registering new category data.

4. A city garbage incremental learning detection system supporting detection of unknown classes, characterized in that: The application discloses a garbage incremental learning detection method for supporting unknown class detection in a city.

5. An electronic device, comprising: The garbage incremental learning detection method for supporting unknown class detection in the city comprises a processor and a memory.

6. A readable storage medium, wherein a computer program is stored on the readable storage medium, and the computer program is executed to implement the garbage incremental learning detection method for supporting unknown class detection in the city.

Images