Class increment smoke segmentation method based on multi-level regional knowledge distillation and prototype equilibrium contrast learning

Abstract

The invention provides a class increment smoke segmentation method based on multi-level regional knowledge distillation and prototype balance contrast learning, and the method comprises the steps: inputting an input image into encoders of a teacher model and a student model, extracting shallow feature maps of different levels, carrying out the feature fusion through a decoder, obtaining a deep feature map, and carrying out the segmentation of the deep feature map; performing prediction by a classifier to obtain a prediction result, and calculating a mixed pseudo tag; calculating multi-level regional knowledge distillation loss based on different levels of shallow feature maps and mixed pseudo labels; on the basis of the deep feature map and the mixed pseudo tag, updating the prototype of each type of cigarettes, and calculating prototype balance comparison learning loss; and carrying out weighted summation on all losses, carrying out gradient back propagation to update the student model, and using the updated student model to carry out smoke segmentation on the test sample. According to the method, the defects in the prior art can be effectively overcome, the characteristics of the old type of cigarettes are better reserved while the characteristics of the new type of cigarettes are continuously learned, and the segmentation precision of the class increment cigarette segmentation method is improved.

Description

Technical Field

[0001] The present invention relates to the field of fire prevention, and predicts fire by identifying smoke in the early stage of fire, and in particular to a quasi-incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrast learning. Background Art

[0002] Smoke segmentation aims to segment different types of smoke from complex backgrounds, which helps to identify and respond to smoke in a timely manner and reduce fire risks. In actual application scenarios, there are many types of smoke, such as smoke on grasslands, smoke in forests, smoke from straw burning, smoke from factory warehouse fires, smoke from high-rise residential fires, smoke from shopping mall fires, smoke from factory emissions, smoke from chemical reactions, etc. In intelligent monitoring and automation systems, accurate distinction between different types of smoke can provide more accurate information for emergency response and reduce the false alarm rate of fires. However, as the environment changes, the characteristics of smoke are constantly changing. How to enable the model to have the ability to continuously learn new types of smoke is a key issue. Incremental smoke segmentation aims to continuously learn the features of new types of smoke to segment new types of smoke, while retaining the features of old types of smoke learned in previous tasks. Incremental smoke segmentation faces the following challenges: (1) Smoke is highly variable in shape, color, and concentration, and its characteristics may vary significantly under different ambient lighting and background conditions; (2) The stability-plasticity dilemma, i.e., the model must be both plastic in order to continuously learn new categories of smoke features and stable in order to not forget old categories of smoke features; and (3) Background semantic drift, i.e., at the current stage, the smoke in the background area may be of the old category that has been learned before, or it may be of the new category that will be learned in the future.

[0003] To address the above challenges, existing incremental smoke segmentation methods can be roughly divided into three categories: (1) Replay-based methods usually retain a portion of previous old category samples or generate additional old category samples for incremental training to prevent the forgetting of old category smoke features. The performance improvement of this type of method comes at the expense of a large amount of memory. The greater the performance improvement, the higher the memory consumption. In addition, this type of method has the risk of privacy leakage; (2) Dynamic structure-based methods usually use two parallel networks to store old category smoke features and learn new category smoke features respectively. As the model structure expands, this type of method requires more computational cost during training; (3) Knowledge distillation-based methods alleviate the catastrophic forgetting problem by transferring the knowledge of the old model to the new model. Compared with the first two types of methods, this type of method requires less memory and computing resources and has therefore received more attention.

[0004] However, current methods based on knowledge distillation still face the following two problems: (1) Existing methods usually only perform intermediate-layer knowledge distillation in a single-level manner without distinguishing between new and old categories. This cannot fully constrain the changes in smoke feature representations in old category areas and will limit the learning of smoke feature representations in new category areas. (2) Existing methods rarely consider the class imbalance problem between new and old categories in incremental datasets, which can easily lead to classification confusion. Summary of the invention

[0005] To solve the above problems, the present invention discloses a class-incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning, which retains the smoke features of old categories while continuously learning the smoke features of new categories to achieve better segmentation performance.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A quasi-incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning includes the following steps:

[0008] (1) The input image is input into the encoders of the teacher model and the student model respectively, and shallow feature maps of different levels are extracted. The deep feature maps are then obtained by feature fusion through the decoder. The classifier is then used to predict the result and calculate the mixed pseudo-label.

[0009] (2) Calculate the multi-level regional knowledge distillation loss based on shallow feature maps and mixed pseudo-labels at different levels;

[0010] (3) Based on the deep feature map and mixed pseudo-labels, update the prototype of each category of smoke and calculate the prototype balanced contrastive learning loss;

[0011] (4) All losses are weighted summed and gradient backpropagation is performed to update the student model. The updated student model will be used to perform smoke segmentation on the test sample.

[0012] Furthermore, the calculation method of the shallow feature maps of different levels in step (1) includes the following steps (taking the student model as an example, the teacher model is similar):

[0013] (11) Take an input image I∈R 3×H×W Cropped into a sequence of patches, where each patch is of size P×P and the length of the sequence is N=(H×W) / (P×P), and then the patch sequence is mapped into a D-dimensional vector through the patch embedding operation;

[0014] (12) The D-dimensional vector is input into an encoder consisting of four Transformer blocks to extract four shallow feature maps at different levels, which are denoted as F e,1 、F e,2 、F e,3 and F e,4 .

[0015] Furthermore, the calculation method of the deep feature map in step (1) includes the following steps (taking the student model as an example, the teacher model is similar):

[0016] (13) For four different levels of shallow feature maps F e,1 、F e,2 、F e,3 and F e,4 Perform convolution calculation and upsampling operations respectively to obtain F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 , where the number of channels and spatial size of the output feature map remain consistent;

[0017] (14) The feature map F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 Concatenate by channel dimension and then perform convolution operation to obtain the deep feature map F d .

[0018] Furthermore, the process of predicting the deep feature map by the classifier in step (1) includes the following steps (taking the student model as an example, the teacher model is similar):

[0019] (15) The deep feature map F d Input into the classifier to obtain the initial prediction result;

[0020] (16) The initial prediction result is upsampled to obtain the final prediction result, which is consistent with the spatial size of the input image.

[0021] Furthermore, the calculation process of the final pseudo-label in step (1) includes the following steps:

[0022] (17) The result L predicted by the teacher model t-1 Perform the sigmoid operation, and then perform the threshold segmentation operation for each channel of each pixel position i. If the predicted value is less than the threshold, the predicted value of the position is set to 0, otherwise, no processing is performed. The purpose of performing threshold segmentation is to prevent low-confidence pseudo-labels from misleading the student model to inherit and retain the smoke features of the old category;

[0023] (18) The processed prediction result L t-1 Perform the argmax operation to obtain the pseudo label PL of the old category smoke t-1 , and then calculate the final pseudo label according to the following formula:

[0024]

[0025] Among them, y represents the true label of the current step.

[0026] Furthermore, the calculation process of the multi-level regional knowledge distillation loss in step (2) includes the following steps:

[0027] (21) Extract the feature representation of each category area based on the pseudo-label (taking the student model as an example, the teacher model is similar):

[0028]

[0029] in, represents the kth shallow feature map extracted by the student model, Represents the pseudo-label after downsampling (spatial size and shallow feature map Keep consistent), c∈C 1:t Indicates the category, d k Represents shallow feature map The number of channels, N k.c Represented in the shallow feature map The number of features corresponding to category c, reshape(·) means modifying the shape of the feature vector;

[0030] (22) Calculate the multi-level regional knowledge distillation loss L according to the following formula mrkd :

[0031]

[0032] Among them, ω k represents the weights of different levels, cos(·) represents the calculation of cosine similarity, c∈C t Indicates that c is a new category, c∈C 1:t-1 Indicates that c is the old category and b is the background category.

[0033] Furthermore, the updating process of the prototype of each type of smoke in step (3) is as follows:

[0034]

[0035]

[0036] in, represents the prototype of category c at stage t, B represents the batch size, represents the feature vector at the i-th position in the deep feature map, represents an indicator function (if the condition in the brackets is met, the function value is 1, otherwise, it is 0), Represents the sample x n The corresponding pseudo labels The result of downsampling, |·| represents the number of elements in the set, and 0 represents the zero vector.

[0037] Furthermore, the calculation process of the prototype balanced contrastive learning loss in step (3) includes the following steps:

[0038] (31) Based on the old category pseudo-label, extract the prototype of each old category area in the deep feature map:

[0039]

[0040] Where pooling(·) represents the pooling operation;

[0041] (32) Based on the final pseudo-label, extract the prototype of each category area in the deep feature map:

[0042]

[0043] (33) The prototype balanced contrastive learning loss L is calculated according to the following formula pbcl :

[0044] L pbcl =L old +βL new

[0045]

[0046] f(x1,x2)=exp(cos(x1,x2) / τ)

[0047] Among them, β represents the weight coefficient, N c and N m They represent the number of prototypes corresponding to categories c and m in the batch of samples, exp(·) represents the exponential function, and τ represents the temperature coefficient.

[0048] Furthermore, the calculation process of the overall loss in step (4) includes the following steps:

[0049] (41) Calculate the mBCE loss L based on the prediction results of the student model and the true label mbce , as shown below:

[0050]

[0051] Among them, γ represents the weight coefficient, represents the probability that the student model predicts pixel i as category c;

[0052] (42) Calculate the AC loss L based on the prediction results of the student model ac , as shown below:

[0053]

[0054] in, represents the score of the auxiliary classifier output of the student model;

[0055] (43) Calculate the KD loss L based on the prediction results of the teacher model and the student model respectively kd and DKD loss L dkd , as shown below:

[0056]

[0057] in, and denote the positive inference scores of the teacher model and the student model predicting pixel i as category c, respectively. and denote the negative inference scores of the teacher model and the student model predicting pixel i as category c, respectively;

[0058] (44) The weighted sum of each loss item is obtained to obtain the overall loss, as shown below:

[0059] L=L mbce +L ac +λ kd L kd +λ dkd L dkd +λ mrkd L mrkd +λ pbcl L pbcl

[0060] Among them, λ kd , dkd , mtkd and λ pbcl Represent the weight coefficients of each loss term respectively.

[0061] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The present invention proposes a multi-level regional knowledge distillation strategy, which constrains the changes in the smoke feature representation of the old category region at different levels of the intermediate layer and does not restrict the learning of the smoke feature representation of the new category region, thereby achieving a better stability-plasticity balance; (2) The present invention proposes a prototype balanced contrast learning strategy, which makes each category make an approximate contribution to the optimization of the model through class averaging and class compensation operations, thereby alleviating the classification confusion problem. Combining the above two advantages, the technical invention further improves the segmentation accuracy of the incremental smoke segmentation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0063] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0064] The present invention provides a method for incremental smoke segmentation based on multi-level regional knowledge distillation and prototype balanced contrast learning, and its flow chart is as follows: Figure 1 As shown, the following steps are included:

[0065] Step 1: Input the input image into the encoders of the teacher model and the student model respectively, extract shallow feature maps of different levels, and then fuse the features through the decoder to obtain the deep feature map. The classifier then predicts the result and calculates the mixed pseudo-label.

[0066] Among them, the calculation method of shallow feature maps at different levels includes the following steps (taking the student model as an example, the teacher model is similar):

[0067] Step 1.1: Take an input image I∈R 3×H×W Cropped into a sequence of patches, where each patch is of size P×P and the length of the sequence is N=(H×W) / (P×P), and then the patch sequence is mapped into a D-dimensional vector through the patch embedding operation;

[0068] Step 1.2: Input the D-dimensional vector into the encoder composed of four Transformer blocks to extract four shallow feature maps at different levels, denoted as F e,1 、F e,2 、F e,3 and F e,4 .

[0069] The calculation method of the deep feature map includes the following steps (taking the student model as an example, the teacher model is similar):

[0070] Step 1.3: Four different levels of shallow feature maps F e,1 、F e,2 、F e,3 and F e,4 Perform convolution calculation and upsampling operations respectively to obtain F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 , where the number of channels and spatial size of the output feature map remain consistent;

[0071] Step 1.4: Transform the feature map F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 Concatenate by channel dimension and then perform convolution operation to obtain the deep feature map F d .

[0072] The process of classifier predicting deep feature maps includes the following steps (taking the student model as an example, the teacher model is similar):

[0073] Step 1.5: Transform the deep feature map F d Input into the classifier to obtain the initial prediction result;

[0074] Step 1.6: Upsample the initial prediction result to obtain the final prediction result, which is consistent with the spatial size of the input image.

[0075] The final pseudo-label calculation process includes the following steps:

[0076] Step 1.7: The result L predicted by the teacher model t-1 Perform the sigmoid operation, and then perform the threshold segmentation operation for each channel of each pixel position i. If the predicted value is less than the threshold, the predicted value of the position is set to 0, otherwise, no processing is performed. The purpose of performing threshold segmentation is to prevent low-confidence pseudo-labels from misleading the student model to inherit and retain the smoke features of the old category;

[0077] Step 1.8: Process the predicted result L t-1 Perform the argmax operation to obtain the pseudo label PL of the old category smoke t-1 , and then calculate the final pseudo label according to the following formula:

[0078]

[0079] Among them, y represents the true label of the current step.

[0080] Step 2: Calculate the multi-level regional knowledge distillation loss based on shallow feature maps and pseudo labels at different levels.

[0081] The calculation process of multi-level regional knowledge distillation loss includes the following steps:

[0082] Step 2.1: Extract the feature representation of each category area based on the pseudo-label (taking the student model as an example, the teacher model is similar):

[0083]

[0084] in, represents the kth shallow feature map extracted by the student model, Represents the pseudo-label after downsampling (spatial size and shallow feature map Keep consistent), c∈C 1:t Indicates the category, d k Represents shallow feature map The number of channels, N k.c Represented in the shallow feature map The number of features corresponding to category c, reshape(·) means modifying the shape of the feature vector;

[0085] Step 2.2: Calculate the multi-level regional knowledge distillation loss L according to the following formula mrkd :

[0086]

[0087] Among them, ω k represents the weights of different levels, cos(·) represents the calculation of cosine similarity, c∈C t Indicates that c is a new category, c∈C 1:t-1 Indicates that c is the old category and b is the background category.

[0088] Step 3: Based on the deep feature map and pseudo labels, update the prototype of each category of smoke and calculate the prototype balanced contrastive learning loss.

[0089] The updating process of the prototype of each type of smoke is as follows:

[0090]

[0091] in, represents the prototype of category c at stage t, B represents the batch size, represents the feature vector at the i-th position in the deep feature map, represents an indicator function (if the condition in the brackets is met, the function value is 1, otherwise, it is 0), Represents the sample x n The corresponding pseudo labels The result of downsampling, |·| represents the number of elements in the set, and 0 represents the zero vector.

[0092] The calculation process of prototype balanced contrastive learning loss includes the following steps:

[0093] Step 3.1: Based on the old category pseudo-label, extract the prototype of each old category area in the deep feature map:

[0094]

[0095] Where pooling(·) represents the pooling operation;

[0096] Step 3.2: Based on the final pseudo-label, extract the prototype of each category area in the deep feature map:

[0097]

[0098] Step 3.3: Calculate the prototype balanced contrastive learning loss L according to the following formula pbcl :

[0099] L pbcl =L old +βL new

[0100]

[0101]

[0102] f(x1,x2)=exp(cos(x1,x2) / τ)

[0103] Among them, β represents the weight coefficient, N c and N m They represent the number of prototypes corresponding to categories c and m in the batch of samples, exp(·) represents the exponential function, and τ represents the temperature coefficient.

[0104] Step 4: Perform weighted summation of all losses and perform gradient backpropagation to update the student model.

[0105] The calculation process of the overall loss includes the following steps:

[0106] Step 4.1: Calculate the mBCE loss L based on the student model’s predictions and the true labels mbce , as shown below:

[0107]

[0108] Among them, γ represents the weight coefficient, represents the probability that the student model predicts pixel i as category c;

[0109] Step 4.2: Calculate the AC loss L based on the prediction results of the student model ac , as shown below:

[0110]

[0111] in, represents the score of the auxiliary classifier output of the student model;

[0112] Step 4.3: Calculate the KD loss L based on the prediction results of the teacher model and the student model respectively kd and DKD loss L dkd , as shown below:

[0113]

[0114] in, and denote the positive inference scores of the teacher model and the student model predicting pixel i as category c, respectively. and denote the negative inference scores of the teacher model and the student model predicting pixel i as category c, respectively;

[0115] Step 4.4: Perform a weighted summation of the various loss items to obtain the overall loss, as shown below:

[0116] L=L mbce +L ac +λ kd L kd +λ dkd L dkd +λ mrkd L mrkd +λ pbcl L pbcl

[0117] Among them, λ kd , dkd , mrkd and λ pbcl Represent the weight coefficients of each loss term respectively.

[0118] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A method for incremental smoke segmentation based on multi-level regional knowledge distillation and prototype balanced contrastive learning, characterized in that: The steps include: (1) The input image is input into the encoders of the teacher model and the student model respectively, and shallow feature maps of different levels are extracted. The deep feature maps are then obtained by feature fusion through the decoder. The classifier is then used to predict the result and calculate the mixed pseudo-label. (2) Calculate the multi-level regional knowledge distillation loss based on shallow feature maps and mixed pseudo-labels at different levels; (3) Based on the deep feature map and mixed pseudo-labels, update the prototype of each category of smoke and calculate the prototype balanced contrastive learning loss; (4) All losses are weighted summed and gradient backpropagation is performed to update the student model. The updated student model will be used to perform smoke segmentation on the test sample.

2. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation method of the shallow feature maps of different levels in step (1) includes the following steps (taking the student model as an example, the teacher model is similar): (11) Take an input image I∈R 3×H×W Cropped into a sequence of patches, where each patch is of size P×P and the length of the sequence is N=(H×W) / (P×P), and then the patch sequence is mapped into a D-dimensional vector through the patch embedding operation; (12) The D-dimensional vector is input into an encoder consisting of four Transformer blocks to extract four shallow feature maps at different levels, which are denoted as F e,1 、F e,2 、F e,3 and F e,4 .

3. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation method of the deep feature map in step (1) includes the following steps (taking the student model as an example, the teacher model is similar): (13) For four different levels of shallow feature maps F e,1 、F e,2 、F e,3 and F e,4 Perform convolution calculation and upsampling operations respectively to obtain F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 , where the number of channels and spatial size of the output feature map remain consistent; (14) The feature map F′ e,1 , F′ e,2 , F′ e,3 and F′ e,4 Concatenate by channel dimension and then perform convolution operation to obtain the deep feature map F d .

4. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The process of prediction by the classifier in step (1) includes the following steps (taking the student model as an example, the teacher model is similar): (15) The deep feature map F d Input into the classifier to obtain the initial prediction result; (16) The initial prediction result is upsampled to obtain the final prediction result, which is consistent with the spatial size of the input image.

5. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation process of the final pseudo-label in step (1) includes the following steps: (17) The result L predicted by the teacher model t-1 Perform the sigmoid operation, and then perform the threshold segmentation operation for each channel of each pixel position i. If the predicted value is less than the threshold, the predicted value of the position is set to 0, otherwise, no processing is performed. The purpose of performing threshold segmentation is to prevent low-confidence pseudo-labels from misleading the student model to inherit and retain the smoke features of the old category; (18) The processed prediction result L t-1 Perform the argmax operation to obtain the pseudo label PL of the old category smoke t-1 , and then calculate the final pseudo label according to the following formula: Among them, y represents the true label of the current step.

6. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation process of the multi-level regional knowledge distillation loss in step (2) includes the following steps: (21) Extract the feature representation of each category area based on the pseudo-label (taking the student model as an example, the teacher model is similar): in, represents the kth shallow feature map extracted by the student model, Represents the pseudo-label after downsampling (spatial size and shallow feature map Keep consistent), c∈C 1:t Indicates the category, d k Represents shallow feature map The number of channels, N k.c Represented in the shallow feature map The number of features corresponding to category c, reshape(·) means modifying the shape of the feature vector; (22) Calculate the multi-level regional knowledge distillation loss L according to the following formula mrkd : Among them, ω k represents the weights of different levels, cos(·) represents the calculation of cosine similarity, c∈C t Indicates that c is a new category, c∈C 1:t-1 Indicates that c is the old category and b is the background category.

7. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The updating process of the prototype of each type of smoke in step (3) is as follows: in, represents the prototype of category c at stage t, B represents the batch size, represents the feature vector at the i-th position in the deep feature map, represents an indicator function (if the condition in the brackets is met, the function value is 1, otherwise, it is 0), Represents the sample x n The corresponding pseudo labels The result of downsampling, |·| represents the number of elements in the set, and 0 represents the zero vector.

8. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation process of the prototype balanced contrastive learning loss in step (3) includes the following steps: (31) Based on the old category pseudo-label, extract the prototype of each old category area in the deep feature map: Where pooling(·) represents the pooling operation; (32) Based on the final pseudo-label, extract the prototype of each category area in the deep feature map: (33) The prototype balanced contrastive learning loss L is calculated according to the following formula pbcl : L pbcl =L old +βL new f(x1,x2)=exp(cos(x1,x2) / τ) Among them, β represents the weight coefficient, N c and N m They represent the number of prototypes corresponding to categories c and m in the batch of samples, exp(·) represents the exponential function, and τ represents the temperature coefficient.

9. The incremental smoke segmentation method based on multi-level regional knowledge distillation and prototype balanced contrastive learning according to claim 1 is characterized in that: The calculation process of the overall loss in step (4) includes the following steps: (41) Calculate the mBCE loss L based on the prediction results of the student model and the true label mbce , as shown below: Among them, γ represents the weight coefficient, represents the probability that the student model predicts pixel i as category c; (42) Calculate the AC loss L based on the prediction results of the student model ac , as shown below: in, represents the score of the auxiliary classifier output of the student model; (43) Calculate the KD loss L based on the prediction results of the teacher model and the student model respectively kd and DKD loss L dkd , as shown below: in, and denote the positive inference scores of the teacher model and the student model predicting pixel i as category c, respectively. and denote the negative inference scores of the teacher model and the student model predicting pixel i as category c, respectively; (44) The weighted sum of each loss item is obtained to obtain the overall loss, as shown below: L=L mbce +L ac +λ kd L kd +λ dkd L dkd +λ mrkd L mrkd +λ pvcl L pbcl Among them, λ kd , dkd , mrkd and λ pbcl Represent the weight coefficients of each loss term respectively.

Images