Method for acquiring cross-image pixel-to-boundary segmentation loss function

Abstract

The invention discloses a method for acquiring a cross-image pixel-to-boundary segmentation loss function, which comprises the following steps of: 1, generating boundary feature knowledge of each object in an image by processing a mask of a real label; 2, sequentially adding boundary feature knowledge generated by images in the data set into a vector sequence; 3, counting the KL divergence distance of the target in the vector sequence, and calculating a loss function after distillation; through the above steps, a segmentation loss function from a pixel to a boundary of a cross-image is obtained.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, in particular to image processing technology, and specifically to a method for obtaining a distillation loss function based on neural network image processing. Background Art

[0002] Semantic segmentation currently occupies a large proportion in the field of computer vision research. So far, existing semantic segmentation methods have had a significant impact in many computer vision tasks, and eventually became the basic step for tasks that require more advanced intelligence, such as autonomous driving, home robots, work robots, and image editing. In order to meet the needs of high-level tasks that require high-precision results, the usual segmentation networks are based on a large number of parameters and high computational costs. However, actual resources are always limited. In order to maximize resource utilization, people have painstakingly proposed a series of lightweight segmentation networks, such as ESPet, ICNet, and BiSeNet. In addition, many compact networks have been proposed, such as knowledge distillation (KD).

[0003] Knowledge distillation is a method that builds a lightweight small network model and uses the supervisory information of a larger network model with better performance to train the small network model. Since knowledge distillation mainly uses the effective knowledge learned from the teacher network, this method is more efficient and has received more attention than other lightweight methods. In this process, the loss function plays a vital role. The design of the loss function ensures that the student network not only pays attention to the degree of match with the true label during training, but also pays attention to the knowledge learned from the teacher network. By optimizing the loss function, the student network can fit the generalization ability of the teacher network as much as possible, thereby improving its own performance. The loss function of knowledge distillation usually consists of two parts: one is a conventional loss function, such as the cross entropy loss function (L CE ), which measures the gap between the network prediction value and the true label; the other part is the distillation loss function (L KD ), which represents the distance between the probability distributions output by two networks (teacher network and student network), such as KL divergence.

[0004] The usual distillation loss function calculates the KL divergence between pixels. However, there are always meaningful similar parts of pixels in the same image, such as pixels in the same object. Masks of multiple objects in the image always have overlapping parts. After calculating the similarity of all pixels to pixels, many pixels will have very similar results. Boundary features are very helpful in highlighting the differences between regional features, which can improve the accuracy of semantic segmentation. Moreover, boundary information greatly reduces the amount of data embedded in information compared to the entire pixel information and regional information. In order to make accurate boundary knowledge useful for training and passing it to the student network, a label mask can be used to generate boundary representations. The researchers calculated the boundary of each object and the partial pixels in the area where the boundary is removed in the mask to obtain the relationship between boundary and regional feature knowledge. However, only the pixel-level relationship of a single image can be obtained, and the segmentation dependency of all pixels in all images across the current dataset cannot be reconstructed. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problem that the prior art can only obtain the pixel-level relationship of a single image and cannot reconstruct the segmentation dependency of all pixels in all images across the current data set, and to provide a pixel-to-boundary segmentation loss function across images. After the calculation of the loss function designed by the present invention, more accurate boundary information can be extracted, and the boundary feature knowledge of the global image can be linked and learned from each other, thereby improving the segmentation result.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for obtaining a pixel-to-boundary segmentation loss function across an image comprises the following steps:

[0008] Step 1: Generate boundary feature knowledge of each object in the image by processing the mask of the true label;

[0009] Step 2: Add the boundary feature knowledge generated by the images in the dataset to the vector sequence in sequence;

[0010] Step 3: Count the KL divergence distance of the target in the vector sequence and calculate the loss function after distillation;

[0011] The above steps are used to obtain the pixel-to-boundary segmentation loss function across the image.

[0012] In step 1, the steps to process the mask of the true label are as follows:

[0013] 1-1) Input the mask O of the true label r ;

[0014] 1-2) Process the true label mask through the average pooling layer: M l=avg pool (O r ), to obtain a shape that can adapt to the predictions generated by the student and teacher networks,

[0015] 1-3) Apply dilation operation on the label mask (M l ) and Erosion(M l ) to highlight the boundaries of objects and maintain their outlines;

[0016] 1-4) Calculate the morphological gradient of the contour, the formula used is: M edge =Dilation(M l )-Erosion(M l ), thus we get the boundary representation M edge , this M edge Has the same shape as the segmentation network.

[0017] In step 2, the following steps are specifically included:

[0018] 2-1) From the teacher similarity matrix, represent M according to the boundary in step 1 edge Take out the boundary pixel embedding s of each category c c , the formula is as follows:

[0019]

[0020] 2-2) Use a queue-like vector sequence D e To save a large number of boundary representations. The shape of the boundary queue is represented as: Where E e is the number of boundary embeddings for each category in the current image, and d is the embedding size;

[0021] 2-3) Embedding s by boundary pixels of multiple categories c The boundary similarity matrix S is formed e , and put it into the boundary queue; the queue is output with the comparison quantity C e Boundary embedding for the target number of random indices Specifically expressed as:

[0022] 2-4) Compose the pixel-to-boundary similarity matrix S of all the images in the queue e and the output of the teacher network is modeled as the boundary embedding F e . Similarly, this step is applied to the output of the student network. It is expressed as:

[0023]

[0024] 2-5) Repeat the above steps on the training set images until the queue is full. According to the characteristics of the queue, the advanced data is deleted and the queue is updated by the late data.

[0025] The calculated boundary embedding is used for subsequent loss function calculation.

[0026] In step 3, the following steps are specifically included:

[0027] 3-1) Calculate the pixel-to-boundary embedding F from step 2 e Finally, the divergence distance between the teacher network and the student network is calculated using the typical KD function KL divergence

[0028] 3-2) By optimizing the distance loss, the distance is minimized iteratively to reduce the gap between the student network and the teacher network. The loss function formula for boundary embedding is calculated as follows:

[0029]

[0030] 3-3) Using the above process, perform semantic segmentation test on the CamVid dataset.

[0031] Compared with the prior art, the present invention has the following technical effects:

[0032] 1) The present invention improves the loss function by calculating the distance from the pixel points stored in the memory in the form of a queue to the boundary, comparing the distance results of the teacher network and the student network, and continuously optimizing the loss function, so as to mine the relationship between the global boundary feature knowledge across images;

[0033] 2) The present invention adopts knowledge distillation technology to enable the student network with smaller parameters to learn from the teacher network to reduce the model size;

[0034] 3) The present invention temporarily stores the boundary knowledge of each target in the image into the memory and analyzes the global boundary feature knowledge to improve the accuracy of boundary segmentation in semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings and examples:

[0036] Figure 1 It is the overall flow chart of the present invention;

[0037] Figure 2 for Figure 1 The network diagram in which boundary feature data is generated and pushed to the boundary queue. DETAILED DESCRIPTION

[0038] A method for obtaining a pixel-to-boundary segmentation loss function across an image comprises the following steps:

[0039] Step 1: Generate boundary feature knowledge of each object in the image by processing the mask of the true label;

[0040] Based on the existing mask labels of the dataset images, we first extract the boundary representation through the average pooling layer, which can effectively prevent the loss of several object types. In step 1, the steps for processing the mask of the real label are as follows:

[0041] 1) Input the mask O of the true label r ;

[0042] 2) Process the true label mask through the average pooling layer: M l =avg pool (O r ), to obtain a shape that can adapt to the predictions generated by the student and teacher networks,

[0043] 3) Apply dilation operation on the label mask (M l ) and Erosion(M l ) to highlight the boundaries of objects and maintain their outlines;

[0044] 4) Calculate the morphological gradient of the contour using the formula: M edge =Dilation(M l )-Erosion(M l ), thus we get the boundary representation M edge , this M edge Has the same shape as the segmentation network.

[0045] Step 2: Add the boundary feature knowledge generated by the images in the dataset to the vector sequence in sequence

[0046] 1) From the teacher similarity matrix, represent M according to the boundary in step 1 edge Take out the boundary pixel embedding s of each category c c , the formula is as follows:

[0047]

[0048] 2) Use a queue-like vector sequence D e To save a large number of boundary representations. The shape of the boundary queue is represented as: Where E e is the number of boundary embeddings for each category in the current image, and d is the embedding size;

[0049] 3) Embedding s by boundary pixels of multiple categories c The boundary similarity matrix S is formed e , and put it into the boundary queue; the queue is output with the comparison quantity C e Boundary embedding for the target number of random indices Specifically expressed as:

[0050] 4) Compute the pixel-to-boundary similarity matrix S of all the images in the queue e and the output of the teacher network is modeled as the boundary embedding F e . Similarly, this step is applied to the output of the student network. It is expressed as:

[0051]

[0052] 5) Repeat the above steps on the training set images until the queue is full. According to the characteristics of the queue, the advanced data is deleted and the queue is updated by the late data.

[0053] The calculated boundary embedding is used for subsequent loss function calculation.

[0054] Step 3: Count the KL divergence distance between the teacher and the student in the vector sequence and calculate the loss function after distillation

[0055] 1) Calculate the pixel-to-boundary embedding F from step 2 e Finally, the divergence distance between the teacher network and the student network is calculated using the typical KD function KL divergence

[0056] 2) By optimizing the distance loss, the distance is minimized iteratively to reduce the gap between the student network and the teacher network. The loss function formula for boundary embedding is calculated as follows:

[0057]

[0058] 3) Using the above process, perform semantic segmentation test on the CamVid dataset.

[0059] Example:

[0060] The present invention was tested on a server with an RTX 3080x2 graphics card and 20GB of video memory. At the same time, the operating system of the server is ubuntu18.04, and the versions of other tools are python3.8, pytorch1.8.1, and cuda11.1. SGD is used for the loss optimization method, with the default momentum of 0.9, weight decay (0.0001), and a total of 40,000 iterations. The batch size during training depends on the cropping size and is usually set to 8. The initial learning rate is set to 0.02, and is multiplied by 0.01 after each round of training according to the polynomial annealing strategy. For the Camvid dataset, we set the crop size to 360x360. For data augmentation, we randomly apply flip and scale to the input images in the range of [0.5, 2].

[0061] In order to obtain more average and representative results, we use mean Intersection over Union (mIoU) to measure the performance of segmentation results.

[0062] As shown in Table 1, after testing multiple distillation methods on the Camvid training set, it is shown that the distillation method with the pixel-to-boundary loss function across images of the present invention is the best, and the semantic segmentation effect of the student network is closest to the teacher network.

[0063] Table 1 Performance comparison of different distillation methods on the Camvid training set

[0064]

[0065] As shown in Table 2, after ablation experiments were conducted on the Camvid validation set, different distillation loss results indicate that the distillation method of the loss function from pixels to boundaries across images of the present invention can better indicate the differences between features of different regions, thereby improving the accuracy of semantic segmentation.

[0066] Table 2 Performance comparison of different loss functions

[0067]

[0068] The results of various tests show that after quantification by the mIoU measurement indicator, the semantic segmentation effect has reached an accuracy of 76.3%. The distillation method of the loss function from pixels to boundaries across images of the present invention improves the recognition effect of boundary information, thereby improving the accuracy of the semantic segmentation model, and also proves the applicability of this solution.

Claims

1. A method for obtaining a pixel-to-boundary segmentation loss function across an image, characterized in that: The following steps are involved: Step 1: Generate boundary feature knowledge of each object in the image by processing the mask of the true label; Step 2: Add the boundary feature knowledge generated by the images in the dataset to the vector sequence in sequence; Step 3: Count the KL divergence distance of the target in the vector sequence and calculate the loss function after distillation; The above steps are used to obtain the pixel-to-boundary segmentation loss function across the image.

2. The method according to claim 1, characterized in that In step 1, the steps to process the mask of the true label are as follows: 1-1) Input the mask Or of the real label; 1-2) The true label mask is processed through an average pooling layer: Ml = avgpool(Or) to obtain a shape that can adapt to the predicted values ​​generated by the student and teacher networks. 1-3) Apply dilation(Ml) and erosion(Ml) operations on the label mask to highlight the boundaries of the object and maintain the outline of the object; 1-4) Calculate the morphological gradient of the contour using the formula: Medge = Dilation(Ml) - Erosion(Ml), thereby obtaining the boundary representation Medge, which has the same shape as the segmentation network.

3. The method according to claim 2, characterized in that In step 2, the following steps are specifically included: 2-1) From the teacher similarity matrix, extract the boundary pixel embedding sc of each category c according to the boundary representation Medge in step 1, and obtain the formula as follows: 2-2) Use a queue-shaped vector sequence De to store a large number of boundary representations; the shape of the boundary queue is represented as: Where Ee is the number of boundary embeddings for each category in the current image, and d is the embedding size; 2-3) The boundary similarity matrix Se is formed by embedding the boundary pixels of multiple categories sc and put into the boundary queue; the output of the queue is the boundary embedding with random indexes with the comparison number Ce as the target number Specifically expressed as: 2-4) Model the pixel-to-boundary similarity matrix Se composed of all images in the queue and the output of the teacher network as the boundary embedding Fe; similarly apply this step to the output of the student network; expressed as: 2-5) Repeat the above steps on the training set images until the queue is full; according to the characteristics of the queue, the advanced data is deleted and the queue is updated by the late data; The calculated boundary embedding is used for subsequent loss function calculation.

4. The method according to claim 3, characterized in that In step 3, the following steps are specifically included: 3-1) After calculating the pixel-to-boundary embedding Fe in step 2, the divergence distance between the teacher network and the student network is calculated using the typical KD function KL divergence 3-2) By optimizing the distance loss, the distance is minimized iteratively to reduce the gap between the student network and the teacher network; the loss function formula for boundary embedding is calculated as follows: 3-3) Using the above process, perform semantic segmentation test on the CamVid dataset.

Images