A bilateral edge image segmentation method based on representative features

Abstract

This invention discloses a dual-edge image segmentation method based on representational features. First, an image segmentation model is constructed. Then, training images are fed into the constructed image segmentation model for training, resulting in a trained image segmentation model. Finally, the image to be segmented is fed into the trained image segmentation model for prediction, completing the image segmentation. The method optimizes the segmentation results using representational edge information, directly optimizing the inference results from the representational features of the original image. It does not use any feature extraction module and is not a dual-branch network. The optimization process parameters can be unconditionally added to any scene segmentation network and used directly. The loss function focuses on solving the problem of a severe imbalance between the number of edge pixels and non-edge pixels, improving the performance of edge-based segmentation networks and can be used for edge detail optimization in other networks. The dropout method for edge pixels addresses the imbalance between the number of pixels at the object's edge and inside the object, preventing overfitting in edge optimization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a double-edge image segmentation method based on representation features. BACKGROUND

[0002] In the field of computer vision, image segmentation refers to a process of dividing pixels in a picture into different categories (or marked with different labels) according to certain rules, which is widely used in target recognition and detection, object tracking, automatic driving, medical image processing and other fields. The image segmentation method based on deep learning uses an image segmentation model to understand the real object represented by each pixel of the image, trains the image segmentation model through the segmented training image, and uses the trained image segmentation model to predict the unsegmented test image. When the image is segmented by using the method of deep learning, the accuracy of the edge of the segmentation result is an important problem to be considered, especially the segmentation accuracy of the edge position. In the current mainstream image segmentation model, the main reason affecting the segmentation accuracy is that the down-sampling process of the convolutional neural network causes the gradual loss of image details. SUMMARY

[0003] The present application solves the problem of the gradual loss of image details caused by the down-sampling process of the existing image segmentation model, and provides a double-edge image segmentation method based on representation features.

[0004] To solve the above problems, the present application is realized by the following technical scheme:

[0005] A double-edge image segmentation method based on representation features, comprising the following steps:

[0006] Step 1, constructing an image segmentation model; the segmentation network is mainly composed of a backbone network, an edge extraction module, an up-sampling module and an edge refinement module; wherein the backbone network is a deep residual network; the input of the backbone network and the input of the edge extraction module together form the first input of the image segmentation model; the output of the backbone network is connected to the second input of the edge refinement module through the up-sampling module; the first stage output of the backbone network is connected to the third input of the edge refinement module; the output of the edge extraction module is connected to another input of the edge refinement module; and the output of the edge extraction module forms the output of the image segmentation model;

[0007] Step 2, in training, the training image is sent into the image segmentation model constructed in step 1 for training to obtain a trained image segmentation model; in the training process, the backbone network extracts deep features of the image, the first stage of the backbone network extracts shallow features of the image, and the edge extraction module extracts an edge heat map of the image; the up-sampling module up-samples the deep features; the edge refinement module first calculates a class loss using the up-sampled deep features, calculates a segmentation loss using the shallow features, and calculates an edge loss using the edge heat map, and then takes a weighted sum of the class loss, the segmentation loss and the edge loss as a loss function of the image segmentation model;

[0008] Step 3, in prediction, the image to be segmented is sent into the trained image segmentation model obtained in step 2 for prediction to complete image segmentation of the image to be segmented.

[0009] In the above step 1, the edge refinement module is composed of 5 convolution layers, 2 pixel multiplication layers, 3 pixel addition layers, 2 batch normalization layers, and 1 Sigmoid activation function layer; an input of the Sigmoid activation function layer forms a first input of the edge refinement module, one input of the first pixel multiplication layer forms a second input of the edge refinement module, and the first convolution layer forms a third input of the edge refinement module; an output of the first Sigmoid activation function layer is connected to one input of the first pixel addition layer and the second pixel addition layer; one output of the first Sigmoid activation function layer is connected to the other input of the first pixel multiplication layer, and the other output of the first Sigmoid activation function layer is connected to one input of the second pixel multiplication layer; an output of the first pixel multiplication layer is connected to the other input of the first pixel addition layer, an output of the first pixel addition layer is connected to an input of the second convolution layer, an output of the second convolution layer is connected to an input of the first batch normalization layer, an output of the first batch normalization layer is connected to an input of the third convolution layer; an output of the first convolution layer is connected to the other input of the second pixel multiplication layer, an output of the second pixel multiplication layer is connected to the other input of the second pixel addition layer, an output of the second pixel addition layer is connected to an input of the fourth convolution layer, an output of the fourth convolution layer is connected to an input of the second batch normalization layer, and an output of the second batch normalization layer is connected to an input of the fifth convolution layer; an output of the third convolution layer is connected to one input of the third pixel addition layer, an output of the fifth convolution layer is connected to the other input of the third pixel addition layer, and an output of the third pixel addition layer forms an output of the edge refinement module.

[0010] In the step 1, the edge extraction module is composed of 3 Gaussian convolution layers, 2 x direction derivative convolution layers, 2 y direction derivative convolution layers, 2 batch normalization layers, 2 pixel addition layers, 2 ReLU activation function layers, 1 channel splicing layer, and 1 Sigmoid activation function layer; the outputs of the first and second Gaussian convolution layers form the input of the edge extraction module; the output of the first Gaussian convolution layer is connected to the input of the third batch normalization layer, the two outputs of the third batch normalization layer are respectively connected to the inputs of the first x direction derivative convolution layer and the first y direction derivative convolution layer, the outputs of the first x direction derivative convolution layer and the first y direction derivative convolution layer are respectively connected to the two inputs of the fourth pixel addition layer, the output of the fourth pixel addition layer is connected to the input of the first ReLU activation function layer, and the output of the first ReLU activation function layer is connected to one input of the channel splicing layer; the output of the second Gaussian convolution layer is connected to the input of the fourth batch normalization layer, the two outputs of the fourth batch normalization layer are respectively connected to the inputs of the second x direction derivative convolution layer and the second y direction derivative convolution layer, the outputs of the second x direction derivative convolution layer and the second y direction derivative convolution layer are respectively connected to the two inputs of the fifth pixel addition layer, the output of the fifth pixel addition layer is connected to the input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is connected to the other input of the channel splicing layer; the output of the channel splicing layer is connected to the input of the third Gaussian convolution layer, the output of the third Gaussian convolution layer is connected to the input of the second Sigmoid activation function layer, and the output of the second Sigmoid activation function layer forms the output of the edge extraction module.

[0011] In the step 1, the backbone network is a deep residual network ResNet50.

[0012] In the step 2, the edge extraction module randomly inactivates the input image to obtain an edge heat map, wherein the random inactivation strategy is random inactivation according to uncertainty or random inactivation according to a random inactivation rate.

[0013] The process of random inactivation according to uncertainty is as follows: ① generate an empty mask vector, the rows and columns of the all-zero vector correspond to the channels and pixels of the input image respectively; ② traverse each pixel in each channel of the input image, if the pixel value of the current pixel in the current channel is greater than α and less than β, then the value at the corresponding position in the mask vector is set to 1; otherwise, the value at the corresponding position in the mask vector is set to 0, and the obtained vector is the random inactivation mask; ③ multiply the random inactivation mask with the input image to obtain the edge heat map; wherein α is the lower limit of the set discard threshold, and β is the upper limit of the set discard threshold;

[0014] The process of random inactivation according to the random inactivation rate is: ① a linear random number sequence with a length of W*H is first generated randomly, and the value range is [0, 1), and then it is converted into a mask tensor with a width of W and a height of H; ② the part of the mask tensor with a pixel value less than or equal to δ is set to 0, and the part greater than δ is set to 1, and then it is expanded in dimension to the same dimension as the input image, that is, C copies are copied in the dimension channel, and thus a random inactivation mask is obtained; ③ the random inactivation mask and the input image are multiplied, and an edge heat map is obtained; wherein δ is the set random inactivation rate, W, H and C are the length, width and depth of the input image respectively.

[0015] Compared with the prior art, the present application has the following characteristics:

[0016] 1. The edge information is used to optimize the segmentation result, the feature of the original image is directly used to optimize the inference result, any feature extraction module is not used, it does not belong to a double-branch network, the optimization process parameters can be unconditionally added to any scene segmentation network for direct use;

[0017] 2. A double-edge based target function (loss function) is proposed, which focuses on solving the serious imbalance problem between edge pixels and non-edge pixels, improves the effect of the above-mentioned edge-based segmentation network, and can be used for edge detail optimization of other networks;

[0018] 3. An edge pixel dropout method is proposed in cooperation with the optimization method based on variables, which solves the imbalance between object edge pixels and object internal pixels and prevents overfitting in edge optimization. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 It is a flowchart of a double-edge image segmentation method based on a feature representation, (a) training stage, (b) prediction stage.

[0020] Figure 2 It is a schematic diagram of the image segmentation model of the present application.

[0021] Figure 3 It is a schematic diagram of the edge extraction module of the present application.

[0022] Figure 4 It is a visual image of the edge heat map in the driving scene.

[0023] Figure 5 It is a schematic diagram of the edge refinement module of the present application.

[0024] Figure 6 It is a part of the edge visualization result after edge dropout. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with specific examples.

[0026] Referring to Figure 1 A bilateral edge image segmentation method based on feature representation, comprising the following steps:

[0027] Step 1, constructing an image segmentation model.

[0028] The segmentation network is mainly composed of a backbone network, an edge extraction module, an up-sampling module and an edge refinement module, as shown in Figure 2 The input of the backbone network and the input of the edge extraction module together form the first input of the image segmentation model. The output of the backbone network is connected to the second input of the edge refinement module via the up-sampling module. The first stage output of the backbone network is connected to the third input of the edge refinement module. The output of the edge extraction module is connected to another input of the edge refinement module. The output of the edge extraction module forms the output of the image segmentation model.

[0029] The backbone network of the present application is a deep residual network, which is widely used in target classification and other fields, and is a typical convolutional network as part of the classic neural network backbone of computer vision tasks. In the preferred embodiment of the present application, a deep residual network ResNet50 containing 50 convolution operations is selected, and the specific results are shown in Table 1:

[0030]

[0031] In the first stage of the backbone network, the input successively passes through the convolution layer, the batch normalization layer, the activation function ReLU, and the maximum pooling layer to obtain an output with a shape of (256, 56, 56). The output of this stage will be one input of the edge refinement module. The output channel number of the second, third and fourth stages of the backbone network gradually increases, and the size gradually decreases. The total output of the backbone network will be another input of the edge refinement module.

[0032] The edge extraction module is composed of 3 Gaussian convolution layers, 2 x-direction derivative convolution layers, 2 y-direction derivative convolution layers, 2 batch normalization layers, 2 pixel addition layers, 2 ReLU activation function layers, 1 channel concatenation layer, and 1 Sigmoid activation function layer, as shown in Figure 3Gaussian(*) represents a Gaussian convolution layer using a Gaussian kernel of size * as the convolution kernel, G(x) represents an x-direction derivative convolution layer using an x-direction derivative operator as the convolution kernel, and G(y) represents a y-direction derivative convolution layer using a y-direction derivative operator as the convolution kernel. The outputs of the first and second Gaussian convolution layers form the input of the edge extraction module. The output of the first Gaussian convolution layer is connected to the input of the third batch normalization layer, the two outputs of the third batch normalization layer are respectively connected to the inputs of the first x-direction derivative convolution layer and the first y-direction derivative convolution layer, the outputs of the first x-direction derivative convolution layer and the first y-direction derivative convolution layer are respectively connected to the two inputs of the fourth pixel addition layer, the output of the fourth pixel addition layer is connected to the input of the first ReLU activation function layer, and the output of the first ReLU activation function layer is connected to one input of the channel concatenation layer. The output of the second Gaussian convolution layer is connected to the input of the fourth batch normalization layer, the two outputs of the fourth batch normalization layer are respectively connected to the inputs of the second x-direction derivative convolution layer and the second y-direction derivative convolution layer, the outputs of the second x-direction derivative convolution layer and the second y-direction derivative convolution layer are respectively connected to the two inputs of the fifth pixel addition layer, the output of the fifth pixel addition layer is connected to the input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is connected to the other input of the channel concatenation layer. The output of the channel concatenation layer is connected to the input of the third Gaussian convolution layer, the output of the third Gaussian convolution layer is connected to the input of the second Sigmoid activation function layer, and the output of the second Sigmoid activation function layer forms the output of the edge extraction module.

[0033] The edge extraction module is a convolutional implementation of a feature extraction module that extracts edges of an image without the need for training. The output of the edge extraction module is used to guide the optimization process of the subsequent edge refinement module. The edge extraction module accepts an original image as input and outputs an edge heat map. The forward propagation process of the edge extraction module is as follows:

[0034] 1) The input X is convolved with 5x5 and 7x7 kernels that are Gaussian kernels, smoothed, and batch normalized. Taking the 7x7 Gaussian kernel as an example, the process can be described as follows:

[0035]

[0036] where * represents convolution operation. The generating function of the above Gaussian kernel and any size Gaussian kernel is as follows:

[0037]

[0038] 2) The smoothed image is processed by x and y direction derivative operators, added, and activated by an activation function to obtain an original edge map. The operation can be described as follows:

[0039]

[0040] Finally, pass through ReLU activation, i.e. Out x = ReLU(G(x)), Out y = ReLU(G(y)), where * represents convolution operation;

[0041] 3) Take the above Out x , Out y Concatenate features in the channel dimension, smooth using a 3x3 Gaussian kernel, and then activate to obtain the final output.

[0042] The output of the edge extraction module is called an edge heat map, or edge probability map, which has a channel number equal to the number of classes, and each pixel position is measured by a decimal a∈[0,1] to measure the possibility of the existence of an edge at a certain position. 0 represents a very small possibility, and 1 represents a very large possibility. Figure 4 The visualization image of the edge heat map in the driving scene.

[0043] The edge refinement module is composed of 5 convolution layers, 2 pixel multiplication layers, 3 pixel addition layers, 2 batch normalization layers, and 1 Sigmoid activation function layer, as shown in Figure 5 The input of the Sigmoid activation function layer forms the first input of the edge refinement module, one input of the first pixel multiplication layer forms the second input of the edge refinement module, and the first convolution layer forms the third input of the edge refinement module. The output of the first Sigmoid activation function layer is connected to one input of the first pixel addition layer and the second pixel addition layer. One output of the first Sigmoid activation function layer is connected to the other input of the first pixel multiplication layer, and the other output of the first Sigmoid activation function layer is connected to one input of the second pixel multiplication layer. The output of the first pixel multiplication layer is connected to the other input of the first pixel addition layer, the output of the first pixel addition layer is connected to the input of the second convolution layer, the output of the second convolution layer is connected to the input of the first batch normalization layer, and the output of the first batch normalization layer is connected to the input of the third convolution layer. The output of the first convolution layer is connected to the other input of the second pixel multiplication layer, the output of the second pixel multiplication layer is connected to the other input of the second pixel addition layer, the output of the second pixel addition layer is connected to the input of the fourth convolution layer, the output of the fourth convolution layer is connected to the input of the second batch normalization layer, and the output of the second batch normalization layer is connected to the input of the fifth convolution layer. The output of the third convolution layer is connected to one input of the third pixel addition layer, the output of the fifth convolution layer is connected to the other input of the third pixel addition layer, and the output of the third pixel addition layer forms the output of the edge refinement module.

[0044] The input image is up-sampled after the backbone network to obtain a coarse segmentation result, and the shallow feature map of the backbone network is reserved for subsequent refinement. At the same time, the input image is subjected to an edge extraction module to obtain an edge heat map. The edge refinement module fuses the edge details of the shallow features output by the first stage of the backbone network and the coarse segmentation result obtained by up-sampling the backbone network after the backbone network to generate an output result of edge refinement, and the forward propagation process is as follows:

[0045] 1) Deep features (output by the backbone network) are activated to obtain activated deep features. The part of the features is filtered by the confidence of the activation value. When the default threshold is 1, it is specifically represented as a confident a and an unconfident 1-a in operation;

[0046] 2) The unconfident part of the activated deep feature value is multiplied by the edge heat map, and the result is added to the original deep feature;

[0047] 3) The confident part of the activated deep feature value is multiplied by the shallow feature, and the result is added to the original deep feature;

[0048] 4) The results obtained in 2) and 3) are respectively subjected to 3x3 convolution and 1x1 convolution and are added after standardization to form the output result.

[0049] In the edge refinement module, the edge heat map should be described as an edge probability map, the number of channels of which is equal to the number of classes, which is used to represent the possibility of a pixel position being an object edge. The main idea of the edge refinement module is to distinguish between "confident" and "unconfident" parts through the activation value of the deep feature. The confident part indicates that the shallow feature can be trusted, and the shallow feature is directly used for refinement. The unconfident part still believes in the deep feature, but uses the edge feature for detail enhancement.

[0050] Step 2, during training, the training image is sent into the image segmentation model constructed in step 1 for training to obtain a trained image segmentation model.

[0051] The training process of the image segmentation model is as follows:

[0052] 1) The input image obtains shallow features in the first stage of the backbone network and deep features at the end of the backbone network; a refinement mask generated from the shallow features calculates a segmentation loss, and a low-resolution coarse segmentation mask generated by up-sampling the deep features calculates a class loss;

[0053] 2) The input image is subjected to an edge feature extraction module to generate an edge feature map representing an edge heat map, and an edge loss (the edge loss is used to measure the refinement degree of the segmentation result) is calculated after the edge heat map is subjected to edge dropout (random inactivation); Figure 6 Part of the edge visualization result generated after edge dropout;

[0054] The edge generated due to edge longevity is not necessarily a semantic edge, i.e. it can include the external contour of an object, and also contains internal textures. During training, on a specific data set, this feature can cause the model to overfit on the internal textures close to the contour. The present application designs edge dropout to prevent overfitting of edge positions, which does not participate in the inference process itself, but only in training before the edge loss is calculated. The edge extraction module randomly inactivates the input image to obtain the edge heat map, where the random inactivation strategy is random inactivation according to uncertainty or random inactivation according to a random inactivation rate.

[0055] The process of random inactivation according to uncertainty is: ① generate an empty mask vector, the all-zero vector has rows and columns corresponding to the channels and pixels of the input image, respectively; ② traverse each pixel in each channel of the input image, if the pixel value of the current pixel in the current channel is greater than α and less than β, then the value at the corresponding position in the mask vector is set to 1; otherwise, the value at the corresponding position in the mask vector is set to 0, thus obtaining the random inactivation mask; ③ multiply the random inactivation mask with the input image to obtain the edge heat map; where α is the lower limit of the set dropout threshold, and β is the upper limit of the set dropout threshold;

[0056] The process of random inactivation according to a random inactivation rate is: ① first generate a linear random number sequence with a length of WxH, with a value range of [0, 1), and then convert it to a mask tensor with a width of W and a height of H; ② set the part of the mask tensor whose pixel value is less than or equal to δ to 0, and set the part whose value is greater than δ to 1, then expand its dimensions to the same as the input image, i.e. copy C times in the dimension channel, thus obtaining the random inactivation mask; ③ multiply the random inactivation mask with the input image to obtain the edge heat map; where δ is the set random inactivation rate, W, H, C are the length, width and depth of the input image, respectively.

[0057] 3) Calculate the overall loss using the segmentation loss, class loss and edge loss obtained by 2) and 3) as the loss function of the image segmentation model;

[0058] When training the image segmentation model, the joint loss function from the edge, coarse segmentation result and final result is used to optimize the network. Let the edge loss function be L bound , the segmentation loss be L seg , and the class loss be L catagory , then the loss function for training the image segmentation model is:

[0059] L=λL bound +γL seg +θL catagory

[0060] where λ, γ, θ are the weight parameters of edge, segmentation and class respectively, which can be adjusted according to the use scenario of the network. For example, if the scenario requires higher edge segmentation accuracy, the values of λ and γ can be appropriately increased.

[0061] ①Segmentation loss L seg The calculation is performed using MaskIoU.

[0062] ②Class loss L catagory The calculation is performed using the coarse segmentation result, and the main goal is to optimize whether the semantic of the segmentation result is roughly accurate, without considering the edge refinement degree. The loss function is expressed as:

[0063]

[0064] In the formula, G is groundtruth, G b is the edge heat map from groundtruth, and P is the network prediction output. The loss is not sensitive to the edge, but sensitive to the overall segmentation class accuracy.

[0065] ③Edge loss L bound The edge fitting is optimized, and the number of object edge pixels and object internal pixels is considered to be unbalanced. The normalized edge loss L bound is calculated at the edge position and expressed as:

[0066]

[0067] In the formula, g i,c represents the edge probability map from groundtruth (the label of the input image), p i,c represents the edge probability map after edge dropout, c represents the channel, and i represents the pixel.

[0068] For all pixels, only when the edge probability map calculated from groundtruth and the edge probability map calculated from the prediction output are both uncertain (i.e., the pixel values at the position are neither 0 nor 1) will they be recorded as part of L bound .

[0069] 4) The refined mask generated by the edge refinement module is the final output of the network. After that, the network needs to be back propagated according to the above three losses.

[0070] Step 3, during prediction, the image to be segmented is sent into the trained image segmentation model obtained in step 2 for prediction, and the image segmentation of the image to be segmented is completed.

[0071] It should be noted that the above embodiments of the present application are illustrative, and not restrictive, of the present application, and thus the present application is not limited to the above specific embodiments. Other embodiments of the present application, which are within the scope of the present application, will be apparent to those skilled in the art from the disclosure herein.

Claims

1. A method for bilaterally edge image segmentation based on characterizing features, characterized in that, The steps comprise the following: Step 1, constructing an image segmentation model; the image segmentation network is composed of a backbone network, an edge extraction module, an up-sampling module and an edge refinement module; wherein the backbone network is a deep residual network; the input of the backbone network and the input of the edge extraction module jointly form the first input of the image segmentation model; the output of the backbone network is connected to the second input of the edge refinement module through the up-sampling module; the first stage output of the backbone network is connected to the third input of the edge refinement module; the output of the edge extraction module is connected to another input of the edge refinement module; the output of the edge refinement module forms the output of the image segmentation model; Step 2, during training, the training image is sent into the image segmentation model constructed in step 1 for training to obtain a trained image segmentation model; during the training process, the backbone network extracts deep features of the image, the first stage of the backbone network extracts shallow features of the image, and the edge extraction module extracts an edge heat map of the image; the up-sampling module up-samples the deep features; The edge refinement module uses edge information from the edge heat map to fuse the coarse segmentation result obtained by the backbone network after up-sampling and the edge details of the shallow features output by the first stage of the backbone network to generate an output result of edge refinement, and the forward propagation process thereof is as follows: 1) the deep features output by the backbone network are activated to obtain activated deep features, and the part of the features that is not trusted is filtered according to the different activation values, and when the default threshold is 1, the operation is specifically represented as trusted a and untrusted 1-a; 2) the untrusted part of the activated deep feature value is multiplied by the edge heat map, and the result is added to the original deep feature; 3) the trusted part of the activated deep feature value is multiplied by the shallow feature, and the result is added to the original deep feature; 4) the results obtained in 2) and 3) are respectively subjected to 3x3 convolution and 1x1 convolution and standardization and then added to form the output result; the edge refinement module calculates a class loss using the up-sampled deep features, a segmentation loss using the shallow features, and an edge loss using the edge heat map, and then takes the weighted sum of the class loss, the segmentation loss and the edge loss as the loss function of the image segmentation model; Step 3, during prediction, the image to be segmented is sent into the trained image segmentation model obtained in step 2 for prediction to complete the image segmentation of the image to be segmented.

2. The method according to claim 1, characterized in that, In step 1, the edge refinement module is composed of 5 convolution layers, 2 pixel multiplication layers, 3 pixel addition layers, 2 batch normalization layers and 1 Sigmoid activation function layer; The input of the Sigmoid activation function layer forms the first input of the edge refinement module, one input of the first pixel multiplication layer forms the second input of the edge refinement module, and the first convolution layer forms the third input of the edge refinement module; The output of the first Sigmoid activation function layer is connected to one input of the first pixel addition layer and the second pixel addition layer; one output of the first Sigmoid activation function layer is connected to the other input of the first pixel multiplication layer, and the other output of the first Sigmoid activation function layer is connected to one input of the second pixel multiplication layer; The output of the first pixel multiplication layer is connected to another input of the first pixel addition layer, the output of the first pixel addition layer is connected to an input of the second convolution layer, the output of the second convolution layer is connected to an input of the first batch normalization layer, and the output of the first batch normalization layer is connected to an input of the third convolution layer; The output of the first convolution layer is connected to another input of the second pixel multiplication layer, the output of the second pixel multiplication layer is connected to another input of the second pixel addition layer, the output of the second pixel addition layer is connected to an input of the fourth convolution layer, the output of the fourth convolution layer is connected to an input of the second batch normalization layer, and the output of the second batch normalization layer is connected to an input of the fifth convolution layer; The output of the third convolution layer is connected to one input of the third pixel addition layer, the output of the fifth convolution layer is connected to another input of the third pixel addition layer, and the output of the third pixel addition layer forms the output of the edge refinement module.

3. The method according to claim 1, characterized in that, In step 1, the edge extraction module is composed of 3 Gaussian convolution layers, 2 x direction derivative convolution layers, 2 y direction derivative convolution layers, 2 batch normalization layers, 2 pixel addition layers, 2 ReLU activation function layers, 1 channel splicing layer, and 1 Sigmoid activation function layer; The outputs of the first Gaussian convolution layer and the second Gaussian convolution layer form the input of the edge extraction module; The output of the first Gaussian convolution layer is connected to an input of the third batch normalization layer, the two outputs of the third batch normalization layer are respectively connected to inputs of the first x direction derivative convolution layer and the first y direction derivative convolution layer, the outputs of the first x direction derivative convolution layer and the first y direction derivative convolution layer are respectively connected to two inputs of the fourth pixel addition layer, the output of the fourth pixel addition layer is connected to an input of the first ReLU activation function layer, and the output of the first ReLU activation function layer is connected to one input of the channel splicing layer; The output of the second Gaussian convolution layer is connected to an input of the fourth batch normalization layer, the two outputs of the fourth batch normalization layer are respectively connected to inputs of the second x direction derivative convolution layer and the second y direction derivative convolution layer, the outputs of the second x direction derivative convolution layer and the second y direction derivative convolution layer are respectively connected to two inputs of the fifth pixel addition layer, the output of the fifth pixel addition layer is connected to an input of the second ReLU activation function layer, and the output of the second ReLU activation function layer is connected to another input of the channel splicing layer; The output of the channel splicing layer is connected to an input of the third Gaussian convolution layer, the output of the third Gaussian convolution layer is connected to an input of the second Sigmoid activation function layer, and the output of the second Sigmoid activation function layer forms the output of the edge extraction module.

4. The method according to claim 1, characterized in that, In step 1, the backbone network is a deep residual network ResNet50.

5. The method of claim 1, wherein the method is characterized by, In step 2, the edge extraction module randomly inactivates the input image to obtain an edge heat map, wherein the random inactivation strategy is random inactivation according to uncertainty or random inactivation according to a random inactivation rate.

6. The method according to claim 5, wherein the method is characterized by, In step 2 The process of random inactivation according to uncertainty is: ① generating an empty mask vector, the rows and columns of which correspond to the channels and pixels of the input image respectively; ② traversing each pixel in each channel of the input image, if the pixel value of the current pixel in the current channel is greater than alpha and less than beta, the value at the corresponding position in the mask vector is set to 1; Otherwise, the value at the corresponding position in the mask vector is set to 0, thus the vector obtained is the random inactivation mask; ③ dot product of the random inactivation mask and the input image to obtain the edge heat map; wherein alpha is the lower limit of the set discard threshold, beta is the upper limit of the set discard threshold; The process of random inactivation according to the random inactivation rate is: ① first randomly generate a linear random number sequence with length WxH, whose value range is [0, 1), and then convert it into a mask tensor with width W and height H; ② set the part of the mask tensor whose pixel value is less than or equal to delta to 0, and set the part greater than delta to 1, then expand its dimension to the same dimension as the input image, that is, copy C times in the dimension channel, thus obtain the random inactivation mask; ③ dot product of the random inactivation mask and the input image to obtain the edge heat map; wherein delta is the set random inactivation rate, W, H, C are the length, width and depth of the input image respectively.

Images