Edge detection method based on multi-scale feature fusion

Abstract

The invention provides an edge detection method based on multi-scale feature fusion, and belongs to the field of computer vision. According to the method, a convolutional neural network (CNN) backbone network is constructed, an up-sampling module and a partial convolution module are combined, extraction and fusion of multi-scale features are realized, and global and local edge information is optimized through an improved loss function. The backbone network adopts a shallow feature weighted extraction and jump connection technology to improve the feature multiplexing capability, the up-sampling module adjusts the size of a feature map by using a deconvolution operation, and the fusion module strengthens edge detail expression through selective convolution and efficiently extracts multi-dimensional feature information through partial convolution. The model is excellent in performance in fine edge detection, supports application scenes such as sketch retrieval and image segmentation, and has the characteristics of high efficiency and low parameter quantity.

Description

Technical Field

[0001] This study belongs to the field of computer vision technology and proposes an edge detection method based on multi-scale feature fusion. Background Art

[0002] Edge detection is a core task in computer vision for extracting object boundaries and is widely used in image segmentation, sketch retrieval and other fields. Existing low-parameter lightweight edge detection algorithms usually ignore the rich edge information in shallow features and over-rely on deep feature extraction, resulting in poor performance of the model in fine edge detection. To this end, this paper proposes an efficient multi-scale feature fusion method to improve the performance of the edge detection model while having the characteristics of low parameter volume and fast iteration. In the multi-scale feature fusion, shallow feature extraction is optimized and a partial convolution module is introduced to improve the edge detection accuracy while keeping the model lightweight. In addition, this paper adopts a new training convergence loss function to achieve high-precision edge information detection by dynamically adjusting the global and local iteration process of the model. Summary of the invention

[0003] The purpose of the present invention is to provide an efficient multi-scale feature fusion edge detection network, aiming to solve the problems of missing detailed edges and low computational efficiency in existing edge detection methods. This network introduces shallow feature weighted extraction and multi-scale fusion mechanism to accurately predict the fine edges of multi-dimensional features while keeping the model lightweight. To achieve the above purpose, the present invention proposes the following scheme:

[0004] An edge detection method based on multi-scale feature fusion includes the following steps:

[0005] 1. Constructing the backbone network: The model is an end-to-end edge detection network based on a convolutional neural network (CNN). The model design contains multiple convolution blocks to extract features of different scales in the image layer by layer, and perform weighted processing on shallow features to highlight edge information.

[0006] 2. Build upsampling module: Pre-train the model using common edge detection datasets.

[0007] 3. Construct feature fusion module: The model uses partial convolution as the fusion module, realizes multi-scale feature fusion through selective convolution, and effectively extracts edge information at different scales.

[0008] 4. Based on the backbone network and feature fusion module, a multi-scale fusion network architecture is constructed: the feature fusion module is combined with the edge detection model to achieve efficient multi-scale edge detection. The model uses adaptive weight adjustment and loss function design during the training process to improve the effect of edge detection.

[0009] 5. Based on the integrated network architecture, fine edge prediction is achieved: the edge information of each pixel is accurately predicted through the fused feature map to generate an edge map.

[0010] Preferably, the model comprises the following network structure:

[0011] Network backbone structure: The backbone network of the model contains five convolutional blocks. The first convolutional block adopts the structure of [convolutional layer 1 + ReLU + convolutional layer 2 + ReLU] to extract shallow features. The remaining convolutional blocks are structured as [convolutional layer 1 + Smish + convolutional layer 2 + Smish] to enhance edge detail information, while enhancing the compactness of the low-parameter model and strengthening the iteration capability.

[0012] o Let the output feature of the first convolutional block be f1, and the outputs of subsequent convolutional blocks are fused through skip connections and 1x1 convolutions to obtain f2, f3, and f4 respectively. This feature connection design is used to establish dense connections between network layers, improve the feature reuse capability of the model, and solve the gradient disappearance problem in deep networks.

[0013] Upsampling module: For the output features f1, f2, f3 and f4 of the backbone network, the upsampling module performs three times of 2-fold upsampling and one time of 4-fold upsampling. The upsampling module is mainly implemented through deconvolution, and the specific formula is as follows:

[0014] Y=Concat(Filter(UP(f i )))

[0015] Among them, UP represents the upsampling operation, Filter is a 1x1 convolution filter, and the feature maps of different scales f are transformed into i Adjust to the same spatial size before concatenating (Concat).

[0016] Partial convolution fusion module: This module uses partial convolution (PartialConv) to fuse multi-scale features to reduce redundant information and focus on important edge details. The specific partial convolution calculation formula is as follows:

[0017] y fuse =Smish(ewa(Pc1(Y)+Pc2(Y Pc1 )))

[0018] Among them, Pc1 and Pc2 are two partial convolution layers, Y is the feature map output by the upsampling module, ewa is the element-by-element addition operation, and Sminsh is the activation function.

[0019] Preferably, the model uses an adaptive loss function, which includes a tracking loss (L trc ) and Dice loss (L dice ) can optimize the detection of edge information both globally and locally. dice The calculation formula is:

[0020]

[0021] Among them, Y and They are the marginal areas of the true label and the predicted result respectively. Tracking loss L trc is defined as:

[0022] L trc =L ce +λ1L bd +λ2L tex

[0023]

[0024] Among them, L trc Lower L tex is the texture loss, L bd Used for edge tracking, λ1 and λ2 are weight parameters. L tex The following l represents different texture layers, corresponding to edges of different levels. and They represent the Gram matrix at the lth layer, which is used to represent the texture features of images x and y. bd Down represents the boundary of the target area, Represents the prediction boundary The distance function from the true boundary y. ∫d S It means integrating the distance error of the entire boundary and calculating the overall difference between the predicted boundary and the true boundary.

[0025] The present invention effectively improves the accuracy and computational efficiency of edge detection by combining multi-scale feature extraction and fusion strategies. By using pre-trained models to guide and fuse edge features, the computational burden in the training process is reduced, and the model performs better in practical applications (such as sketch retrieval, image enhancement, etc.). The present invention successfully solves the problem of edge detection being out of touch with downstream tasks in the prior art, and provides an efficient and practical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The figure is an overall flow chart of an edge detection method based on multi-scale feature fusion.

[0027] Figure 2 This is a model framework diagram of the present invention, showing the structure and modules of the designed multi-scale feature fusion network, including convolution blocks, upsampling modules and fusion modules.

[0028] Figure 3 Schematic diagram of partial convolution used in the fusion module of the present invention, showing the structure of partial convolution and comparison with conventional convolution. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with specific examples and with reference to the accompanying drawings.

[0030] See also Figure 1 and Figure 2 The present invention proposes an edge detection method based on multi-scale feature fusion, which specifically comprises the following steps:

[0031] Step 1: Build a backbone network

[0032] The backbone network consists of five convolutional blocks, which are responsible for multi-scale feature extraction of the input image.

[0033] The structure of convolution block 0 is [convolution layer 1 + ReLU + convolution layer 2 + ReLU], which is used to extract shallow feature information. The calculation formula of ReLU activation function is as follows:

[0034] f(x)=max(0,x)

[0035] The structure of convolutional blocks 1 to 4 is [convolutional layer 1 + Smish + convolutional layer 2 + Smish], which gradually extracts deeper features. The output of each convolutional block is feature fused through skip connections and addition operations to form a dense feature flow to retain edge information at different levels. The calculation formula of the Smish activation function is as follows:

[0036] f(x)=x·tanh(ln(1+sigmoid(x)))

[0037]

[0038] Through this layer-by-layer connection and fusion approach, the backbone network is able to extract features at different levels so that multi-scale feature fusion can be performed in subsequent stages.

[0039] Step 2: Build upsampling module

[0040] The upsampling module is responsible for enlarging the feature maps of different levels to the same spatial size and providing multi-scale edge information for the fusion module.

[0041] For the feature maps f1, f2, f3, and f4 generated by the backbone network, the upsampling module performs three times of 2x upsampling and one time of 4x upsampling in sequence.

[0042] Each upsampling includes 1x1 convolution and deconvolution operations to ensure that the resolution of the feature map is gradually improved. For example,

[0043] f1, f2, and f3 are upsampled by a factor of 2, while f4 is upsampled by a factor of 4.

[0044] Specific upsampling operations include:

[0045] o Upsampling x2: 1x1 convolution + 2x2 deconvolution

[0046] o Upsampling x4: 1x1 convolution + 2x2 deconvolution + 1x1 convolution + 4x4 deconvolution

[0047] Step 3: Build feature fusion module

[0048] The feature fusion module is used to fuse upsampled feature maps from different scales to extract fine edge information. The specific implementation steps are as follows:

[0049] 1. Multi-scale feature map concatenation

[0050] The feature maps generated by different upsampling levels are concatenated to form a comprehensive feature map containing multi-scale information. This concatenation process retains the edge information of each scale and ensures the effective fusion of shallow and deep features.

[0051] 2. Partial Convolution for Feature Extraction and Fusion

[0052] In the feature fusion process, two partial convolution layers (PartialConv) are used to selectively convolve the spliced ​​multi-scale feature maps. This partial convolution method combines the advantages of standard convolution and point convolution, and can perform convolution operations on key areas of multi-scale feature maps, retaining important edge information and removing redundant information.

[0053] oPart of the convolution operation process:

[0054] Partial convolution consists of a combination of a standard convolution layer and a point-wise convolution layer.

[0055] W×H is used to capture the edge information of the local area; the point convolution (size is 1×1×L) is used for further fusion in the channel dimension of the feature map.

[0056] The partial convolution operation first processes the input feature map through a T-shaped convolution (i.e., a combination of standard convolution and point convolution) to generate a fused feature map with multi-scale information. The size of this feature map is W×W×H.

[0057] 3. Application of Smish activation function

[0058] In the final output of the fusion module, the Smish activation function is used to perform nonlinear activation on the fused feature map. The Smish activation function can effectively retain the detailed information of edge features while preventing gradient disappearance and ensuring the accuracy of edge prediction. Its formula is as follows:

[0059] f(x)=x·tanh(ln(1+sigmoid(x)))

[0060]

[0061] Step 4: Construct a multi-scale fusion network architecture based on the backbone network and feature fusion module In the multi-scale fusion network architecture, the backbone network is responsible for extracting multi-level edge features, while the feature fusion module further processes these features to obtain accurate edge information. The backbone network consists of five convolutional blocks with a layer-by-layer skip connection structure. Convolutional block 0 uses the ReLU activation function, and the remaining convolutional blocks use the Smish activation function to extract deep edge features layer by layer, and retain the information flow of different scales through feature fusion. In order to ensure accurate edge detection, this paper proposes an improved loss function that combines cross entropy loss, Dice loss, and edge tracking loss.

[0062] Specifically, the loss function consists of the following parts:

[0063] 1. Cross Entropy Loss (L ce ): Used for edge detection at the pixel level to improve the model's sensitivity to edges. The formula is as follows:

[0064] L ce =-sum(p(x i )*log(q(x i )))

[0065] Among them, p(x i ) is the target distribution, q(x i ) is the predicted distribution. The smaller the value of the cross entropy loss function, the closer the predicted distribution is to the target distribution.

[0066] 2. Edge tracking loss (L trc ): Fusion of boundary tracking and texture suppression ensures significant differences between edges and adjacent backgrounds, thereby reducing the impact of noise. The formula is as follows:

[0067] L trc =L ce +λ1L bd +λ2L tex

[0068] Among them, L trc Lower L tex is the texture loss, L bd is the boundary loss used for edge tracking, and λ1 and λ2 are weight parameters.

[0069] 3. Dice loss (L dice ): The Dice coefficient is used to calculate the similarity between edge prediction and true edge, effectively reducing the false positive and false negative results of the prediction task. dice The prediction result of has higher accuracy for thin boundaries, because L can be reduced only when the predicted boundary pixels overlap with the annotated thin boundaries and there are no predicted boundary pixels in other areas. dice The formula is as follows:

[0070]

[0071] The overall loss function formula is:

[0072]

[0073] Among them, y i Represents edge features at different levels, with a total of 4 levels, y fuse is the fused edge image. This loss function weighs the local and global losses, enabling the model to accurately capture the real edge.

[0074] Step 5: Output edge detection results

[0075] Through the trained multi-scale fusion network architecture, the edge detection result is finally output. In practical applications, the model can achieve accurate edge prediction on images of different resolutions and support further image processing tasks such as sketch recognition and super-resolution reconstruction.

[0076] Furthermore, an ablation experiment is conducted on the embodiment of the present invention. When testing the performance of a module, it is always ensured that the basic modules and network structures used by different experimental projects in the same group of experiments are the same, and that different experimental projects have complete edge detection functions to stabilize non-experimental variables. The experiment tests the performance of the backbone structure, fusion module and loss function of the present invention respectively, and the ablation experiment is completed on the UDED dataset.

[0077] The evaluation criteria used are OIS (Optimal Image Scale) and ODS (Optimal Dataset Scale), which are commonly used evaluation criteria in edge detection. OIS represents the precision and recall rate under the best detection threshold found on each image. For each image, an optimal threshold is selected so that the edge detection result of the image is as close to the ground truth as possible. Then, the OIS of the entire dataset is averaged to evaluate the best edge detection effect found by the model on each image; ODS is the single best threshold found on the entire dataset. This threshold is global and applicable to all images. The precision and recall rate are calculated under this threshold to evaluate the overall edge detection performance of the model on the entire dataset. ODS is more inclined to measure the stability of the model's performance on the dataset.

[0078] Table 1

[0079]

[0080] The ablation experiment data in Table 1 shows that the improvement of this embodiment is gradually improved with the addition of our method.

[0081] Table 2

[0082]

[0083] This embodiment and other advanced edge detection methods are trained on the public dataset BIPED and compared on UDED, and the comparison results are shown in Table 2. While keeping the traditional performance indicators comparable to those of existing advanced methods, this embodiment is significantly superior to other methods in edge detection accuracy and detection efficiency, and its performance in fine edge detection tasks is close to the ideal detection effect of the original image.

[0084] It can be seen that the present invention can significantly improve the accuracy of edge detection results in subsequent model tasks while maintaining excellent edge fidelity and information retention. Through innovative feature extraction and fusion module design, this model achieves accurate prediction of fine edges while ensuring high efficiency, providing reliable edge information support for various image processing tasks.

Claims

1. An edge detection method based on multi-scale feature fusion, characterized in that: The following steps are involved: οBuild a backbone network: Establish an edge detection backbone network based on a convolutional neural network (CNN), which includes multiple convolution blocks to extract multi-scale features of the input image layer by layer; among them, the first convolution block adopts the structure of convolution layer 1 + ReLU activation function + convolution layer 2 + ReLU activation function convolution layer 1 + ReLU activation function + convolution layer 2 + ReLU activation function convolution layer 1 + ReLU activation function + convolution layer 2 + ReLU activation function to extract shallow features, and the remaining convolution blocks adopt the structure of convolution layer 1 + Smish activation function + convolution layer 2 + Smish activation function convolution layer 1 + Smish activation function + convolution layer 2 + Smish activation function convolution layer 1 + Smish activation function + convolution layer 2 + Smish activation function to extract deep features; the outputs of each convolution block are fused through skip connection and 1×1 convolution to obtain feature maps f1, f2, f3 and f4. οBuild upsampling module: Upsample the feature maps f1, f2, f3 and f4 generated by the backbone network respectively, where f1, f2 and f3 are upsampled three times by a factor of 2, and f4 is upsampled once by a factor of 4 to make their spatial sizes consistent; the upsampling operation is implemented by deconvolution, specifically: Y=Concat(Filter(UP(f i ))) Among them, UP represents the upsampling operation, Filter is a 1x1 convolution filter, and the feature maps f_i of different scales are adjusted to the same spatial size through the deconvolution layer and then concatenated (Concat). οConstruct feature fusion module: Use partial convolution to perform multi-scale feature fusion on the upsampled feature map Y. The specific calculation formula is: and fuse =Smish(ewa(Pc1(Y)+Pc2(Y Pc1 ))) Among them, Pc1 and Pc2 are two partial convolutional layers, Y is the feature map output by the upsampling module, ewa is the element-by-element addition operation, and Smish is the activation function. οBuild a multi-scale fusion network architecture: combine the backbone network, upsampling module and feature fusion module to form a multi-scale fusion network architecture; during the model training process, use an adaptive loss function to optimize the global and local edge information. οAchieve fine edge prediction: Use the fused feature map y fuse , accurately predict the edge information of each pixel and generate an edge map.

2. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that: The structure of the first convolutional block is: Convolutional layer 1 → ReLU → Convolutional layer 2 → ReLU Among them, ReLU(x)=max(0,x).

3. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that: The structures of the convolution blocks 1 to 4 are as follows: Convolutional layer 1 → Smish → Convolutional layer 2 → Smish Among them, the calculation formula of the Smish activation function is: f(x)=x·tanh(ln(1+sigmoid(x))) 4. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that: The partial convolution operation in the partial convolution fusion module includes: οUse a combination of standard convolution and point convolution to selectively convolve the concatenated feature map Y to retain important edge information; The outputs of some convolutional layers Pc1 and Pc2 are linearly transformed by element-by-element addition and Smish activation function to obtain the fused feature map y fuse .

5. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that: The adaptive loss function includes the cross entropy loss L ce , tracking loss L trc and Dice loss L dice The specific formula is: Among them, y i are edge features at different levels, and y is the true label.

6. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that: The method is suitable for applications in computer vision such as sketch retrieval, image segmentation and image enhancement.

7. An edge detection device based on multi-scale feature fusion, comprising: οBackbone network module: used to establish a convolutional neural network structure and extract multi-scale features; οUpsampling module: used to upsample the feature map and adjust the spatial size; οFeature fusion module: used to fuse the upsampled multi-scale feature maps; οPrediction module: used to generate edge prediction results based on the fused feature map; οLoss calculation module: used to calculate the adaptive loss function and optimize model training.

8. The device according to claim 7, characterized in that The structures and functions of the backbone network module, upsampling module and feature fusion module are as described in any one of claims 1 to 10.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the edge detection method based on multi-scale feature fusion according to any one of claims 1 to 8 is implemented.

Images