An edge detection method based on multi-scale feature fusion

Abstract

The application provides an edge detection method based on multi-scale feature fusion, and belongs to the field of computer vision. The method realizes extraction and fusion of multi-scale features by constructing a convolutional neural network (CNN) backbone network, combining an up-sampling module and a partial convolution module, and optimizing global and local edge information through an improved loss function. The backbone network adopts shallow feature weighted extraction and skip connection technology to improve feature reuse capability, the up-sampling module adjusts feature map size through deconvolution operation, the fusion module strengthens edge detail expression through selective convolution, and the partial convolution efficiently extracts multi-dimensional feature information. The model performs excellently in fine edge detection, supports sketch retrieval, image segmentation and other application scenarios, and has the characteristics of high efficiency and low parameter quantity.

Description

Technical Field

[0001] This study belongs to the field of computer vision technology and proposes an edge detection method based on multi-scale feature fusion. Background Technology

[0002] Edge detection is a core task in computer vision for extracting object boundaries, widely used in image segmentation, sketch retrieval, and other fields. Existing lightweight edge detection algorithms with low parameter counts often ignore the rich edge information in shallow features and rely excessively on deep feature extraction, resulting in poor performance in fine-grained edge detection. To address this, this paper proposes an efficient multi-scale feature fusion method to improve the performance of edge detection models while maintaining low parameter counts and fast iteration. In multi-scale feature fusion, shallow feature extraction is optimized, and a partial convolution module is introduced to improve edge detection accuracy while keeping the model lightweight. Furthermore, this paper employs a novel training convergence loss function, dynamically adjusting the model's global and local iteration processes to achieve high-precision edge information detection. Summary of the Invention

[0003] The purpose of this invention is to provide an efficient multi-scale feature fusion edge detection network, aiming to solve the problems of missing detailed edges and low computational efficiency in existing edge detection methods. This network, by introducing shallow feature weighted extraction and a multi-scale fusion mechanism, achieves accurate prediction of fine edges of multi-dimensional features while maintaining a lightweight model. To achieve the above objective, this invention proposes the following scheme:

[0004] An edge detection method based on multi-scale feature fusion includes the following steps:

[0005] 1. Constructing the backbone network: The model is an end-to-end edge detection network based on a convolutional neural network (CNN). The model design includes multiple convolutional blocks to extract features of different scales in the image layer by layer, and performs weighted processing on shallow features to highlight edge information.

[0006] 2. Constructing an upsampling module: Pre-training the model using common edge detection datasets.

[0007] 3. Constructing a feature fusion module: The model uses partial convolution as the fusion module, and achieves multi-scale feature fusion through selective convolution, effectively extracting edge information at different scales.

[0008] 4. A multi-scale fusion network architecture is constructed based on the backbone network and feature fusion module: The feature fusion module is combined with the edge detection model to achieve efficient multi-scale edge detection. During training, the model employs adaptive weight adjustment and loss function design to improve edge detection performance.

[0009] 5. Achieve fine edge prediction based on integrated network architecture: Accurately predict the edge information of each pixel by using the fused feature map to generate an edge map.

[0010] Preferably, the model comprises the following network structure:

[0011] • Network backbone structure: The backbone network of the model contains five convolutional blocks. The first convolutional block adopts the structure of [convolutional layer 1 + ReLU + convolutional layer 2 + ReLU] to extract shallow features. The remaining convolutional blocks have the structure of [convolutional layer 1 + Smish + convolutional layer 2 + Smish] to enhance edge detail information, while also improving the compactness of the low-parameter model and strengthening iterative capabilities.

[0012] Let the output feature of the first convolutional block be f1. The outputs of subsequent convolutional blocks are fused through skip connections and 1x1 convolutions to obtain f2, f3, and f4, respectively. This feature connection design is used to establish dense connections between network layers, improve the model's feature reuse capability, and solve the gradient vanishing problem in deep networks.

[0013] • Upsampling Module: For the output features f1, f2, f3, and f4 of the backbone network, the upsampling module performs three 2x upsampling operations and one 4x upsampling operation, respectively. The upsampling module is mainly implemented through deconvolution, with the specific formula as follows:

[0014] Y = Concat(Filter(UP(f)) i )))

[0015] Where UP represents the upsampling operation, Filter is a 1x1 convolutional filter, and the feature maps f of different scales are processed by a deconvolutional layer. i Adjust the dimensions to the same size before assembling (Concat).

[0016] • Partial Convolution Fusion Module: This module uses partial convolution (PartialConv) to fuse multi-scale features, reducing redundant information and focusing on important edge details. The specific formula for calculating partial convolution is as follows:

[0017] y fuse =Smish(ewa(Pc1(Y)+Pc2(Y)) Pc1 )))

[0018] In this structure, Pc1 and Pc2 are two partially convolutional layers, Y is the feature map output by the upsampling module, ewa is the element-wise addition operation, and Sminsh is the activation function.

[0019] Preferably, the model uses an adaptive loss function, which includes a tracking loss (L... trc ) and Dice loss (L dice The combination of these technologies can optimize edge detection at both the global and local levels. dice The calculation formula is:

[0020]

[0021] Among them, Y and These represent the edge regions of the true labels and the predicted results, respectively. Tracking loss L trc The definition of is:

[0022] L trc =L ce +λ1L bd +λ2L tex

[0023]

[0024] Among them, L trc L tex For texture loss, L bd Used for edge tracking, λ1 and λ2 are weight parameters. L tex The 'l' below represents different texture layers, corresponding to the edges of different layers. and L represents the Gram matrix at layer l, used to represent the texture features of image x and y. bd Down Indicates the boundary of the target area. Represents the prediction boundary The distance function between the object and the true boundary y. ∫d S This means integrating the distance error over the entire boundary to calculate the overall difference between the predicted boundary and the true boundary.

[0025] This invention effectively improves the accuracy and computational efficiency of edge detection by combining multi-scale feature extraction and fusion strategies. By utilizing a pre-trained model to guide and fuse edge features, the computational burden during training is reduced, resulting in superior model performance in practical applications (such as sketch retrieval and image enhancement). This invention successfully solves the problem of edge detection being disconnected from downstream tasks in existing technologies, providing an efficient and practical solution. Attached Figure Description

[0026] Figure 1 This is a flowchart of an edge detection method based on multi-scale feature fusion.

[0027] Figure 2 The diagram below illustrates the model framework of this invention, showing the structure and modules of the designed multi-scale feature fusion network, including convolutional blocks, upsampling modules, and fusion modules.

[0028] Figure 3 This is a schematic diagram of partial convolution used in the fusion module of the present invention, showing the structure of partial convolution and its comparison with conventional convolution. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0030] See Figure 1 and Figure 2 This invention proposes an edge detection method based on multi-scale feature fusion, which specifically includes the following steps:

[0031] Step 1: Constructing the backbone network

[0032] The backbone network consists of five convolutional blocks, which are responsible for multi-scale feature extraction from the input image.

[0033] • Convolutional block 0 has the structure [convolutional layer 1 + ReLU + convolutional layer 2 + ReLU], and is used to extract shallow feature information. The formula for calculating the ReLU activation function is as follows:

[0034] f(x) = max(0,x)

[0035] The structure of convolutional blocks 1 through 4 is [Convolutional Layer 1 + Smish + Convolutional Layer 2 + Smish], progressively extracting features from deeper layers. The outputs of each convolutional block are fused through skip connections and summation operations to form a dense feature flow that preserves edge information at different levels. The formula for calculating the Smish activation function is as follows:

[0036] f(x)=x·tanh(ln(1+sigmoid(x)))

[0037]

[0038] Through this layer-by-layer connection and fusion approach, the backbone network can extract features from different levels, enabling multi-scale feature fusion in subsequent stages.

[0039] Step 2: Construct the upsampling module

[0040] The upsampling module is responsible for scaling up feature maps from different levels to the same spatial size, providing multi-scale edge information for the fusion module.

[0041] • For the feature maps f1, f2, f3 and f4 generated by the backbone network, the upsampling module performs three 2x upsampling operations and one 4x upsampling operation in sequence.

[0042] Each upsampling step involves a 1x1 convolution and deconvolution operation to ensure that the resolution of the feature map is progressively increased. For example,

[0043] For f1, f2, and f3, perform upsampling by a factor of 2, while for f4, perform upsampling by a factor of 4.

[0044] The specific upsampling operation includes:

[0045] upsampling x2: 1x1 convolution + 2x2 deconvolution

[0046] Upsampling x4: 1x1 convolution + 2x2 deconvolution + 1x1 convolution + 4x4 deconvolution

[0047] Step 3: Construct the feature fusion module

[0048] The feature fusion module is used to fuse upsampled feature maps from different scales in order to extract fine edge information. The specific implementation steps are as follows:

[0049] 1. Multi-scale feature map stitching

[0050] Feature maps generated at different upsampling levels are concatenated to form a comprehensive feature map containing multi-scale information. This concatenation process preserves edge information at each scale, ensuring effective fusion of shallow and deep features.

[0051] 2. Partial Convolution for Feature Extraction and Fusion

[0052] During feature fusion, two partial convolutional layers (PartialConv) are used to selectively convolve the stitched multi-scale feature maps. This partial convolution method combines the advantages of standard convolution and point convolution, enabling convolution operations on key regions of the multi-scale feature maps, preserving important edge information and removing redundant information.

[0053] o Partial convolution operation process:

[0054] Partial convolution combines standard convolutional layers and point-like convolutional layers. Standard convolutional layers (size W × ...

[0055] W×H is used to capture edge information of local regions; point convolution (size 1×1×L) is used to further fuse the feature maps along the channel dimension.

[0056] Partial convolution operations first process the input feature map using T-shaped convolution (i.e., a combination of standard convolution and point convolution) to generate a fused feature map with multi-scale information. The size of this feature map is W×W×H.

[0057] 3. Application of Smish activation function

[0058] In the final output of the fusion module, the Smish activation function is used to perform non-linear activation on the fused feature map. The Smish activation function effectively preserves the detailed information of edge features while preventing gradient vanishing, ensuring the accuracy of edge prediction. Its formula is as follows:

[0059] f(x)=x·tanh(ln(1+sigmoid(x)))

[0060]

[0061] Step 4: Constructing a Multi-Scale Fusion Network Architecture Based on the Backbone Network and Feature Fusion Module. In this architecture, the backbone network is responsible for extracting multi-level edge features, while the feature fusion module further processes these features to obtain accurate edge information. The backbone network consists of five convolutional blocks with a layer-by-layer skip connection structure. Convolutional block 0 uses the ReLU activation function, while the remaining blocks use the Smish activation function, extracting deep edge features layer by layer and preserving the information flow at different scales through feature fusion. To ensure accurate edge detection, this paper proposes an improved loss function that combines cross-entropy loss, Dice loss, and edge tracking loss.

[0062] Specifically, the loss function includes the following parts:

[0063] 1. Cross-entropy loss (L ce (): Used for pixel-level edge detection, improving the model's sensitivity to edges. Its formula is as follows:

[0064] L ce =-sum(p(x) i )*log(q(x i )))

[0065] Where p(x) i ) is the target distribution, q(x) i The predicted distribution is denoted as Σ(cross-entropy loss function). The smaller the value of the cross-entropy loss function, the closer the predicted distribution is to the target distribution.

[0066] 2. Edge tracking loss (L trc This method combines boundary tracking and texture suppression to ensure a significant difference between edges and adjacent background, thereby reducing the impact of noise. The formula is as follows:

[0067] L trc =L ce +λ1L bd +λ2L tex

[0068] Among them, L trc L tex For texture loss, L bd The boundary loss is used for edge tracking, and λ1 and λ2 are weight parameters.

[0069] 3. Dice loss (L) dice (): By calculating the similarity between predicted and true edges using the Dice coefficient, false positives and false negatives in prediction tasks are effectively reduced. Using L... dice The prediction results have higher accuracy for thin boundaries because L can be reduced only when the predicted boundary pixels overlap with the labeled thin boundary and there are no predicted boundary pixels in other areas. dice The formula is as follows:

[0070]

[0071] The overall loss function formula is:

[0072]

[0073] Among them, y i Representing edge features at different levels, there are a total of 4 levels, y fuse This is the fused edge image. The loss function balances local and global losses, enabling the model to accurately capture true edges.

[0074] Step 5: Output edge detection results

[0075] The model outputs edge detection results through a multi-scale fusion network architecture trained on the network. In practical applications, this model can achieve accurate edge prediction on images of different resolutions and supports further image processing tasks, such as sketch recognition and super-resolution reconstruction.

[0076] Furthermore, ablation experiments were conducted on the embodiments of the present invention. When testing the performance of a module, the basic modules and network structures used in different experimental projects within the same group were always kept identical, and different experimental projects were ensured to have complete edge detection functions to stabilize non-experimental variables. The performance of the backbone structure, fusion module, and loss function of the present invention were tested respectively. The ablation experiments were performed on the UDED dataset.

[0077] The evaluation metrics used are OIS (Optimal Image Scale) and ODS (Optimal Dataset Scale), commonly used in edge detection. OIS represents the precision and recall at the optimal detection threshold found on each image. For each image, an optimal threshold is chosen that makes the edge detection result as close as possible to the ground truth. Then, the OIS of the entire dataset is averaged to evaluate the model's best edge detection performance on each image. ODS is a single optimal threshold found across the entire dataset. This threshold is global and applicable to all images. Precision and recall are calculated at this threshold to evaluate the model's overall edge detection performance across the entire dataset. ODS is more focused on measuring the model's stability across the dataset.

[0078] Table 1

[0079]

[0080] The ablation experiment data in Table 1 show that this embodiment demonstrates that the performance improvement is gradual as our method is incorporated.

[0081] Table 2

[0082]

[0083] This embodiment was trained on the public dataset BIPED and compared with other advanced edge detection methods on UDED. The comparison results are shown in Table 2. While maintaining traditional performance metrics comparable to existing advanced methods, this embodiment significantly outperforms other methods in edge detection accuracy and efficiency, and its performance in fine edge detection tasks is close to the ideal detection effect of the original image.

[0084] It is evident that this invention can significantly improve the accuracy of edge detection results in subsequent model tasks, while maintaining excellent edge fidelity and information retention. Through innovative feature extraction and fusion module design, this model achieves accurate prediction of fine edges while ensuring high efficiency, providing reliable edge information support for various image processing tasks.

Claims

1. An edge detection method based on multi-scale feature fusion, the method being applicable to computer vision fields such as sketch retrieval, image segmentation, and image enhancement, characterized in that... Includes the following steps: o Constructing the backbone network: An edge detection backbone network based on a convolutional neural network (CNN) is established, containing 5 convolutional blocks to extract multi-scale features from the input image layer by layer. The first convolutional block uses a structure of [convolutional layer 1 + ReLU + convolutional layer 2 + ReLU] to extract shallow features; the remaining convolutional blocks use a structure of [convolutional layer 1 + Smish + convolutional layer 2 + Smish] to enhance edge detail information. The outputs of each convolutional block are fused through skip connections and 1×1 convolutions to obtain the feature map. , , and Let the output features of the first convolutional block be... The outputs of subsequent convolutional blocks are fused through skip connections and 1x1 convolutions to obtain... , and ; o Constructing the upsampling module: Processing the feature maps generated by the backbone network , , and Upsampling was performed separately, among which... , , Perform three 2x upsampling operations. Perform a 4x upsampling to ensure consistent spatial dimensions; the upsampling operation is implemented through deconvolution, specifically: Here, UP represents the upsampling operation, and Filter is a 1x1 convolutional filter that uses a deconvolutional layer to process feature maps of different scales. Adjust to the same spatial dimensions before splicing. ), where i represents the index of the feature map at different scales, and the value ranges from 1 to 4. o Constructing a feature fusion module: Using partial convolution on the upsampled feature map Multi-scale feature fusion is performed, and the specific calculation formula is as follows: In this structure, Pc1 and Pc2 are two partially convolutional layers, Y is the feature map output by the upsampling module, ewa is the element-wise addition operation, and Smish is the activation function. o Construct a multi-scale fusion network architecture: Combine the backbone network, upsampling module and feature fusion module to form a multi-scale fusion network architecture; during model training, use an adaptive loss function to optimize global and local edge information. o Achieving fine-grained edge prediction: Utilizing the fused feature map It accurately predicts the edge information of each pixel to generate an edge map.

2. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that, The partial convolution operations in the partial convolution fusion module include: o Uses a combination of standard convolution and point convolution to concatenate the feature maps Perform selective convolution to preserve important edge information; o Partial convolutional layer and The output is obtained through element-wise addition and The activation function undergoes a nonlinear transformation to obtain the fused feature map. .

3. The edge detection method based on multi-scale feature fusion according to claim 1, characterized in that, The adaptive loss function includes cross-entropy loss. Tracking losses and loss The combination of , the specific formula is: in, For edge features at different levels, For real labels, This is the fused feature map output by the feature fusion module.

Images