Edge detection network method based on HED and Transform fusion
By combining HED and Transformer models, using convolutional encoder and multi-head self-attention mechanism, the limitations of traditional edge detection in complex contexts are solved, and more accurate edge detection is achieved.
Patent Information
- Application Number
- CN202510549215.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-22
AI Technical Summary
Traditional edge detection methods are difficult to deal with fuzzy boundaries and long-distance objects in complex backgrounds, and are sensitive to noise and light changes, making it difficult to detect weak edges.
Combining the HED and Transformer models, complementary optimization of local details and global features is achieved through convolutional encoder, HED weighted fusion module, multi-head self-attention decoder, Transformer weighted fusion module and total weighted fusion module.
It significantly improves the accuracy and detail retention ability of edge detection, especially in complex scenarios, the edges are more refined, the position is accurate, and the information is complete.
Smart Images
Figure CN120526166A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and image processing technology, and in particular to an edge detection network method based on the fusion of HED and Transformer. Background Art
[0002] Edge detection has long been a classic task in computer vision. Traditional methods rely primarily on local information for edge detection, making them difficult to detect for issues such as blurred boundaries in complex backgrounds, distant objects, and color variations within objects. These methods are also sensitive to noise and lighting variations, making it difficult to detect subtle edges.
[0003] In recent years, the rapid development of deep learning methods has significantly transformed traditional edge detection techniques. Through automated feature learning, improved robustness, multi-scale fusion, and end-to-end training, new edge detection methods can provide more accurate and stable results in complex real-world scenarios, further promoting the advancement of computer vision technology.
[0004] The Holistically Nested Edge Detection (HED) algorithm is a deep learning-based edge detection algorithm proposed by SNXie et al. in 2015 based on the VGG-16 network architecture. It achieves image-to-image prediction through a fully convolutional neural network (FCN) and a deep supervised network. HED's design of convolutional and pooling layers enables it to accurately capture image details during edge detection. Furthermore, the edge probability map generated by multi-scale and multi-level feature fusion can more comprehensively capture edge information in the image.
[0005] Transformer is a deep learning model architecture based on the self-attention mechanism. This model architecture was first proposed by Ashish Vaswani et al. in 2017 and is commonly used in the field of Natural Language Processing (NLP). In 2022, MYPu et al. proposed the EDTER (Edge Detection Transformer for Edge Recognition) model based on the Transformer architecture, which used the Transformer for the first time in edge detection tasks. The Transformer uses a multi-head self-attention mechanism to extract global contextual information from the image, thereby achieving accurate edge recognition. With its global modeling capabilities, the Transformer performs outstandingly in edge detection tasks in complex scenes, especially in the recognition of large-scale edges and complex texture structures, significantly outperforming traditional methods.
[0006] Combining CNN and Transformer can solve the problem of CNN's ability to model long-distance correlations, and at the same time solve the problem that Transformer requires massive amounts of data to learn visual inductive biases. Using CNN as the feature extraction part means that Transformer does not need pre-training, while being able to perform edge detection tasks excellently. Summary of the Invention
[0007] The purpose of the present invention is to provide an edge detection network method based on the fusion of HED and Transformer. In scenarios where complex backgrounds and fuzzy boundaries coexist, this method can effectively improve the processing of details and the overall control, and effectively improve the accuracy of edge detection.
[0008] To achieve the above object, the technical solutions adopted by the present invention are as follows:
[0009] An edge detection network method based on the fusion of HED and Transformer is proposed. The method utilizes a convolution-based encoder, a HED weighted fusion module, a decoder based on a multi-head self-attention mechanism, a Transformer weighted fusion module, and a total weighted fusion module. The method specifically comprises the following steps:
[0010] S1, using a convolution-based encoder, performs feature extraction and information fusion on the input image through multiple convolutional layers, and outputs feature maps C1, C2, C3, C4, and C5;
[0011] S2, input the feature maps C1, C2, C3, C4, and C5 into the HED weighted fusion module, and the feature map after Concat weighted output is FHED ;
[0012] S3, input the feature maps C1, C2, C3, C4, and C5 into the decoder based on the multi-head self-attention mechanism, and obtain the feature maps T1, T2, T3, and T4 after processing by the decoder;
[0013] S4, input the feature maps T1, T2, T3, and T4 into the Transformer weighted fusion module, and the feature map after Concat weighted output is F transformer ;
[0014] S5, the feature map F HED and feature map F transformer After processing by the total weighted fusion module, the final weighted fusion feature map F is obtained End .
[0015] Furthermore, the convolution-based encoder inputs the training set into a convolutional neural network (CNN), outputs 5 layers of feature maps, and then labels each layer of feature maps output by the CNN through residual connections as feature maps C1, C2, C3, C4, and C5 in order;
[0016] Furthermore, the HED weighted fusion module has a Conv2d and sigmoid function with kernel_size=1 and stride=1, which is used to compress the image and generate an edge probability map mapped between 0 and 1; the specific processing process is: the feature maps C1, C2, C3, C4, and C5 are first passed through Conv2d, the output size remains unchanged, and the number of channels is 1. Then, bilinear interpolation upsampling operation is performed through interpolate to change the spatial size of the image (changed to the same size as the input image), and then weighted fusion is performed through Concat to compress and obtain the feature map F HED , the specific processing is shown in formula (1);
[0017]
[0018] In formula (1): w m Represents the feature map C m The fusion weight size is updated through training; M=5.
[0019] Furthermore, the decoder based on the multi-head self-attention mechanism adds relative position coding to the multi-head self-attention mechanism of the decoder, becoming an efficient self-attention decoder with relative position coding; the specific processing method is: by inputting the position information of the image unit (token), the corresponding position coding (relative position coding) is obtained, the corresponding position coding matrix in two dimensions is defined respectively, and finally the multi-head is modified, wherein the calculation of the multi-head self-attention mechanism is shown in formula (2);
[0020]
[0021] In formula (2), Q, are the matrix representations of query, key, and value, respectively, d is the vector dimension of the key, and Represents the relative position encoding matrix based on height and width respectively; Attention(Q, ) represents the multi-head self-attention mechanism.
[0022] Furthermore, the Transformer weighted fusion module has a Conv2d with kernel_size=1 and stride=1 and a sigmoid function, which is used to compress the image and generate an edge probability map mapped between 0 and 1; the specific processing process is: the feature maps T1, T2, T3, and T4 first pass through Conv2d with kernel_size=1, and the output size remains unchanged and the number of channels is 1. Then, a bilinear interpolation upsampling operation is performed through interpolate to change the spatial size of the image (to the same size as the input image), and then the Concat weighted fusion is performed to compress and obtain the feature map F transformer , the specific processing is shown in formula (3);
[0023]
[0024] In formula (3): w n Represents the feature map T n The weight size is integrated and updated through training; N=4.
[0025] Furthermore, the total weighted fusion module obtains the F HED and F obtained after processing by the Transformer weighted fusion module transformer Through the fusion process of formula (4), the edge detection map F that takes into account both local and global processing is finally obtained. End ;
[0026] F End =wFHED +F transformer (4)
[0027] In formula (4), w is a hyperparameter. The optimal value of w is found through ablation experiments, and the value range of w is 0-1.
[0028] The advantages and beneficial effects of the present invention are as follows:
[0029] 1. The method of the present invention is an edge detection network method that integrates Holistically Nested Edge Detection (HED) and Transformer with an improved network based on the HED algorithm. This method fully utilizes the advantages of the HED network in extracting image details, while introducing the multi-head self-attention mechanism of the Transformer and implementing the modeling of long-range dependencies through relative position encoding. As a result, the edge detection results generated by weighted fusion achieve complementary optimization of local details and global features, making the model perform well in edge detection tasks in complex scenes.
[0030] 2. Experimental data demonstrates that this method significantly improves edge detection performance compared to traditional HED networks. In tests on the BSDS500 dataset, the Field of Object (FOM) metric improved by nearly 10%. Visually, even in complex scenes with blurred edges, edges are refined, their positions remain unchanged, and edge information is more complete, preserving more detail. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 Schematic diagram of the edge detection network method based on HED and Transformer of the present invention.
[0032] Figure 2 It is a decoder based on a multi-head self-attention mechanism with position encoding of the present invention.
[0033] Figure 3 It is a schematic diagram of the HED weighted fusion module of the present invention.
[0034] Figure 4 It is a schematic diagram of the Transformer weighted fusion module of the present invention.
[0035] Figure 5 This is a comparison chart of the edge detection effects of the present invention and HED; there are two groups of figures, the left figures of the upper and lower groups are the original images, the middle figures are the effects after HED processing, and the right figures are the effects after processing by the method of the present invention.
[0036] Figure 6This is a comparison of the accuracy of the two algorithms of the present invention under the FOM (Figure of Merit) evaluation quality index. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:
[0038] This paper proposes an edge detection network method based on the fusion of HED and Transformer. Figure 1 As shown. The method utilizes a convolution-based encoder, a HED weighted fusion module, a decoder based on a multi-head self-attention mechanism, a Transformer weighted fusion module, and a total weighted fusion module. Among them, the convolution-based encoder is used to extract feature maps with edge details, output multi-scale and multi-level feature maps for information extraction, generate context information, and provide powerful feature representation for the decoder based on the multi-head self-attention mechanism. The HED weighted fusion module is used to enhance local information on the edge probability map output by the convolution encoder. The decoder based on the multi-head self-attention mechanism is used to dynamically select relevant information in the input sequence and capture the relationship between images at long distances, thereby generating output consistent with the context information. The Transformer weighted fusion module is used to enhance global information on the edge probability map output by the decoder based on the multi-head self-attention mechanism. The total weighted fusion module fuses local and global edges through weights, and finally outputs an edge detection map that takes into account both global processing and local details.
[0039] Example 1:
[0040] This embodiment provides an edge detection network method based on the fusion of HED and Transformer. The specific process is as follows:
[0041] S1: Using a convolutional encoder, through multiple convolutional layers, it performs feature extraction and information fusion on the input image, outputs multi-scale, multi-level feature maps, and provides a powerful feature representation for the subsequent decoder based on the multi-head self-attention mechanism. Specifically, the convolutional encoder inputs the training set into a convolutional neural network (CNN) and outputs 5 layers of feature maps. The feature maps of each layer output by the CNN through the residual connection are labeled C1, C2, C3, C4, and C5 in order.
[0042] Acquiring a data set for a complex environment requires data on large, medium, and small objects as well as data with blurred edges. In this embodiment, the BSDS500 data set is used.
[0043] The HED backbone network, namely the VGG-16 network structure, is selected in the network. The last layer of each convolutional block of the HED backbone network is replaced with a Transformer module (the specific processing method in the Transformer module is the multi-head self-attention mechanism shown in formula (2)). The feature map is passed through three different projection layers to obtain Q, Detailed long-range relationships are extracted on high-resolution feature maps, providing powerful feature representation for the subsequent decoder based on the multi-head self-attention mechanism.
[0044] Step S2: Input the feature maps C1, C2, C3, C4, and C5 into the HED weighted fusion module, and the feature map after Concat weighted output is F HED ;
[0045] Among them, the HED weighted fusion module is constructed as follows Figure 3 As shown in the figure, specifically: the feature maps C1, C2, C3, C4, and C5 are first passed through Conv2d with kernel_size=1, the output size remains unchanged, and the number of channels is 1. Then, bilinear interpolation upsampling operation is performed through interpolate to change the image space size, and then weighted fusion is performed through Concat. Finally, the image is compressed through Conv2d with kernel_size=1 and stride=1 and sigmoid function to generate an edge probability map mapped to between 0 and 1, and the feature map F is obtained. HED , the specific processing is shown in formula (1);
[0046]
[0047] In formula (1): w m Represents the feature map C m The fusion weight size is updated through training; M=5.
[0048] Step S3: Input the feature maps C1, C2, C3, C4, and C5 into the decoder based on the multi-head self-attention mechanism, and collect long-range dependencies of multiple scales at the same time to achieve the fusion of low-resolution and high-resolution feature maps, and generate multi-scale feature maps T1, T2, T3, and T4.
[0049] Among them, the decoder based on the multi-head self-attention mechanism is constructed as follows Figure 2As shown in FIG, relative position coding is added to the multi-head self-attention mechanism of the decoder to become an efficient self-attention decoder with relative position coding; that is, by inputting the position information of the image unit (token), the corresponding position coding (relative position coding) is obtained, and the corresponding position coding matrices in two dimensions are defined respectively, and finally the multi-head self-attention mechanism is modified. The multi-head self-attention mechanism of the decoder is shown in formula (2);
[0050]
[0051] In formula (2), Q, are the matrix representations of query, key, and value, respectively, d is the vector dimension of the key, and Represents the relative position encoding matrix based on height and width respectively; Attention(Q, ) represents the multi-head self-attention mechanism.
[0052] Specifically, in the decoder, the deep feature map C5 is first decoded by multi-head self-attention. Through the masking mechanism (Masked Multi-Head Attention), it is ensured that the current position only focuses on the generated part to prevent information leakage. Then, the decoded C5 and the high-resolution feature map C4 are channel-spliced to fuse deep semantics with shallow details, and finally T1 is output through the feedforward network; T2 and T3 are generated in the same way; T4 is generated differently. It is first bilinearly interpolated and up-sampled through C2, and the image size is adjusted to be consistent with C1. Then, the number of channels of the image is unified through Conv2d with kernel_size=1, and then Concat operation is performed with C1 to finally obtain T4.
[0053] Step S4: Input the feature maps T1, T2, T3, and T4 into the Transformer weighted fusion module, and the feature map after Concat weighted output is F transformer ;
[0054] The construction of Transformer weighted fusion module is as follows Figure 4 As shown, the feature maps T1, T2, T3, and T4 first pass through Conv2d with kernel_size=1, and the output size remains unchanged, the number of channels is 1, and then bilinear interpolation upsampling operation is performed through interpolate to change the spatial size of the image, and then weighted fusion is performed through Concat to obtain a smoother feature map F transformer , the specific processing is shown in formula (3);
[0055]
[0056] In formula (3): w n Represents the feature map T n The weight size is integrated and updated through training; N=4.
[0057] Step S5: F HED and F transformer Through the total weighted fusion module, local detail extraction and global feature processing are fused to generate the final edge detection map F End , the total weighted fusion module is shown in formula (4);
[0058] F End =wF HED +F transformer (4)
[0059] In formula (4), w is a hyperparameter. The optimal value of w is found through ablation experiments, and the value range of w is 0-1.
[0060] Combine Figure 5-6 To demonstrate the superiority of the method of the present invention, the simulation is as follows:
[0061] To verify the superiority of the proposed method, a simulation analysis of the accuracy of the edge detection network method based on the fusion of HED and Transformer was conducted. The simulation conditions are: training set BSDS500 dataset, training number of epochs = 150, batch_size = 2. Figure 5 This is a comparison chart of the edge detection effects of the present invention and the traditional convolutional neural network (HED). Figure 5 It can be seen that from a visual perspective, compared with HED, the edge detection network method based on the fusion of HED and Transformer in the present invention is more refined and the position is not shifted in complex scenes and blurred edges, which effectively improves the integrity of edge information and the retention of details. Figure 6 This is a comparison of the accuracy of the two algorithms of the present invention under the FOM (Figure of Merit) quality evaluation index. Figure 6 It can be seen that in terms of the FOM-worthy evaluation quality index, the edge detection accuracy of the present invention is significantly improved.
[0062] Therefore, the edge detection network method based on the fusion of HED and Transformer proposed in this invention has significant advantages.
[0063] The above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may be modified or some of the technical features may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention and are therefore intended to be included within the scope of protection of the present invention.
Claims
1. An edge detection network method based on HED and Transformer fusion, characterized by: The edge detection network method is performed using a convolution-based encoder, a HED weighted fusion module, a decoder based on a multi-head self-attention mechanism, a Transformer weighted fusion module, and a total weighted fusion module; the method comprises the following steps: S1, using a convolution-based encoder, performs feature extraction and information fusion on the input image through multiple convolutional layers, and outputs feature maps C1, C2, C3, C4, and C5; S2, input the feature maps C1, C2, C3, C4, and C5 into the HED weighted fusion module, and the feature map after Concat weighted output is F HED ; S3, input the feature maps C1, C2, C3, C4, and C5 into the decoder based on the multi-head self-attention mechanism, and obtain the feature maps T1, T2, T3, and T4 after processing by the decoder; S4, input the feature maps T1, T2, T3, and T4 into the Transformer weighted fusion module, and the feature map after Concat weighted output is F transformer ; S5, the feature map F HED and feature map F transformer After processing by the total weighted fusion module, the final weighted fusion feature map F is obtained End .
2. The edge detection network method based on HED and Transformer fusion according to claim 1 is characterized by: The convolution-based encoder inputs the training set into a convolutional neural network (CNN) and outputs 5 layers of feature maps. Each layer of feature maps output by the CNN through residual connection is marked in sequence as feature maps C1, C2, C3, C4, and C5.
3. The edge detection network method based on HED and Transformer fusion according to claim 1 is characterized in that: The HED weighted fusion module has a Conv2d with kernel_size=1 and stride=1 and a sigmoid function, which is used to compress the image and generate an edge probability map mapped between 0 and 1. The specific processing process is: the feature maps C1, C2, C3, C4, and C5 first pass through Conv2d with kernel_size=1, and the output size remains unchanged and the number of channels is 1. Then, bilinear interpolation upsampling operation is performed through interpolate to change the image space size (to the same size as the input image), and then weighted fusion is performed through Concat to compress and obtain the feature map F HED , the specific processing is shown in formula (1); In formula (1): w m Represents the feature map C m The fusion weight size is updated through training; M=5.
4. The edge detection network method based on HED and Transformer fusion according to claim 1 is characterized in that: The decoder based on the multi-head self-attention mechanism adds relative position coding to the multi-head self-attention mechanism of the decoder, thereby becoming an efficient self-attention decoder with relative position coding. The specific processing method is as follows: by inputting the position information of the image unit (token), the corresponding position coding (relative position coding) is obtained, and the corresponding position coding matrices in two dimensions are defined respectively, and finally the multi-head self-attention mechanism is modified. The calculation of the multi-head self-attention mechanism of the decoder is shown in formula (2); In formula (2), Q, are the matrix representations of query, key, and value, respectively, d is the vector dimension of the key, and Represent the height-based and width-based relative position encoding matrices respectively; Represents the multi-head self-attention mechanism.
5. The edge detection network method based on HED and Transformer fusion according to claim 1 is characterized in that: The Transformer weighted fusion module has a Conv2d with kernel_size=1 and stride=1 and a sigmoid function, which is used to compress the image and generate an edge probability map mapped between 0 and 1. The specific processing process is: the feature maps T1, T2, T3, and T4 first pass through Conv2d with kernel_size=1, and the output size remains unchanged and the number of channels is 1. Then, bilinear interpolation upsampling operation is performed through interpolate to change the spatial size of the image (to the same size as the input image), and then weighted fusion is performed through Concat to compress and obtain the feature map F transformer , the specific processing is shown in formula (3); In formula (3): w n Represents the feature map T n The weight size is integrated and updated through training; N=4.
6. The edge detection network method based on HED and Transformer fusion according to claim 1, characterized in that: The total weighted fusion module is to obtain the F HED and F obtained after processing by the Transformer weighted fusion module transformer Through the fusion process of formula (4), the edge detection map F that takes into account both local and global processing is finally obtained. End ; F End =wF HED +F transformer (4) In formula (4), w is a hyperparameter. The optimal value of w is found through ablation experiments, and the value range of w is 0-1.
7. The edge detection network method based on HED and Transformer fusion according to claim 1 is characterized in that: The HED backbone network, namely the VGG-16 network structure, is selected in the network of this edge detection network method, and the feature maps output by different layers of the backbone network are selected for subsequent networks to perform edge feature extraction, information fusion, long-distance dependency capture and other functions.