A dark vision and low-light image edge detection method based on deep learning
By introducing self-attention and channel self-attention mechanisms in the edge detection model, combining multi-scale feature map fusion and similarity enhancement methods, the problem of edge detection difficulties in dark vision/low illumination environment is solved, and the accuracy of detection is improved and continuity, achieving better edge feature extraction and model training efficiency.
Patent Information
- Application Number
- CN202210366311.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-08
AI Technical Summary
Existing edge detection methods are difficult to effectively detect image edges in dark vision or low illumination environments, especially due to small pixel differences and large noise, edge information is flooded by strong backgrounds, and the limitations of existing data sets lead to the model in these environments. Poor performance.
Introduce self-attention mechanism and channel self-attention mechanism, combine spatial self-attention and channel self-attention mechanism to enhance the feature extraction ability of convolution, and improve weighting through multi-scale feature map fusion and similarity-based edge feature enhancement methods. Cross-entropy loss function, using Dice coefficient to calculate the loss fusion weight, dynamically adjust the enhancement effect of edge features.
Improve the sensitivity and feature extraction ability of the network to pixel differences, improve the accuracy and continuity of edge detection, enhance the edge detection effect in dark vision/low illumination environment, and avoid unreasonable manual weighting on the detection effect Influence.
Smart Images

Figure CN114693712B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of image processing, computer vision and deep learning, and in particular to a dark vision and low-illumination image edge detection method based on deep learning. Background Art
[0002] Edge detection design is often used to locate and identify the shape or type of target objects in images. It is one of the basic steps in image processing, image analysis, pattern recognition, computer vision and human vision, and is widely used in visual navigation, medical diagnosis and other fields.
[0003] Existing edge detection methods can be divided into traditional edge detection algorithms and edge detection algorithms based on deep learning. Traditional edge detection algorithms, such as the Canny operator and the Sobel operator, only rely on single-scale features such as image gradient changes for edge detection. When processing high-noise images with low contrast and low gradient such as dark vision or low-light images, problems such as edge blur and large-area edge loss will occur. Deep learning can automatically establish the dependency relationship between the underlying features of the image, abstract the underlying pixel information into semantics and concepts containing multi-scale features, and combine the correlation of multi-scale features to screen edges. For example, methods such as DeepEdge, HED, and CED do not require manual design, and can obtain edge features of different scales by simply modifying the subnet structure. Compared with traditional methods, edge detection algorithms based on deep learning have higher accuracy and stronger generalization ability.
[0004] In dark vision or low illumination scenes, due to the low brightness of the surrounding environment, the pixel differences of the collected images are small, the noise is large, and the edge information is easily submerged by the strong background. To perform edge detection tasks on such images, the algorithm must have a good perception of pixel differences. Among the existing edge detection methods, there are many studies on normal illumination images, but few people have proposed a complete solution to the above problems. In addition, the existing edge detection datasets contain limitations in the scenes, which makes it difficult for the trained edge detection model to perform well in dark vision or low illumination environments. Therefore, it is necessary to further explore edge detection methods suitable for dark vision and low illumination environments. Summary of the invention
[0005] In view of this, in order to improve the sensitivity of the detection model to pixel differences, the present invention introduces a self-attention mechanism in the edge detection model. Convolution mainly senses features through a limited receptive field and weight sharing, so convolution is inherently local and translationally equivalent, and cannot obtain the global context of the image. The global context is essential for the recognition of target features, and the self-attention mechanism introduces the position information of the pixel to enable long-range interaction between features, that is, each position has information about any other feature in the same image. By replacing or combining with convolution, a larger receptive field than conventional convolution can be obtained, and the feature extraction ability of convolution can be enhanced. Extracting edge feature maps of different scales for fusion can avoid the loss of features as the network deepens, which is conducive to better retaining edge details. For edge detection models that output multi-scale feature maps, as the network depth gradually increases, the receptive field of the feature map continues to increase, and the scales of edge features that each module pays attention to are different. It is unreasonable to use feature maps of different scales and the same edge label to calculate the loss. Therefore, in the edge detection method, a channel self-attention mechanism is introduced to learn the edge scales corresponding to edge features at different stages. At the same time, the weighted cross entropy loss is improved. In the original weighted cross entropy loss, the loss between the multiple single-scale edge features and edge labels output by the network will be fused according to the manually set weights, and the unreasonable manual weights will have a greater impact on the detection effect of the model. In order to avoid this problem, the present invention first uses the Dice coefficient to calculate the similarity between the single-scale features and the edge labels, and uses the similarity as the fusion weight of the multi-scale edge loss. Secondly, in order to increase the gradient difference between edge pixels and non-edge pixels, the present invention proposes an edge feature enhancement method based on pixel similarity. First, a primary edge feature map is extracted from a dark vision / low illumination image using a neural network with a small number of layers, and then the pixel similarity between the primary edge feature map and the dark vision / low illumination image is measured by calculating the pixel Euclidean distance. In the dark vision / low illumination image, the part with the highest similarity to the edge feature map is the edge feature in the dark vision / low illumination image. Taking similarity as the evaluation criterion, the edge feature in the dark vision / low illumination image will obtain a higher similarity score, and the non-edge feature will obtain a lower similarity score. Fusion of the similarity matrix and the dark vision image can enhance the edge features in the dark vision image and suppress the non-edge features.
[0006] The purpose of the present invention is to provide a dark vision and low-light image edge detection method based on deep learning, the main steps are as follows:
[0007] Acquiring a dark vision / low illumination image, and preprocessing the dark vision / low illumination image;
[0008] Input the dark vision / low illumination image into the pre-trained primary subnet for edge detection, extract multiple edge feature maps, and form the first edge feature after upsampling;
[0009] Inputting the dark vision / low illumination image and the corresponding first edge feature into an edge enhancement module to enhance the edge of the dark vision / low illumination image;
[0010] The edge-enhanced dark vision / low-light image is input into the secondary subnetwork for edge detection, and multiple edge feature maps are extracted;
[0011] The first edge feature and multiple edge feature maps generated by the primary subnet are fused with multiple edge feature maps generated by the secondary subnet to obtain the edge detection effect of the dark vision / low illumination image.
[0012] Beneficial effects of the present invention:
[0013] The present invention extracts feature maps of different scales for fusion, and combines the spatial self-attention mechanism in the convolution so that the network can pay attention to the global feature information at a shallow level, thereby improving the network's feature extraction ability and sensitivity to pixel differences, and using the channel self-attention mechanism to adaptively learn the edge scales corresponding to edge features of different levels. The weighted cross entropy loss function is improved, and the similarity between single-scale edge features and edge labels is used as the loss fusion weight to avoid the influence of unreasonable manual weights on the network detection effect and improve the training efficiency of the model. Secondly, a similarity-based edge feature enhancement method is proposed. Through this method, the real-time parameters of the training process can be used to dynamically adjust the edge feature enhancement effect. As the loss decreases, the positioning accuracy of the edge feature of this method will gradually improve, which is helpful to improve the detection effect of the edge detection model in dark vision / low illumination environment, and the extracted edge continuity is enhanced, and the small-grained information is richer. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only part of the embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0015] Figure 1 It is a network architecture diagram of a dark vision and low-light edge detection method based on deep learning of the present invention;
[0016] Figure 2 It is a system flow chart of a dark vision and low-light image edge detection method based on deep learning of the present invention;
[0017] Figure 3 This is a diagram showing the effect of edge detection of a dark vision image by the pre-trained primary subnet of the present invention;
[0018] Figure 4 is a similarity calculation process between the first edge feature and the dark vision / low illumination image of the present invention;
[0019] Figure 5 It is the effect diagram of the edge feature enhancement method of the present invention;
[0020] Figure 6 This is a comparison of the effects of the method described in the present invention and other edge detection models in dark vision and low-light image edge detection. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0022] Figure 1 is a network architecture diagram of one embodiment of a method for dark vision and low-light image edge detection based on deep learning in an embodiment of the present invention, such as Figure 1 As shown, in an embodiment of the present invention, the network architecture includes a primary subnet, an edge enhancement module and a secondary subnet, wherein the primary subnet includes a primary feature extraction module, a composite feature extraction module and a deep feature fusion module, and the secondary subnet includes a composite feature extraction module and a deep feature fusion module; the final edge detection result is obtained through the joint action of the primary subnet, the edge enhancement module and the secondary subnet.
[0023] Based on the network architecture of the above embodiment, Figure 2 This is a system flow chart of a dark vision and low-light image edge detection method based on deep learning provided by the present invention, which specifically includes the following steps:
[0024] S1: Acquire a dark vision / low illumination image, and preprocess the dark vision / low illumination image;
[0025] Dark vision / low illumination imaging refers to the use of image acquisition equipment in an environment with a brightness lower than 0.001cd / m 2 The original image is collected in the imaging environment at that time, and the average background grayscale of the original image is between 0 and 47. The present invention does not limit the content of the original image.
[0026] Among them, a well-annotated dark vision / low illumination edge detection data set can be used, or a self-built dark vision / low illumination edge detection data set can be used. The dark vision / low illumination edge detection data set used in this embodiment is divided into a training set and a test set. The training set is a LOL data set (selecting 50 dark vision and 50 low illumination images) and a self-built data set (selecting 50 dark vision and 50 low illumination images), annotating and generating edge labels, and the test set is a LOL data set (selecting 20 dark vision images and 20 low illumination images) and a self-built data set (selecting 20 dark vision images and 20 low illumination images), annotating and generating edge labels. The training set can be enhanced to form 57,600 training samples after pre-processing operations such as flipping, cropping, and scaling.
[0027] It can be understood that, in an embodiment of the present invention, the dark vision / low illumination image can be either a training set image with a known edge label or a test image with an unknown edge label; when the dark vision / low illumination image is a training set image with a known edge label, the primary subnet can be pre-trained as a training set, or the edge detection network composed of the pre-trained primary subnet, the untrained edge enhancement module and the untrained secondary subnet can be trained again; when the dark vision / low illumination image is a test image with an unknown edge label, the trained edge detection network can be used to detect and identify it, and the edge label of the test image can be predicted.
[0028] S2: Input the dark vision / low illumination image into the pre-trained primary subnet for edge detection, extract multiple edge feature maps, and form the first edge feature after upsampling;
[0029] In this embodiment of the present invention, the primary subnet needs to be trained first, wherein a dark vision / low illumination edge detection dataset can be used to train the primary subnet, the network is trained for 25 rounds, the initial learning rate is set to 0.0001, the optimizer is set to Adam, and the number of training times, learning rate and optimizer are all optional.
[0030] Figure 3 is the edge detection effect after the primary subnet completes pre-training, such as Figure 3 As shown in the figure, it can be seen that due to the small number of convolutional layers in the primary subnet, the receptive field of the feature map is small, and the edge detection results describe more edge details of the local pattern.
[0031] After the primary subnetwork pre-training is completed, it can be used for edge detection of dark vision / low illumination images, thereby extracting edge feature maps of dark vision / low illumination images at different scales. After upsampling these edge feature maps at different scales, the first edge feature can be formed.
[0032] Specifically, in the embodiment of the present invention, the primary subnet includes ι1 primary feature extraction modules and ι2 composite feature extraction modules, a deep feature fusion module and ι1+ι2+1 adaptive edge scale extraction modules. The ι1 primary feature extraction modules and ι2 composite feature extraction modules of the primary subnet generate ι1+ι2 edge features, and the ι1+ι2 edge features are input into the adaptive edge scale extraction module to extract the adaptive edge scale of each edge feature, and the scale and the corresponding edge feature are fused to obtain ι1+ι2 edge feature maps. After upsampling the ι1+ι2 edge feature maps, the deep feature fusion module is input according to channel splicing to obtain the first edge feature. Wherein, ι1 and ι2 are positive integers.
[0033] In a specific embodiment of the present invention, it is assumed that the primary subnet includes 1 primary feature extraction module, 2 composite feature extraction modules, 1 deep feature fusion module and 4 adaptive edge scale extraction modules, as shown in Table 1.
[0034] Table 1 Detailed parameters of each layer of the primary subnet
[0035]
[0036]
[0037] In the above embodiment, the primary feature extraction module: the primary feature extraction module includes 2 convolution layers in total, the convolution kernel size of the first convolution layer is 3*3, the number of channels is 3, the number is 32, the step size is 2, the filling method is SAME, the activation function is the ReLU function, and batch normalization is performed before passing it into the ReLU function; the convolution kernel size of the second convolution layer is 3*3, the number of channels is 32, the number is 64, the step size is 1, the filling is 1, the activation function is the ReLU function, and batch normalization is performed before passing it into the activation function. In summary, the primary edge feature conv1_1 is obtained.
[0038] In the above embodiment, the first composite edge feature extraction module: the input is conv1_1, which includes a local feature extraction network and a global feature extraction network. The local feature extraction network contains 4 convolutional layers, the last convolutional layer is used for downsampling, the convolution kernel size of the first convolutional layer is 3*3, the number of channels is 64, the number is 128, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it into the ReLU function; the convolution kernel size of the second convolutional layer is 3*3, the number of channels is 128, the number is 128, the step size is 1, the padding is 1, and the activation function is the ReLU function; the convolution kernel size of the third convolutional layer is 3*3, the number of channels is 128, the number is 128, the step size is 1, the padding is 1, and the activation function is the ReLU function. In addition, conv1_1 will undergo a 1*1 convolution to adjust the number of channels to 128, and then map it to the tail of the first composite edge feature extraction module and the output of the fourth convolutional layer to obtain the feature map conv2_2.
[0039] Among them, the input of the global feature extraction network is conv1_1, which includes a spatial self-attention layer. The output of the global feature extraction network will be mapped to the local feature extraction network in the form of residual, and added to the output of the first, second and third convolutional layers of the local feature extraction network respectively as the input of the next convolutional layer. The output of the third convolutional layer is the composite edge feature conv2_1.
[0040] In the above embodiment, the second composite edge feature extraction module: the input is conv2_2, which includes a local feature extraction network and a global feature extraction network. The local feature extraction network contains 4 convolutional layers. The convolution kernel size of the first convolutional layer is 3*3, the number of channels is 128, the number is 256, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing into the ReLU function; the convolution kernel size of the second convolutional layer is 3*3, the number of channels is 256, the number is 256, the step size is 1, and the padding is 1; the convolution kernel size of the third convolutional layer is 3*3, the number of channels is 256, the number is 256, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing into the ReLU function; the convolution kernel size of the fourth convolutional layer is 3*3, the number of channels is 256, the number is 256, the step size is 1, the padding is 1, and the output is the composite edge feature conv3_1. In addition, conv2_2 will undergo a 1*1 convolution to adjust the number of channels to 256, and then map it to the tail of the second composite edge feature extraction module and the output of the fourth convolution layer to obtain the feature map conv3_2.
[0041] Among them, the input of the global feature extraction network is conv2_2, which includes a spatial self-attention layer. The output of the global feature extraction network will be mapped to the local feature extraction network in the form of residual, and added to the output of the first to third convolutional layers of the local feature extraction network as the input of the fourth convolutional layer.
[0042] In the above embodiment, the deep feature fusion module: conv1_1, conv2_1 and conv3_2 are upsampled to 400*400 after being processed by the adaptive edge scale module, and are spliced by channel as the input of the deep feature fusion module. The deep feature fusion module contains a total of 4 convolutional layers. The convolution kernel size of the first convolution layer is 3*3, the number of channels is 3, the number is 10, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function; the convolution kernel size of the second convolution layer is 3*3, the number of channels is 10, the number is 20, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function. The convolution kernel size of the third convolution layer is 3*3, the number of channels is 20, the number is 20, the step size is; the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function. The convolution kernel size of the fourth convolution layer is 3*3, the number of channels is 20, the number is 1, the step size is 1, the padding is 1, and the output is the first edge feature fuse_1.
[0043] S3: inputting the dark vision / low illumination image and the corresponding first edge feature into an edge enhancement module to enhance the edge of the dark vision / low illumination image;
[0044] In an embodiment of the present invention, the first edge feature may be used to perform edge enhancement on the originally input dark vision / low illumination image, wherein the process of enhancing the edge of the dark vision / low illumination image by the edge enhancement module includes:
[0045] The gradient operator is used to extract the gradient map of the dark vision / low illumination image in the horizontal and vertical directions;
[0046] Using a gradient operator to extract the gradient map of the first edge feature along the horizontal direction and the vertical direction;
[0047] Based on the gradient map in the corresponding direction, the similarities between the first edge feature and the dark vision / low illumination image pixels are respectively calculated to obtain a similarity matrix;
[0048] The similarity matrix and the dark vision / low illumination image are multiplied pixel by pixel to obtain the dark vision / low illumination image after edge enhancement.
[0049] Among them, the similarity between the first edge feature and the dark vision / low illumination image pixels is calculated based on the gradient map of the corresponding direction, and the similarity matrix obtained includes:
[0050]
[0051] Similarity(X1, X2)=1-F norm (Distance(X1,X2))
[0052] Where Distance(·) is the Euclidean distance of pixels, X1 and X2 represent the two pixels whose similarity is calculated, x represents the gradient of the pixel along the horizontal direction, y represents the gradient of the pixel along the vertical direction, Similarity(X1, X2) represents the similarity between pixels X1 and X2, and F norm Indicates normalization.
[0053] To further illustrate the process of edge enhancement module for edge enhancement of the dark vision / low illumination image, Figure 4 As shown, in this embodiment, two filter templates of the Prewwit edge detection operator are selected to extract the first edge feature and the gradient map of the dark vision / low illumination image along the horizontal direction and the vertical direction, as shown below:
[0054]
[0055] Using G x and G y By convolving fuse_1 and the dark vision / low illumination image with a step size of 1, the gradient maps of the two in the horizontal and vertical directions can be obtained. The similarity matrix of the two is calculated according to the pixel similarity calculation formula, and the similarity matrix and the dark vision / low illumination image are multiplied pixel by pixel to obtain the dark vision / low illumination image EEM after edge enhancement. The effect diagram can be referred to as follows Figure 5 shown.
[0056] S5: Input the edge-enhanced dark vision / low illumination image into the secondary subnetwork for edge detection, and extract multiple edge feature maps;
[0057] In an embodiment of the present invention, the pre-trained primary subnet, the untrained edge enhancement module and the untrained secondary subnet constitute an edge detection network and need to be trained together. The training process needs to ensure the convergence of the loss function so that the final prediction result is accurate.
[0058] When the edge detection network training is completed, it can be used for edge detection of dark vision / low illumination images. The primary subnet is used to extract the first edge feature and multiple edge feature maps of different scales. The edge enhancement module is used to enhance the dark vision / low illumination image. The secondary subnet is then used to extract multiple edge feature maps of different scales after image enhancement. Finally, the first edge feature and multiple edge feature maps generated by the primary subnet can be fused with the multiple edge feature maps generated by the secondary subnet to obtain the edge detection effect of the dark vision / low illumination image.
[0059] The secondary subnetwork comprises ι3 composite feature extraction modules, a deep feature fusion module and ι3+1 adaptive edge scale extraction modules; the ι3 composite feature extraction modules of the secondary subnetwork generate ι3 edge features, the ι3 edge features are input into the adaptive edge scale extraction module, the adaptive edge scale of the edge features is extracted, and the scale and edge features are fused to obtain ι3 edge feature maps; wherein ι3 is a positive integer.
[0060] In a specific embodiment of the present invention, it is assumed that the secondary subnetwork includes 3 composite feature extraction modules, 1 deep feature fusion module and 3 adaptive edge scale modules, as shown in Table 2.
[0061] Table 2 Detailed parameters of each layer of the secondary subnet
[0062]
[0063]
[0064]
[0065] In the above embodiment, the first composite edge feature extraction module: the input is EEM, which includes a local feature extraction network and a global feature extraction network. The local feature extraction network contains 8 convolutional layers. The convolution kernel size of the first convolutional layer is 3*3, the number of channels is 3, the number is 16, the step size is 1, the padding is 1, and the activation function is the ReLU function. Batch normalization is performed before passing it into the ReLU function; the convolution kernel size of the second convolutional layer is 1*1, the number of channels is 16, the number is 16, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the third convolutional layer is 3*3, the number of channels is 16, the number is 16, the step size is 2, and the padding is 1; the convolution kernel size of the fourth convolutional layer is 1*1, the number of channels is 16, the number is 32, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the fifth convolutional layer is 1*1, the number of channels is 32, the number is 32, the step size is 1, and the padding is is 0, and the activation function is the ReLU function; the convolution kernel size of the sixth convolution layer is 3*3, the number of channels is 32, the number is 32, the step size is 1, and the padding is 1; the convolution kernel size of the seventh convolution layer is 1*1, the number of channels is 32, the number is 64, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the eighth convolution layer is 3*3, the number of channels is 64, the number is 128, the step size is 1, the padding is 1, and the activation function is the ReLU function. Batch normalization is performed before passing it into the ReLU function; in addition, EEM will adjust the number of channels to 128 through a 1*1 convolution, and then map it to the tail of the first composite edge feature extraction module and add the output of the fourth convolution layer to obtain the feature map conv4_2.
[0066] In the above embodiment, the input of the global feature extraction network is EEM, which includes a spatial self-attention layer. The output of the global feature extraction network is mapped to the local feature extraction network in the form of a residual, and is added to the outputs of the first, fourth and seventh convolutional layers of the local feature extraction network respectively as the input of the next convolutional layer. The output of the eighth convolutional layer is the composite edge feature conv4_1.
[0067] In the above embodiment, the second composite edge feature extraction module: the input is conv4_2, which includes a local feature extraction network and a global feature extraction network. The local feature extraction network contains 9 convolutional layers. The convolution kernel size of the first convolutional layer is 3*3, the number of channels is 128, the number is 128, the step size is 1, the padding is 1, and the activation function is the ReLU function. Batch normalization is performed before passing it to the ReLU function; the convolution kernel size of the second convolutional layer is 1*1, the number of channels is 128, the number is 128, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the third convolutional layer is 3*3, the number of channels is 128, the number is 128, the step size is 2, and the padding is 1; the convolution kernel size of the fourth convolutional layer is 1*1, the number of channels is 128, the number is 256, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the fifth convolutional layer is 3*3, the number of channels is 256, the number is 256, the step size is 1, and the padding is 1; the convolution of the sixth convolutional layer The kernel size is 1*1, the number of channels is 256, the number is 128, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the seventh convolution layer is 3*3, the number of channels is 128, the number is 128, the step size is 1, and the padding is 1; the convolution kernel size of the eighth convolution layer is 1*1, the number of channels is 128, the number is 256, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the ninth convolution layer is 3*3, the number of channels is 256, the number is 512, the step size is 1, the padding is 1, and the activation function is the ReLU function. Batch normalization is performed before passing it into the ReLU function; in addition, conv_4_2 will adjust the number of channels to 512 through a 1*1 convolution, and then map it to the tail of the first composite edge feature extraction module and add the output of the fourth convolution layer to obtain the feature map conv5_2.
[0068] The input of the global feature extraction network is conv4_2, which includes a spatial self-attention layer. The output of the global feature extraction network is mapped to the local feature extraction network in the form of residuals, and is added to the outputs of the first, fourth, fifth, and eighth convolutional layers of the local feature extraction network as the input of the next convolutional layer. The output of the ninth convolutional layer is the composite edge feature conv5_1.
[0069] In the above embodiment, the third composite edge feature extraction module: the input is conv5_2, which includes a local feature extraction network and a global feature extraction network. The local feature extraction network includes 8 convolutional layers. The convolution kernel size of the first convolution layer is 3*3, the number of channels is 512, the number is 256, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function; the convolution kernel size of the second convolution layer is 1*1, the number of channels is 256, the number is 128, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the third convolution layer is 3*3, the number of channels is 128, the number is 128, the step size is 2, and the padding is 1; the convolution kernel size of the fourth convolution layer is 1*1, the number of channels is 128, the number is 256, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the fifth convolution layer is 1*1, the number of channels is 256, the number is 128, the step size is 1, and the padding is The filling is 0, and the activation function is the ReLU function; the convolution kernel size of the sixth convolution layer is 3*3, the number of channels is 128, the number is 128, the step size is 1, and the padding is 1; the convolution kernel size of the seventh convolution layer is 1*1, the number of channels is 128, the number is 256, the step size is 1, the padding is 0, and the activation function is the ReLU function; the convolution kernel size of the eighth convolution layer is 3*3, the number of channels is 256, the number is 128, the step size is 1, the padding is 1, and the activation function is the ReLU function. Batch normalization is performed before passing it into the ReLU function; in addition, conv5_2 will adjust the number of channels to 128 through a 1*1 convolution, and then map it to the tail of the first composite edge feature extraction module and add the output of the fourth convolution layer to obtain the feature map conv6_2.
[0070] Among them, the input of the global feature extraction network is conv5_2, which includes a spatial self-attention layer. The output of the global feature extraction network will be mapped to the local feature extraction network in the form of residual, and added to the output of the first, fourth and seventh convolutional layers of the local feature extraction network respectively.
[0071] S5: Fusing the first edge feature and multiple edge feature maps generated by the primary subnet with the multiple edge feature maps generated by the secondary subnet to obtain an edge detection effect for dark vision / low illumination images.
[0072] In the embodiment of the present invention, in the primary subnet, ι1 primary feature extraction modules and ι2 composite feature extraction modules generate ι1+ι2 edge features, the ι1+ι2 edge features are input into the adaptive edge scale extraction module, the adaptive edge scale of the edge features is extracted, and the scale and the edge features are fused to obtain ι1+ι2 edge feature maps, after upsampling the ι1+ι2 edge feature maps, they are input into the deep feature fusion module according to channel splicing to obtain the first edge feature; in the secondary subnet, ι3 composite feature extraction modules generate ι3 edge features, the ι3 edge features are input into the adaptive edge scale extraction module, the adaptive edge scale of the edge features is extracted, and the adaptive edge scale of the edge features is extracted to obtain the first edge feature; Adapt to the edge scale, and fuse the scale and edge features to obtain ι3 edge feature maps; the ι1+ι2 edge feature maps and the first edge feature generated by the primary subnet, and the ι3 edge feature maps generated by the secondary subnet, that is, the upsampling results of the ι1+ι2+1+ι3 edge feature maps, are spliced by channel and input into the deep feature fusion module of the secondary subnet to obtain deep edge features; based on the edge loss function, the edge losses between the ι1+ι2+1+ι3 edge feature maps and the deep edge features and the edge labels are calculated respectively; when the edge loss converges, the output of the deep feature fusion module of the secondary subnet is the final edge detection result.
[0073] Specifically, the edge feature maps generated by the primary subnet and the secondary subnet are fused using the deep feature fusion module of the secondary subnet. After being processed by the adaptive edge scale module, conv1_1, conv2_1, conv3_2, conv4_1, conv5_1, conv6_2, and fuse_1 are upsampled to 400*400 and concatenated by channel as the input of the deep feature fusion module. The deep feature fusion module contains a total of 4 convolutional layers. The convolution kernel size of the first convolution layer is 3*3, the number of channels is 7, the number is 16, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function; the convolution kernel size of the second convolution layer is 3*3, the number of channels is 16, the number is 32, the step size is 1, the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function. The convolution kernel size of the third convolution layer is 3*3, the number of channels is 32, the number is 16, the step size is; the padding is 1, the activation function is the ReLU function, and batch normalization is performed before passing it to the ReLU function. The convolution kernel size of the fourth convolution layer is 3*3, the number of channels is 16, the number is 1, the step size is 1, the padding is 1, and the output is fuse_2.
[0074] Among them, the number of convolution layers, convolution kernel size, step size, filling method and activation function of the primary feature extraction module, composite feature extraction module and deep feature fusion module described in this embodiment are optional and can be determined according to the specific usage scenario.
[0075] Among them, the spatial self-attention layer described in this embodiment uses the spatial self-attention mechanism model BoTNet, and other spatial self-attention mechanism models may also be used, such as ViT, External Attention, etc.
[0076] Among them, the adaptive edge scale module described in this embodiment uses the channel self-attention mechanism model SENet, and other channel self-attention mechanism models may also be used, such as the pyramid segmentation self-attention module, ECAnet, etc.
[0077] In the second stage of training, the primary subnet and edge enhancement module are connected in series to form a complete dark vision and low-light edge detection network. The number of network training iterations is set to 150,000 times, the initial learning rate is set to 0.0001, and the optimizer is set to Adam. The number of training times, learning rate and optimizer are all optional.
[0078] In this embodiment, the edge loss function of the dark vision and low illumination edge detection network and the primary subnet is improved based on the weighted cross entropy loss. Other loss functions may also be selected, such as Focal loss, Tversky loss, etc., or the edge loss function may be designed by oneself. The improved weighted cross entropy loss function is as follows:
[0079] The loss function at each scale level is:
[0080] ι n (W,ω n )=A+B
[0081] in:
[0082]
[0083]
[0084] β=|Y - | / |Y + +Y - |
[0085] (1-β)=|Y + | / |Y + +Y - |
[0086] The total loss function is:
[0087]
[0088] The calculation formula of Dice coefficient is as follows:
[0089]
[0090] y n ∩y label Represents the single-scale edge feature y n and edge labels y label The intersection of can be calculated approximately by dot multiplication, |·| represents the addition of matrix elements, σ represents the sigmoid function, y n Represents the single-scale edge feature output by the nth module, y label represents the edge label, W is the set of all parameters, ω is the parameter of the nth module, and Y - and Y + Represents the edge and non-edge in the edge label. The edge loss is trained to convergence using the training set, at which point fuse_2 is the final edge detection result.
[0091] During the training process, as the loss converges, the edge detection capabilities of each module of the model will gradually increase, and the non-edge information contained in the first edge feature map will gradually decrease. When performing edge enhancement, it will tend to only enhance edge features, thereby improving the edge detection accuracy of the model.
[0092] Figure 6 It is the edge detection effect of the network model in the embodiment and the other two edge detection models (BDCN and DexiNed) on dark vision and low-light images. All three networks are trained using the training set.
[0093] Table 1 compares the objective evaluation indicators calculated by the network model in the embodiment and the other two edge detection models (BDCN and DexiNed) based on the self-built dark vision and low-illumination edge detection data aggregation and LOL data set. The peak signal-to-noise ratio (PSNR, the larger the better), mean square error (MSE, the smaller the better) and structural similarity (SSIM, the larger the better) are selected in this embodiment.
[0094] Table 3 Comparison of objective evaluation indicators
[0095]
[0096] Among them, Figure 6 It can be concluded from Table 3 that, compared with other models, the dark vision and low-illumination image edge detection method based on deep learning can better retain edge details, and the detected edges are thinner, have better continuity, and converge faster.
[0097] In the description of the present invention, it is necessary to understand that the terms "coaxial", "bottom", "one end", "top", "middle", "the other end", "upper", "one side", "top", "inside", "outside", "front", "center", "both ends" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0098] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.
[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A dark vision and low-light image edge detection method based on deep learning, It is characterized in that The method comprises: Acquiring a dark vision or / and low illumination image, and preprocessing the dark vision or / and low illumination image; Inputting the dark vision or / and low illumination image into the pre-trained primary subnet for edge detection, extracting a plurality of edge feature maps, and forming a first edge feature after upsampling; the step of inputting the dark vision or / and low illumination image into the pre-trained primary subnet for edge detection, extracting a plurality of edge feature maps, and forming a first edge feature after upsampling is included in the primary subnet, ι 1 A primary feature extraction module and 2 A composite feature extraction module generates 1 +ι 2 edge features, 1 +ι 2 The edge features are input into the adaptive edge scale extraction module to extract the adaptive edge scale of the edge features and fuse the scale and edge features to obtain 1 +ι 2 edge feature map, for ι v +ι 2 After upsampling the edge feature maps, they are input into the deep feature fusion module according to channel splicing to obtain the first edge feature; The dark vision or / and low illumination image and the corresponding first edge feature are input into the edge enhancement module to enhance the edge of the dark vision or / and low illumination image; the process of the edge enhancement module enhancing the edge of the dark vision or / and low illumination image includes: using a gradient operator to extract the gradient map of the dark vision or / and low illumination image along the horizontal direction and the vertical direction; using a gradient operator to extract the gradient map of the first edge feature along the horizontal direction and the vertical direction; based on the gradient map in the corresponding direction, respectively calculating the similarity between the first edge feature and the dark vision or / and low illumination image pixel, to obtain a similarity matrix; multiplying the similarity matrix and the dark vision or / and low illumination image pixel by pixel, to obtain the dark vision or / and low illumination image after edge enhancement; Inputting the edge-enhanced dark vision or / and low-light image into the secondary subnet for edge detection, and extracting a plurality of edge feature maps; Inputting the edge-enhanced dark vision or / and low-light image into the secondary subnet for edge detection, and extracting a plurality of edge feature maps are included in the secondary subnet, ι 3 A composite feature extraction module generates 3 edge features, 3 The edge features are input into the adaptive edge scale extraction module to extract the adaptive edge scale of the edge features and fuse the scale and edge features to obtain 3 edge feature maps; The first edge feature and multiple edge feature maps generated by the primary subnet are fused with the multiple edge feature maps generated by the secondary subnet to obtain edge detection effects for dark vision and / or low illumination images.
2. According to the method for dark vision and low illumination image edge detection based on deep learning in claim 1, It is characterized in that The first edge feature and multiple edge feature maps generated by the primary subnet are fused with multiple edge feature maps generated by the secondary subnet to obtain edge detection effects of dark vision and / or low illumination images, which are included in the primary subnet. 1 A primary feature extraction module and 2 A composite feature extraction module generates 1 +ι 2 edge features, 1 +ι 2 The edge features are input into the adaptive edge scale extraction module to extract the adaptive edge scale of the edge features and fuse the scale and edge features to obtain 1 +ι 2 edge feature map, for ι 1 +ι 2 After the edge feature maps are upsampled, they are input into the deep feature fusion module according to channel splicing to obtain the first edge feature; in the secondary subnet, ι 3 A composite feature extraction module generates 3 edge features, 3 The edge features are input into the adaptive edge scale extraction module to extract the adaptive edge scale of the edge features and fuse the scale and edge features to obtain 3 edge feature maps; the primary subnet generates 1 +ι 2 edge feature maps and the first edge feature, as well as the ι generated by the secondary subnet 3 edge feature map, i.e., ι 1 +ι 2 +1+ι 3 The results of upsampling the edge feature maps are concatenated by channel and input into the deep feature fusion module of the secondary subnet to obtain the deep edge features. Based on the marginal loss function, ι is calculated separately 1 +ι 2 +1+ι 3 An edge feature map and an edge loss between the deep edge feature and the edge label; when the edge loss converges, the output of the deep feature fusion module of the secondary subnet is the final edge detection result.
3. A dark vision and low illumination image edge detection method based on deep learning according to claim 1 or 2, It is characterized in that The composite feature extraction module is composed of a global feature extraction network and a local feature extraction network. The input of the local feature extraction network is mapped to the tail of the current composite feature extraction module in the form of a residual, or the tail and output of other composite feature extraction modules are added. The output of the global feature extraction network is mapped to different positions inside the local feature extraction network in the form of a residual, and is added to multiple intermediate inputs of the local feature extraction network respectively. The global features of the input feature map are extracted based on the global feature extraction network, and the local features of the input feature map are extracted based on the local feature extraction network. The local feature extraction network is composed of ι 4 convolutional layers, where ι 4 is a positive integer, and the global feature extraction network is composed of 5 spatial self-attention layers, where ι 5 is a positive integer.
4. A dark vision and low illumination image edge detection method based on deep learning according to claim 1 or 2, It is characterized in that The adaptive edge scale extraction module is composed of 6 It is composed of channel self-attention layers, which are used to extract the edge scale corresponding to any edge feature, and fuse the edge scale and the corresponding edge feature to obtain an edge feature map; where ι 6 Is a positive integer.
5. A dark vision and low illumination image edge detection method based on deep learning according to claim 1 or 2, It is characterized in that The deep feature fusion module consists of 7 It is composed of convolutional layers, which are used to integrate the input multi-scale edge feature maps into deep edge features. 7 is a positive integer.
6. The method for dark vision and low illumination image edge detection based on deep learning according to claim 1, It is characterized in that Based on the gradient map in the corresponding direction, the similarities between the first edge feature and the dark vision or / and low illumination image pixels are calculated respectively, and the obtained similarity matrix includes: Similarity(X 1 ,X 2 )=1-F norm (Distance(X 1 ,X 2 )) Where Distance(·) is the Euclidean distance of pixels, X 1 and X 2 Represents two pixels that calculate similarity, x represents the gradient of the pixel along the horizontal direction, y represents the gradient of the pixel along the vertical direction, Similarity(X 1 ,X 2 ) represents pixel X 1 and X 2 The similarity, F norm Indicates normalization.
7. The method for dark vision and low illumination image edge detection based on deep learning according to claim 1, It is characterized in that The pre-trained primary subnet and edge enhancement module are connected in series to form a complete dark vision and low-light edge detection network, and the Dice similarity between each single-scale edge feature and the edge label is calculated. The Dice similarity is used as an adaptive parameter to construct a weighted cross entropy loss until the weighted cross entropy loss converges, completing the training of the dark vision and low-light edge detection network.
