A bionic target contour extraction algorithm based on deep learning

Abstract

The present invention discloses a bionic target contour extraction algorithm based on deep learning. By combining the outer region suppression mechanism with the deep CNN framework, an edge detection model inspired by biological characteristics is established, and the edge detection model is used to extract the edge of the target image; the edge detection model includes a backbone network, a side output layer and an outer region suppression module SSM. The present invention achieves a better balance between the accuracy and efficiency of edge detection by designing an efficient backbone network, and designs an outer region suppression module SSM with a multi-layer convolutional layer stacking structure to increase the receptive field and characterize the spatial structure at different scales, effectively simulating the outer region suppression effect, and the SSM modulation operation can enable the network to integrate important information over a larger field of view. Moreover, the model of the present invention does not require pre-trained weights to assist in training, the network parameters are significantly reduced, and the reasoning speed under the same hardware conditions is faster. Compared with the prior art, the present invention is more efficient and lightweight.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a bionic target contour extraction algorithm based on deep learning. Background Art

[0002] Edge detection is an algorithm that attempts to find a high-contrast boundary that separates two areas. Intuitively, edges generally exhibit different specific patterns, such as straight lines, corners, and cross-connections. Traditional edge detection methods are inspired by these intuitive feelings and perform edge detection by explicitly calculating pixel differences to encode important gradient information. However, this cannot distinguish between key edges and texture edges perceived by the receptive field. Discoveries in the field of physiology have inspired edge detection algorithm researchers. In the 1960s, studies showed that the human visual system perceives isolated edges and texture edges differently in the early stages of processing visual information.

[0003] By definition, the receptive field in biology refers to the area where the best stimulus causes a strong response from the neuron. For example, for visual neurons, this is a bar or edge of a specific optimal size and direction that triggers the cell to respond. This area is currently called the classical receptive field CRF (Classical Receptive Field). Studies have shown that once a cell is activated by a stimulus in its CRF, stimuli presented simultaneously in a part of the outer circle of the CRF will also have an effect on the cell's response. This influence is called non-classical receptive field nCRF (non-Classical Receptive Field) inhibition, also known as SS (Surrounding Suppression). It can cause V1 neurons to have a weaker or no response to the edge of an object, but a strong response to the edge of texture.

[0004] The above neurophysiological phenomena show good correlation with the results of certain psycho-physical experiments, which show that the perception of a directional stimulus, such as a line, can be affected by the presence of other such stimuli (distractors) in its vicinity. Figure 1 This illustrates the suppression of contour perception by gratings. Contours are most likely to "disappear" in gratings when the gratings consist of elements that are similar to the contours.

[0005] Inspired by the outer region suppression, computer algorithm researchers have proposed many computational models to introduce the outer region suppression of nCRF to make up for the shortcomings of traditional local edge detectors. They have achieved many good results, but there are still many problems to be solved. These models are based on hand-designed feature extractors, such as using half-wave rectified differential Gaussian functions and butterfly-shaped modulation patterns to suppress the response of texture elements to simulate the outer region suppression effect. The suppression strength depends on the homogeneity between the CRF and nCRF regions. In early work, surround homogeneity was mainly obtained by calculating directional similarity. Due to the complexity of the visual environment, contour perception is the result of the synergy of characteristics such as direction, color and scale. In order to obtain better suppression effects, more algorithms combine various visual signals (such as brightness, color and sparsity) to calculate homogeneity. These nCRF-based contour models can significantly reduce texture edges, selectively retain significant structures, and improve the detection results of local edge operators. However, using a fixed feature extractor to simulate the diversity and complexity of neural interactions is obviously lacking, and neurons will not evolve according to subjective visual tasks.

[0006] In recent years, with the revival of deep learning, edge detection algorithms have been re-understood, injected with a lot of new vitality, and achieved more advanced results. Deep CNN (Convolutional Neural Networks) has a very excellent ability to automatically learn rich data representations at abstract levels, which has brought great progress to the research of computer vision tasks including edge detection. External area inhibition belongs to the category of neurophysiology and psychophysics, and deep learning is also a discipline based on biological characteristics. However, there are few studies on edge detection algorithms that combine the external area inhibition effect with modern deep learning networks. As far as we know, there is only one work that combines the external area modulation mechanism with deep learning, that is, the learnable non-classical receptive field modulation contour detection proposed by Tang Qilin et al., which uses a square ring convolution layer to simulate external area inhibition. However, its algorithm model is relatively simple, the number of parameters is large, the inference speed is very slow, and the experimental accuracy is relatively low.

[0007] From the perspective of practical applications of edge detection, as a low-level image processing task that is often used as the underlying layer of high-level visual tasks, edge detection algorithms should be designed to be efficient, compact and accurate. Summary of the invention

[0008] In response to the problems existing in the above-mentioned prior art, the present invention provides a bionic target contour extraction algorithm based on deep learning. By establishing an edge detection model inspired by biological characteristics and combining feature information from the local receptive field with background information from the surrounding environment, efficient and lightweight edge extraction is achieved in complex scenes.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] A bionic target contour extraction algorithm based on deep learning, which combines the outer region suppression mechanism with the deep CNN framework to establish an edge detection model inspired by biological characteristics, and uses the edge detection model inspired by biological characteristics to extract the edge of the target image;

[0011] The biologically inspired edge detection model consists of a backbone network, a side output layer, and an external region suppression module (SSM);

[0012] The backbone network includes multiple stages based on the CNN framework, and a maximum pooling layer is configured between every two stages for downsampling, wherein the first stage is configured as an initial convolution layer and three residual convolution blocks, and the remaining stages are configured as four residual convolution blocks;

[0013] The external area suppression module SSM is configured as follows: a plurality of convolutional layers with a convolution kernel size of 3x3 are connected in series to construct a convolutional block with a receptive field of a specified scale, and a feature map obtained by the convolutional block is subtracted from a feature map of a lower convolutional layer in the convolutional block to form a square ring receptive field simulating nCRF; the external area suppression module SSM is applied to the initial convolutional layer and each residual convolutional block of the backbone network to perform external area suppression modulation to generate a feature map response, wherein the scales of the square ring receptive field of the external area suppression module SSM corresponding to different stages of the backbone network are different to simulate the response of V1 neurons to features of different scales;

[0014] The feature map responses generated by the external area inhibition modulation of each stage in the side output layer are accumulated using the eltwise layer to obtain mixed features, and a feature map corresponding to each stage is obtained through a 1×1-1 conv layer; then the feature maps obtained in other stages except the first stage are upsampled using bilinear interpolation to make the resolution the same as the input image, and the cross entropy loss / sigmoid layer is connected to the upsampling layer to obtain the loss of each stage; all upsampling layers and the conv layer of the first stage are concatenated, and then a 1×1-1 conv layer is used to fuse the feature maps of each stage, and finally the fusion loss / output is obtained through the cross entropy loss / sigmoid layer.

[0015] Specifically, each residual convolution block in the backbone network sequentially includes a 3x3 convolution layer, a ReLU layer, and a 1x1 convolution layer, and the output of the residual convolution block is obtained by adding the convolution result of the 1x1 convolution layer and the feature map input to the residual convolution block.

[0016] Specifically, the backbone network is configured into four stages, and the numbers of channels in the four stages are configured to be 24, 48, 96, and 96 respectively.

[0017] Specifically, the scales of the square ring receptive field of the outer region suppression module SSM include: 9x9 / 3x3, 15x15 / 5x5, 21x21 / 7x7, wherein 9x9, 15x15, 21x21 represent the receptive field scales of the convolutional blocks constructed by connecting multiple convolutional layers in series, and 3x3, 5x5, 7x7 represent the receptive field scales of the small convolutional blocks as CRF.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention achieves a better balance between edge detection accuracy and efficiency by designing an efficient backbone network, and cleverly designs an outer region suppression module SSM with a multi-layer convolutional layer stacking structure to increase the receptive field, effectively simulating the outer region suppression effect, and using SSMs of receptive fields of different sizes to characterize spatial structures at different scales. After the SSM modulation operation, the network can integrate important information over a larger field of view. According to the test, the model network of the present invention does not require pre-trained weights to assist in training, the network parameters are reduced to 1.7M, and the inference speed on the Tesla V100 32GB GPU reaches 44 pictures per second. Compared with the prior art, the present invention is more efficient and lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the extracellular field inhibition effect in psychophysics.

[0021] Figure 2 Schematic diagram of nCRF surround suppression.

[0022] Figure 3 Schematic diagram of outer area suppression of traditional edge detection algorithm.

[0023] Figure 4 This is a schematic diagram of the model network framework of an embodiment of the present invention.

[0024] Figure 5 Schematic diagram of the network structure of the residual convolution block in an embodiment of the present invention.

[0025] Figure 6 Schematic diagram of a network structure of an external zone suppression module SSM in an embodiment of the present invention.

[0026] Figure 7 FIG. 1 is another schematic diagram of a network structure of an external zone suppression module SSM in an embodiment of the present invention.

[0027] Figure 8 It is another network structure diagram of the external zone suppression module SSM in the embodiment of the present invention.

[0028] Fig. 9 This is an algorithm network framework diagram in the prior art. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

[0030] Example

[0031] like Figures 4 to 8 As shown in the figure, the bionic target contour extraction algorithm based on deep learning combines the outer region suppression mechanism with the deep CNN framework to establish an edge detection model inspired by biological characteristics, and uses the edge detection model inspired by biological characteristics to extract the edge of the target image. The edge detection model inspired by biological characteristics includes a backbone network, a side output layer, and an outer region suppression module SSM.

[0032] Efficient backbone network:

[0033] First, we use the four stages of the convolutional neural network (CNN) framework, stage_K, where K = 1, 2, 3, 4. Figure 4 As shown, a maximum pooling layer 2x2MaxPool is configured between each two stages for downsampling. The first stage uses an initial convolution layer init_conv and three residual convolution blocks Block_1_1, Block_1_2, and Block_1_3. The initial convolution layer is used to convolve the three-channel image into feature maps with more channels. The other three stages use four residual convolution blocks. Each residual convolution block contains a 3x3 convolution layer, a ReLU layer, and a 1x1 convolution layer in sequence. The convolution result of the 1x1 convolution layer and the feature map input to the residual convolution block are added to obtain the output of the residual convolution block, as shown in Figure 5 In order to achieve a better balance between accuracy and efficiency, the number of channels in the four stages is configured as 24, 48, 96, and 96 respectively.

[0034] External Zone Suppression Module SSM:

[0035] The neurophysiological and psychophysical principles of the extracellular field inhibition effect have been briefly introduced in the background section of this article. Figure 2 As shown in Figure 2, the surrounding area outside the CRF is defined as the nCRF suppression surround. The outer region suppression response is obtained through this area.

[0036] The traditional edge detection algorithm inspired by outer region suppression proposed by Cosmin et al. uses the Gabor energy operator to simulate the outer region suppression effect. The principle of this algorithm can be briefly summarized as follows: Figure 3As shown, the CRF center responses of points A and B are the same, and the outer region suppression responses of the two points are calculated by integrating their nCRF suppression surrounds. At point A, the suppression response is very small, while at point B, this suppression response becomes quite strong. Subtract the CRF response from the nCRF suppression response to obtain the final edge response. The operator implemented by this algorithm responds strongly to isolated lines, edges, and contours, but has a weak or no response to some texture edges.

[0037] In convolutional neural networks, the definition of receptive field is the size of the area on the input image where the pixels on the feature map output by each layer of the convolutional neural network are mapped. A more popular explanation is that a point on the feature map corresponds to an area on the input image. The calculation method for the size of the receptive field is to calculate from the last layer downwards, that is, first calculate the receptive field of the deepest layer on the previous layer, and then pass it to the first layer layer by layer. The formula used can be expressed as follows:

[0038] RF i =(RF i+1 -1)×stride i +Ksize i (1)

[0039] Among them, RF i is the receptive field of the i-th convolutional layer, RF i+1 is the receptive field on the i+1 layer, stride is the step size of the convolution, and Ksize is the size of the convolution kernel of this layer.

[0040] In this embodiment, the convolution layers used in SSM have a step size of 1 and a convolution kernel size of 3. Therefore, the receptive field increases by 2 for each additional convolution layer.

[0041] The first external region suppression module SMM_1 designed in this embodiment is as follows Figure 6 As shown, the modulation effect of nCRF on the feature map is simulated. Evidence from neurophysiological findings shows that the range of nCRF is usually 2-5 times that of CRF. In this embodiment, the ratio of nCRF to CRF is set to 3. For the outer region suppression module SMM_1, four convolutional layers with a convolution kernel size of 3x3 are connected in series to construct a convolution block with a receptive field of 9×9. The result obtained by the fourth layer is subtracted from the result obtained by the first layer of convolution to form a square ring receptive field module with a large receptive field of 9×9 minus a small receptive field of 3x3.

[0042] In order to simulate the response of V1 neurons to features of different scales, we expand the receptive field of SSM by stacking more convolutional layers. Figure 7 and Figure 8As shown in the figure, the principle of constructing SSM_1 is the same as that of constructing SSM_2 and SSM_3, respectively, forming a 15x15 / 5x5 and 21x21 / 7x7 square ring receptive field module. The visual field information is extracted from fine to coarse. The fine scale is conducive to capturing high-frequency details, while the coarse scale can expand the range of the receptive field. The multi-scale scheme can further improve the contour detection results.

[0043] Modulation characteristics of the jth channel of SSM_n (n=1,2,3) Calculated by the following formulas:

[0044]

[0045]

[0046]

[0047]

[0048]

[0049] In the above formula, SSM_n, n = 1, 2, 3 represents SSM of different scales. represents the characteristic response of the jth channel of the mth convolutional layer of SSM, 2≤m≤10, represents the characteristic response of the i-th channel of CRF, Represents the input feature map The jth convolution kernel on the i-th feature channel of Represents the input feature map The jth convolution kernel on the i-th feature channel of and represents the corresponding bias term.

[0050] The j-th channel response feature of CRF Calculated by the following formula:

[0051]

[0052] In the formula, I i represents the i-th channel feature response of the input feature map I, represents the jth convolution kernel on the i-th feature channel of the input feature map I, represents the corresponding bias term.

[0053] Since the CRF kernel is similar to the convolution kernel in CNN, it can selectively respond to different features of local information within the receptive field. Therefore, we apply SSM to the feature map responses generated by the initial convolution layer of the backbone network and each residual convolution block to perform out-of-region suppression modulation operations.

[0054] Compared with the single function of the backbone network, the SSM modulation operation can integrate important information over a larger field of view.

[0055] Efficient side output layer:

[0056] In order to learn rich feature representation, the side output structure in RCF (Richer Convolutional Features) is used. Each block of the backbone network is connected to the external region suppression module SSM. The external region suppression modulation effect is achieved by subtracting the established characteristics of CRF from the modulation characteristics of nCRF, and then performing a ReLU operation. The formula for the modulation response is as follows:

[0057]

[0058] In the formula, ReLU represents the activation function, represents the characteristic modulation result of the jth channel of the outer region suppression modulation, represents the j-th channel response feature of CRF, Represents the modulation characteristics of the outer zone suppression module SSM on the j-th channel of the backbone network.

[0059] The modulation response features of the four stages are accumulated using eltwise layers to obtain mixed features. Each of the four eltwise layers is followed by a 1×1-1 conv layer to obtain a feature map. Then, bilinear interpolation is used to upsample the feature maps obtained in the last three stages so that the resolution becomes the same as the input image. In the last three stages, a cross entropy loss / sigmoid layer is connected to the upsampling layer to obtain the loss of each stage. All upsampling layers and the 1x1-1conv layer of the first stage are concatenated. Then a 1×1-1 conv layer is used to fuse the feature maps of each stage. Finally, the fused loss / output is obtained through the cross entropy loss / sigmoid layer.

[0060] Loss function:

[0061] Because there are far fewer edge pixels than non-edge pixels in the image, we use a weighted cross entropy loss function. The following is the prediction loss formula for a single pixel:

[0062]

[0063] Among them, y i represents the i-th pixel value of the true value label edge map, X irepresents the activation value of the i-th pixel of the input feature map, W represents the weight parameter, which is obtained by training and learning in the network. P() is the sigmoid function, and th is the threshold. By setting th to 64, some weak edges are ignored, reducing the possible interference to the network.

[0064]

[0065]

[0066] Y + and Y - Represent the number of edge and non-edge real pixels respectively, and Y represents the total number of pixels. The parameter γ is used to balance the relationship between edge and non-edge real pixels. Here, we use γ = 1.1 to highlight the role of edge point pixels.

[0067] This loss function is applied to all side outputs and fusion outputs.

[0068] Therefore, the total loss function is calculated as follows:

[0069]

[0070] Where K represents the number of stages (K=4), Y represents the total number of pixels, represents the activation value of the i-th pixel in the response map of the k-th stage, Represents the activation value of the i-th pixel of the fused feature edge map obtained by the network.

[0071] like Fig. 9 As shown in the figure, the edge detection algorithm of combining outer area suppression with deep learning proposed by Tang Qilin et al. in 2019 used convolution layers with convolution kernels of 5x5 and 7x7 when defining its network structure. It also customized a convolution layer with a square ring convolution kernel, whose sizes were 5 / 15 and 7 / 21, respectively, which means that the outer sizes of the convolution kernels were 15 and 21, and the inner sizes were 5 and 7, respectively. This network model has several defects: 1. Its convolution kernel is too large, so its network parameters are much larger than those formed by the conventional 3x3 convolution layer, and it also causes the network reasoning speed to be very slow. Experiments on Tesla V100 32GB GPU found that it can only process 7 pictures in 1 second. 2. The square ring convolution layer is difficult to implement through the conventional deep learning framework, and the network is difficult to define. 3. The network has only two layers, so it lacks the ability to learn advanced features, so the accuracy obtained in the experiment is relatively low. 4. The accuracy is increased by using the multi-resolution image pyramid method, but this method greatly increases the number of network parameters, and the increased accuracy is limited. 5. Its CRF layer requires the pre-trained SAE (Sparse Autoencoder) weight parameters.

[0072] The present invention proposes a new bionic target contour extraction algorithm based on deep learning, which constructs an efficient edge detection network, which consists of a backbone, a side output layer and an outer area suppression module. The network does not require pre-trained weights to assist in training. After testing, its network parameters were reduced to 1.7M, and the inference speed on the Tesla V100 32GB GPU reached 44 pictures per second. In comparison, the present invention is more efficient and lightweight.

[0073] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and performing non-creative work on this basis should fall within the protection scope of the present invention.

Claims

1. A bionic target contour extraction algorithm based on deep learning, characterized in that: By combining the external region suppression mechanism with the deep CNN framework, an edge detection model inspired by biological characteristics is established, and the edge of the target image is extracted using this edge detection model inspired by biological characteristics. The biologically inspired edge detection model consists of a backbone network, a side output layer, and an external region suppression module (SSM); The backbone network includes multiple stages based on the CNN framework, and a maximum pooling layer is configured between every two stages for downsampling, wherein the first stage is configured as an initial convolution layer and three residual convolution blocks, and the remaining stages are configured as four residual convolution blocks; The external area suppression module SSM is configured as follows: a plurality of convolutional layers with a convolution kernel size of 3x3 are connected in series to construct a convolutional block with a receptive field of a specified scale, and a feature map obtained by the convolutional block is subtracted from a feature map of a lower convolutional layer in the convolutional block to form a square ring receptive field simulating nCRF; the external area suppression module SSM is applied to the initial convolutional layer and each residual convolutional block of the backbone network to perform external area suppression modulation to generate a feature map response, wherein the scales of the square ring receptive field of the external area suppression module SSM corresponding to different stages of the backbone network are different to simulate the response of V1 neurons to features of different scales; The feature map responses generated by the external area inhibition modulation of each stage in the side output layer are accumulated using the eltwise layer to obtain mixed features, and a feature map corresponding to each stage is obtained through a 1×1-1 conv layer; then the feature maps obtained in other stages except the first stage are upsampled using bilinear interpolation to make the resolution the same as the input image, and the cross entropy loss / sigmoid layer is connected to the upsampling layer to obtain the loss of each stage; all upsampling layers and the conv layer of the first stage are concatenated, and then a 1×1-1 conv layer is used to fuse the feature maps of each stage, and finally the fusion loss / output is obtained through the cross entropy loss / sigmoid layer.

2. The bionic target contour extraction algorithm based on deep learning according to claim 1 is characterized in that: Each residual convolution block in the backbone network sequentially includes a 3x3 convolution layer, a ReLU layer, and a 1x1 convolution layer, and the output of the residual convolution block is obtained by adding the convolution result of the 1x1 convolution layer and the feature map input to the residual convolution block.

3. The bionic target contour extraction algorithm based on deep learning according to claim 1 is characterized in that: The backbone network is configured as four stages, and the numbers of channels in the four stages are configured as 24, 48, 96, and 96 respectively.

4. The bionic target contour extraction algorithm based on deep learning according to any one of claims 1 to 3, characterized in that: The scales of the square ring receptive field of the outer region suppression module SSM include: 9x9 / 3x3, 15x15 / 5x5, 21x21 / 7x7, wherein 9x9, 15x15, 21x21 represent the receptive field scales of the convolutional blocks constructed by connecting multiple convolutional layers in series, and 3x3, 5x5, 7x7 represent the receptive field scales of the small convolutional blocks as CRF.

5. The bionic target contour extraction algorithm based on deep learning according to claim 4 is characterized in that: The modulation characteristics of the outer zone suppression module SSM on the jth channel of the backbone network Calculated by the following formulas: In the above formula, SSM_n, n = 1, 2, 3 represents SSM of different scales. represents the characteristic response of the jth channel of the mth convolutional layer of SSM, 2≤m≤10, represents the characteristic response of the i-th channel of CRF, Represents the input feature map The jth convolution kernel on the i-th feature channel of Represents the input feature map The jth convolution kernel on the i-th feature channel of and Represents the corresponding bias term, the j-th channel response feature of CRF Calculated by the following formula: In the formula, I i represents the i-th channel feature response of the input feature map I, represents the jth convolution kernel on the i-th feature channel of the input feature map I, represents the corresponding bias term.

6. The bionic target contour extraction algorithm based on deep learning according to claim 5 is characterized in that: The process of performing external region inhibition modulation to generate characteristic graph response is as follows: By subtracting the established characteristics of CRF from the modulation characteristics of nCRF and then performing the ReLU operation, the formula is as follows: In the formula, ReLU represents the activation function, represents the characteristic modulation result of the jth channel of the outer region suppression modulation, represents the j-th channel response feature of CRF, Represents the modulation characteristics of the outer zone suppression module SSM on the j-th channel of the backbone network.

7. The bionic target contour extraction algorithm based on deep learning according to claim 6 is characterized in that: The cross entropy loss / sigmoid layer uses a weighted cross entropy loss function, and the prediction loss formula for a single pixel is: In the formula, y i represents the i-th pixel value of the true value label edge map, X i represents the activation value of the i-th pixel of the input feature map, W represents the weight parameter, P() represents the sigmoid function, and th represents the threshold value. Where Y + and Y - They represent the number of edge and non-edge real pixels respectively, Y represents the total number of pixels, and γ represents the setting parameter used to balance the relationship between edge and non-edge real pixels. The total loss function is expressed as follows: In the formula, K represents the number of stages, Y represents the total number of pixels, represents the activation value of the i-th pixel in the response map of the k-th stage, Represents the activation value of the i-th pixel of the fused feature edge map obtained by the network.

Images