A Transformer Image Road Detection Method
Through the Transformer image road detection method, multi-layer coded blocks and space-time Transformer model are used, combined with timeline information, and the problem of incomplete establishment of information loss and spatial dependency in the existing technology is solved, and the accuracy of SAR image road detection is improved.
Patent Information
- Application Number
- CN202210346629.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-03-31
AI Technical Summary
The existing multi-time phase SAR image road detection method uses multi-time phase information in the preprocessing stage, ignores the time consistency of road objects, resulting in information loss, and cannot establish a spatial dependence between the pixels and pixels, affecting the detection effect. .
The Transformer image road detection method is adopted to extract features by constructing multi-layer coded blocks, combining the time axis for matrix transformation, input into the spatiotemporal Transformer model, obtain enhanced features, and realize road detection through segmented networks.
Make full use of multi-time phase information to establish a spatial dependence between the pixels of the whole image to realize accurate detection of the image path, and overcome the problem of low detection accuracy of traditional methods in complex environments.
Smart Images
Figure CN114937257B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of SAR image road detection, and in particular to a Transformer image road detection method. Background Art
[0002] In the early days, SAR road detection methods modeled roads as dark areas surrounded by bright edges. Based on this premise, early SAR image road detection methods first extracted features from SAR images based on the characteristics of the roads in the SAR images. The extracted features include: linear features, contrast features, grayscale features, and directional features. Then, the extracted features are binarized to detect the roads in the SAR images.
[0003] However, in the real world, due to the existence of various interferences, the diversity of road types and the complexity of the environmental background, the established road models are often invalid, which creates a bottleneck for traditional SAR image road extraction methods.
[0004] An important factor affecting the performance improvement of existing SAR image road detection methods is speckle noise. In order to reduce the impact of speckle noise, existing single-phase road detection methods usually use various spatial filtering methods to eliminate speckle noise in the preprocessing stage. Although the spatial filtering method can eliminate speckle noise to a certain extent, it also causes a decrease in spatial resolution. Compared with optical images, SAR images have the characteristics of all-day and all-weather. In recent years, the revisit period of SAR satellites has become shorter and shorter, and multiple images of the same area can be taken in a very short period of time. Therefore, many methods using multi-phase SAR images for road detection have emerged in recent years. These methods mainly use various temporal filtering methods in the preprocessing stage to remove speckle noise, and still only use single-view images in the road detection stage. Such methods ignore the characteristics of road objects. In remote sensing images, roads are objects that do not change easily. In a very short period of time, the road objects in SAR images taken at different times are consistent. Therefore, the method of using only multi-phase information in the preprocessing stage causes information loss. In addition to temporal information, spatial information is also very important for detecting roads from SAR images. In order to obtain more spatial information, most methods use methods such as dilated convolution and spatial pyramid model to obtain more spatial information by expanding the receptive field. However, such methods cannot establish the spatial dependency of all image pixels.
[0005] In summary, existing multi-temporal SAR image road detection methods only use multi-temporal information in the preprocessing stage, ignoring the temporal consistency of roads in SAR images, resulting in information loss. Most existing methods use dilated convolution and spatial pyramid models to obtain more spatial information, but these methods cannot obtain the spatial dependency relationship between pixels in the entire image.
[0006] Therefore, there is an urgent need for an image road detection method that can make full use of multi-temporal information and establish the spatial dependency relationship between pixels in the entire image. Summary of the invention
[0007] In view of this, the present invention provides a Transformer image road detection method, which can make full use of multi-temporal information and establish spatial dependency between pixels of the entire image to achieve accurate detection of image roads.
[0008] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is:
[0009] A Transformer image road detection method obtains multi-temporal SAR images from satellites as original images and performs road detection using the following specific steps:
[0010] Step 1: Construct a multi-layer coding block and use the coding block to extract the features of the original image in layers.
[0011] Step 2: Combine the extracted features of each layer according to the time axis and perform matrix transformation to obtain the total image features of each layer corresponding to each layer of coding blocks.
[0012] Step 3: Input the total image features of each layer into the spatiotemporal Transformer model to obtain the model features of each layer; add the model features of each layer to the total image features of the same layer to obtain the enhanced features of the layer.
[0013] Step 4: decode the enhanced features of each layer, and superimpose all the encoding results from back to front in the encoding order to obtain the restored features, and input the restored features into the segmentation network; the segmentation network segments the restored features to obtain the road detection results.
[0014] Furthermore, the features of the original image are extracted hierarchically using the coding blocks, and the extracted features of each layer are combined according to the time axis and matrix transformed to obtain the total image features of each layer corresponding to each coding block. The specific method is as follows:
[0015] The multi-layer coding block includes a total of four layers of coding blocks.
[0016] Images A, B and C at three adjacent moments are taken from the original image as input and respectively input into the first-layer encoding block; the first-layer encoding block convolves the image to obtain the first-layer features A, B and C; the first-layer features A, B and C are combined along the time axis and matrix transformed, and the result is used as the first-layer total image feature.
[0017] The first layer features A, B and C are respectively input into the second layer coding block to obtain the second layer features A, B and C; the second layer features A, B and C are combined according to the time axis, and matrix transformation is performed, and the result is used as the second layer total image feature.
[0018] The second layer features A, B and C are respectively input into the third layer coding block to obtain the third layer features A, B and C; the third layer features A, B and C are combined along the time axis and matrix transformed, and the result is used as the third layer total image feature.
[0019] The third layer features A, B and C are respectively input into the fourth layer coding block to obtain the fourth layer features A, B and C; the fourth layer features A, B and C are combined along the time axis and matrix transformed, and the result is used as the fourth layer total image feature.
[0020] Furthermore, the first layer encoding block is 64 3×3 convolution blocks, the second layer encoding block is 128 3×3 convolution blocks, the third layer encoding block is 256 3×3 convolution blocks, and the fourth layer encoding block is 512 3×3 convolution blocks.
[0021] Furthermore, the number of images is at least 3.
[0022] Furthermore, the total image features of each layer are input into the spatiotemporal Transformer model to obtain the model features of each layer, specifically:
[0023] The total image features of each layer are input into the spatiotemporal Transformer model of the corresponding layer. Each spatiotemporal Transformer model contains three scalar dot product attention modules. The total image features of each layer are input into three scalar dot product attention modules. The specific method is as follows:
[0024] The total image features of the first layer are linearly transformed Q to obtain the first layer Q linear transformation matrix Q i , Q i Input to the first scalar dot multiplication attention module; the total image features of the first layer are linearly transformed K to obtain the first layer K linear transformation matrix Ki , K i Input to the second scalar dot multiplication attention module; the total image features of the first layer are linearly transformed V to obtain the first layer V linear transformation matrix V i , V i Input to the 3rd scalar dot product attention module.
[0025] In the scalar dot product attention module, the attention coefficient A is calculated based on the first layer Q linear transformation matrix and the first layer K linear transformation matrix i ; Attention coefficient A i And the first layer V linear transformation matrix V i Multiply them to get the output headi; where i is the serial number of the scalar dot product attention module, i = 1, 2, 3.
[0026] After obtaining all the output headi of each layer, all the headi of this layer are combined and linearly transformed to obtain the model feature MultiHeadi(Q,K,V) of this layer.
[0027] Furthermore, the segmentation network consists of a 1×1 convolutional layer and a softmax layer.
[0028] Furthermore, the size of the road detection result is the size of the original image.
[0029] Beneficial effects:
[0030] 1. The present invention proposes a Transformer image road detection method. After the total image features are obtained by layered extraction, the total image features of each layer are input into the spatiotemporal Transformer model to obtain the model features of each layer, and then the enhanced features are obtained; after decoding, all decoding results are stacked from back to front to obtain the restored features; the segmentation network segments the restored features to obtain the road detection results. The present invention can make full use of multi-phase information and can establish the spatial dependency relationship between pixels of the entire image to achieve accurate detection of image roads.
[0031] 2. The method of the present invention is based on deep learning, which overcomes the shortcomings of traditional SAR image road detection methods that cannot accurately model roads in SAR images due to the complex environment in SAR images.
[0032] 3. The method of the present invention is based on deep learning and adopts a data-driven approach to learn road objects in SAR images, which effectively avoids modeling of objects in SAR images and achieves better results than traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 Schematic diagram of the method and system of the present invention.
[0034] Figure 2 This is the principle block diagram of the Transformer model. DETAILED DESCRIPTION
[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0036] like Figure 1 As shown, the present invention proposes a Transformer image road detection method. Transformer was first used in the field of natural language processing to model long-range contextual relationships. Recently, Transformer has achieved great success in the field of computer vision. In the present invention, the spatiotemporal Transformer module can make full use of multi-temporal information to establish spatial dependencies between pixels in the entire image.
[0037] like Figure 1 As shown, the input is a multi-temporal SAR image {I0,...,I c ,...,I T-1}, the output is I c The whole network structure is U-shaped. The multi-phase SAR image is firstly extracted through a shared coding block. Each coding block consists of multiple residual blocks and downsampling operations. The features of the multi-phase SAR image {F0,...,F c ,...,F T-1}Combined by time axis and then matrix transformed to get the input of spatiotemporal Transformer module as F IN ∈R (THW)×C It is a two-dimensional matrix with the first dimension of size T×H×W, where T is the number of multi-temporal images, H and W are the length and width of each original multi-temporal feature, and the second dimension is the number of channels C of the input feature.
[0038] The present invention aims at the multi-temporal SAR image road detection task, obtains the multi-temporal SAR image as the original image through the satellite (any set of images, maps or geographical data of the same area obtained at different times can be regarded as "multi-temporal" data), and performs road detection in the following specific steps:
[0039] Step 1: Construct a multi-layer coding block and use the coding block to extract the features of the original image in layers. The image is a multi-phase SAR image of the same area, and the number is at least 3.
[0040] In the embodiment of the present invention, the multi-layer coding block includes a total of four layers of coding blocks.
[0041] Images A, B and C at three adjacent moments are taken from the original image as input and respectively input into the first-layer coding block; the first-layer coding block performs convolution on the image to obtain first-layer features A, B and C; the first-layer features A, B and C are combined according to the time axis and matrix transformed, and the result is used as the first-layer total image feature;
[0042] The first layer features A, B and C are input into the second layer coding block respectively to obtain the second layer features A, B and C; the second layer features A, B and C are combined according to the time axis, and matrix transformation is performed, and the result is used as the second layer total image feature;
[0043] The second layer feature A, the second layer feature B and the second layer feature C are respectively input into the third layer coding block to obtain the third layer feature A, the third layer feature B and the third layer feature C; the third layer feature A, the third layer feature B and the third layer feature C are combined according to the time axis, and matrix transformation is performed, and the result is used as the third layer total image feature;
[0044] The third layer features A, B and C are respectively input into the fourth layer coding block to obtain the fourth layer features A, B and C; the fourth layer features A, B and C are combined along the time axis and matrix transformed, and the result is used as the fourth layer total image feature.
[0045] In the embodiment of the present invention, the first layer coding block is 64 3×3 convolution blocks, the second layer coding block is 128 3×3 convolution blocks, the third layer coding block is 256 3×3 convolution blocks, and the fourth layer coding block is 512 3×3 convolution blocks.
[0046] Step 2: Combine the extracted features of each layer according to the time axis and perform matrix transformation to obtain the total image features of each layer corresponding to each layer of coding blocks.
[0047] Step 3: Input the total image features of each layer into the spatiotemporal Transformer model to obtain the model features of each layer; add the model features of each layer to the total image features of the same layer to obtain the enhanced features of the layer.
[0048] like Figure 2 As shown, the total image features M of each layer are respectively input into the spatiotemporal Transformer model of the corresponding layer. Each spatiotemporal Transformer model contains three scalar dot product attention modules. The total image features M of each layer are respectively input into three scalar dot product attention modules. The specific method is as follows:
[0049] The total image features of the first layer are linearly transformed Q to obtain the first layer Q linear transformation matrix Q i , Q i Input to the first scalar dot multiplication attention module; the total image features of the first layer are linearly transformed K to obtain the first layer K linear transformation matrix K i , K i Input to the second scalar dot multiplication attention module; the total image features of the first layer are linearly transformed V to obtain the first layer V linear transformation matrix V i , V i Input to the third scalar dot product attention module; in the scalar dot product attention module, the attention coefficient A is calculated according to the first layer Q linear transformation matrix and the first layer K linear transformation matrix i ; Attention coefficient A i And the first layer V linear transformation matrix V i Multiply them together to get the output headi. Where i is the serial number of the scalar dot product attention module, i = 1, 2, 3.
[0050] After obtaining all the output headi of each layer, all the headi of this layer are combined and linearly transformed to obtain the model feature MultiHeadi(Q,K,V) of this layer.
[0051] Figure 2 The block diagram of the spatiotemporal Transformer, Q, K, and V are the same value. According to the calculation rules of the Transformer module, a multi-head attention mechanism is adopted, and the input of each scalar dot product attention module is a linear mapping of Q, K, and V. Taking the i-th scalar dot product attention module as an example, the input Q, K, and V are firstly linearly mapped to obtain the input Q of the i-th scalar dot product attention module. i =Q×M i , K i =K×M i , V i =V×M i . M i is the total image feature corresponding to the i-th scalar point.
[0052] In the scalar dot product attention module, we first need to use Q i and K i To calculate the attention coefficient A i .
[0053]
[0054] Among them, D k is the number of channels after each layer transformation, and k is the number of layers.
[0055] The output headi of the scalar dot product attention module is:
[0056] head i =A i ×V i
[0057] like Figure 2 As shown, the output of each point multiplication attention module is combined and then subjected to a linear transformation to obtain the output MultiHeadi(Q, K, V) of the spatiotemporal Transformer, which is:
[0058] MultiHead(Q,K,V)=W O Concat(head1,....,head T )
[0059] Among them, W O ∈R HW×THWT , is the restored feature of each layer of the output.
[0060] The model features of each layer are added to the total image features of the same layer to obtain the enhanced features of the layer.
[0061] Step 4: Decode the enhanced features of each layer, and stack all the encoding results from back to front in the encoding order to obtain the restored features, and input the restored features into the segmentation network; the segmentation network segments the restored features to obtain the road detection results, and the size of the road detection results is the size of the original image. The segmentation network includes a 1×1 convolutional layer and a Softmax layer.
[0062] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A Transformer image road detection method, It is characterized in that The multi-temporal SAR images are obtained through satellites as the original images, and the road detection is performed using the following specific steps: Step 1: Construct a multi-layer coding block and use the coding block to extract the features of the original image in layers; Step 2: Combine the extracted features of each layer according to the time axis and perform matrix transformation to obtain the total image features of each layer corresponding to each layer of coding blocks; Step 3: Input the total image features of each layer into the spatiotemporal Transformer model to obtain the model features of each layer; add the model features of each layer to the total image features of the same layer to obtain the enhanced features of the layer; Step 4: decode the enhanced features of each layer, and stack all the encoding results from back to front in the encoding order to obtain the restored features, and input the restored features into the segmentation network; The segmentation network segments and restores features to obtain road detection results; The method of extracting the features of the original image in layers by using coding blocks, combining the features of each layer extracted according to the time axis and performing matrix transformation, and obtaining the total image features of each layer corresponding to each coding block, is as follows: The multi-layer coding block includes a total of four layers of coding blocks; Take three images A, B and C at adjacent moments from the original image as input and input them into the first layer encoding block respectively; The first-layer encoding block convolves the image to obtain the first-layer features A, B, and C; the first-layer features A, B, and C are combined according to the time axis and matrix transformed, and the result is used as the first-layer total image feature; The first layer features A, B and C are input into the second layer coding block respectively to obtain the second layer features A, B and C; the second layer features A, B and C are combined according to the time axis, and matrix transformation is performed, and the result is used as the second layer total image feature; The second layer feature A, the second layer feature B and the second layer feature C are respectively input into the third layer coding block to obtain the third layer feature A, the third layer feature B and the third layer feature C; the third layer feature A, the third layer feature B and the third layer feature C are combined according to the time axis, and matrix transformation is performed, and the result is used as the third layer total image feature; The third layer features A, the third layer features B and the third layer features C are respectively input into the fourth layer coding block to obtain the fourth layer features A, the fourth layer features B and the fourth layer features C; the fourth layer features A, the fourth layer features B and the fourth layer features C are combined according to the time axis, and matrix transformation is performed, and the result is used as the fourth layer total image feature; The first layer encoding block is 64 3×3 convolution blocks, the second layer encoding block is 128 3×3 convolution blocks, the third layer encoding block is 256 3×3 convolution blocks, and the fourth layer encoding block is 512 3×3 convolution blocks; The total image features of each layer are input into the spatiotemporal Transformer model to obtain the model features of each layer, specifically: The total image features of each layer are input into the spatiotemporal Transformer model of the corresponding layer. Each spatiotemporal Transformer model contains three scalar dot product attention modules. The total image features of each layer are input into three scalar dot product attention modules. The specific method is as follows: The total image features of the first layer are linearly transformed Q to obtain the first layer Q linear transformation matrix ,Will Input to the first scalar dot product attention module; The total image features of the first layer are linearly transformed K to obtain the first layer K linear transformation matrix ,Will Input to the second scalar dot product attention module; The total image features of the first layer are linearly transformed V to obtain the linear transformation matrix of the first layer V ,Will Input to the third scalar dot product attention module; In the scalar dot product attention module, the attention coefficient is calculated according to the first layer Q linear transformation matrix and the first layer K linear transformation matrix ; Attention coefficient And the first layer V linear transformation matrix Multiply them together to get the output headi; among them, i is the serial number of the scalar dot product attention module, i =1,2,3; After obtaining all the output headi of each layer, all the headi of this layer are combined and linearly transformed to obtain the model feature MultiHeadi (Q, K, V) of this layer.
2. The method according to claim 1, It is characterized in that The number of the images is at least 3.
3. The method according to claim 1, It is characterized in that The segmentation network consists of a 1×1 convolutional layer and a softmax layer.
4. The method according to claim 1, It is characterized in that The size of the road detection result is the size of the original image.