A method for semantic segmentation and matching of images

Abstract

The application discloses a kind of semantic segmentation and matching method of image, it is related to image processing technical field.The application provides more accurate image semantic segmentation result for generating three-dimensional model by improving the semantic segmentation module of BGA-Net: along the direction of feature scale reduction, the pixel information of image gradually reduces, semantic information gradually increases, so the attention mechanism of multiple convolution branches is used to weight the first initial feature of different scales, and the pixel information in large-scale feature and the semantic information in small-scale feature are simultaneously focused on;The output of the previous scale decoder is added to the input of each convolution branch decoder, so that the pixel information in larger scale feature can be transmitted to the decoding process of smaller scale feature, the complement of global semantics and local details is realized, and more accurate semantic segmentation result is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for semantic segmentation and matching of images. Background Art

[0002] Semantic segmentation and dense matching are hot research areas. As classic single-task learning problems in vision, these two tasks have been studied for many years. With the rise of deep learning in artificial intelligence, many researchers have designed simple yet advanced networks to improve performance and have made significant progress in both tasks. In the process of generating 3D models from remote sensing imagery, semantic segmentation helps identify land types (such as forests, urban areas, and water bodies) in remote sensing images, providing the underlying structural information for subsequent 3D modeling. Dense matching establishes correspondences between segmented regions, estimates their spatial position and orientation, and reconstructs surface details by comparing and analyzing similarities between images. However, most existing methods address these two tasks separately or use a combination of multiple models. Due to a lack of sufficient information sharing and fusion, remote sensing images with large differences in ground features due to factors such as seasonal changes are difficult to process, making it difficult to generate high-precision 3D models.

[0003] In recent years, multi-task fusion networks for semantic segmentation and dense matching, such as the Bidirectional Guided Attention Network (BGA-Net), have been proposed. To bridge the gap between the semantic segmentation and dense matching tasks, BGA-Net first applies a shared unified backbone module to extract primary features from both images and generate two corresponding feature maps. A bidirectional guided attention module is then designed as a bridge between the semantic segmentation and dense matching tasks. This module aims to extract global image information from one of the two feature maps and generate an attention map. For the semantic segmentation task, this attention map assigns more weight to objects in the image (such as buildings, roads, and trees) and less weight to the background. For the dense matching task, since each object in the image is treated as a rigid surface and identical objects have the same or similar disparity, the attention map assigns more weight to the disparity of identical objects. Guided by the attention map, the semantic segmentation module and feature matching module in BGA-Net perform the semantic segmentation and dense matching tasks, respectively.

[0004] However, when performing semantic segmentation tasks, the semantic segmentation module pays more attention to the global context features of the image and ignores the image spatial detail information contained in the local context features, resulting in inaccurate segmentation of image details. Summary of the Invention

[0005] Based on this, in order to solve the technical problems in the prior art, the present application provides a semantic segmentation and matching method of images.

[0006] The present application provides a semantic segmentation and matching method of images, comprising:

[0007] The semantic segmentation module in the original BGA-Net network is replaced by an improved semantic segmentation module, which comprises a plurality of convolution branches, each of which comprises an attention mechanism module and a decoder module connected in sequence; the output end of the decoder in each convolution branch is connected with the input end of the decoder module in the next convolution branch in the direction of decreasing scale; the attention mechanism module comprises a dot product attention branch and a Conv-BN-ReLU branch in parallel; the feature matching module in the original BGA-Net network is replaced by an improved feature matching module, which comprises a convolution module, a first encoding-decoding module, a second encoding-decoding module, a third encoding-decoding module, a bilinear interpolation layer and a regression layer connected in sequence; a feature pyramid module is added between the original unified backbone module and the improved semantic segmentation module; an improved BGA-Net network comprising the unified backbone module, the feature pyramid module, the bidirectional guided attention module, the improved semantic segmentation module, the improved feature matching module and the bidirectional attention module in the original BGA-Net network is constructed.

[0008] Image data is collected to construct a data set, the improved BGA-Net network is trained using the data set, and a semantic segmentation-dense matching model is obtained.

[0009] The first image and the second image to be processed are input into the semantic segmentation-dense matching model, and preliminary feature extraction is performed on the first image and the second image through the unified backbone module to obtain first initial features of different scales and second initial features of different scales; the context feature extraction and weighting of the first initial features of different scales are performed through the bidirectional guided attention module to obtain a bidirectional attention map; the semantic information in the first initial features of different scales is enhanced through convolution operations of different scales and deconvolution operations of different scales in the feature pyramid module, so as to input the enhanced first initial features into the improved semantic segmentation module; the attention mechanism module of several convolution branches in the improved semantic segmentation module is used to extract context features of different scales and weight them. The first initial features of different scales are subjected to attention weighting operation to obtain semantic enhancement feature maps of different scales; the semantic enhancement feature maps of different scales are decoded by the decoder module to obtain decoding feature maps of different scales; wherein, the input of each decoder module also includes the output of the convolution branch decoder module from the previous scale; an initial segmentation map is obtained by adding a bidirectional attention map to the decoding feature map of the minimum scale; the second initial features of different scales are integrated into a cost body by the feature matching module, and the cost body and the bidirectional attention map are densely matched to obtain an initial disparity map; the initial segmentation map and the initial disparity map are cross-fused by the bidirectional attention module to obtain a final segmentation map and a final disparity map.

[0010] Furthermore, the specific steps of extracting and weighting context features in the bidirectionally guided attention module include:

[0011] After transforming the feature map of the input bidirectional guided attention module into the shape of H×W×F, the softmax function is applied to obtain the weight distribution;

[0012] The feature map with a shape of H×W×F is converted to F×H×W, and the feature map converted to F×H×W is weighted by the weight distribution output by the softmax function; and the weighted feature map is convolved;

[0013] The result of the convolution operation is fused with the feature map of the input bidirectional guided attention module to output a bidirectional attention map.

[0014] Furthermore, the steps of performing attention weighting operation by the attention mechanism module specifically include:

[0015] The dot product attention branch is used to transform the input of the attention mechanism module to obtain matrices Q, K, and V. The softmax function is used to calculate the normalized similarity weight of the element at each position in the input of the attention mechanism module relative to all other elements, and the similarity weight is multiplied by the matrix V to generate the self-attention weight matrix:

[0016]

[0017]

[0018]

[0019]

[0020] in, is the input of the attention mechanism module, C is the number of channels, is the weight matrix of self-attention, and the superscript T represents the transpose operation;

[0021] The first initial feature is sequentially convolved, batch normalized, and activated by the ReLU function through the Conv-BN-ReLU branch:

[0022] The output of the dot product attention branch is fused with the Conv-BN-ReLU branch to obtain the output of the attention mechanism module.

[0023] Furthermore, the specific steps of obtaining the initial disparity map include:

[0024] Performing a three-dimensional convolution operation on the cost volume through a convolution module; obtaining a first disparity map;

[0025] The first disparity map is gradually encoded and decoded by a first encoding-decoding module, a second encoding-decoding module, and a third encoding-decoding module. During the decoding process, dense matching is achieved by introducing a bidirectional attention map output by a bidirectional guided attention module to obtain a dense matching feature map.

[0026] The dense matching feature map is interpolated through a bilinear interpolation layer, and the interpolation result is regressed through a regression layer to output the initial disparity map.

[0027] Furthermore, the dense matching is achieved by introducing a bidirectional attention map output by the bidirectional guided attention module, which is achieved by fusing the bidirectional attention map with the output of the encoding layer in the encoding-decoding module and inputting the result into the decoding layer in the encoding-decoding module for decoding.

[0028] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0029] In the image semantic segmentation and matching method provided by the application, the semantic segmentation module of the BGA-Net is improved to obtain more rich local detail features, and more accurate image semantic segmentation results are provided for generating a three-dimensional model. Specifically, because the pixel information of the image gradually decreases and the semantic information gradually increases along the direction of feature scale reduction of the first initial feature of different scales, the attention mechanism of the multiple convolution branches can simultaneously pay attention to the pixel information in the large-scale feature and the semantic information in the small-scale feature by weighting the first initial feature of different scales; therefore, the output of the previous scale decoder is added to the input of each convolution branch decoder, so that the pixel information in the larger scale feature can be effectively transmitted to the decoding process of the smaller scale feature, the global semantic and the local detail are complementary, and more accurate image semantic segmentation results are obtained. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0031] Figure 1 The image semantic segmentation and matching method provided by the application is shown in the flowchart;

[0032] Figure 2 The original BGA-Net network structure provided by the application is shown in the schematic diagram;

[0033] Figure 3 The improved BGA-Net network structure provided by the application is shown in the schematic diagram;

[0034] Figure 4 The feature extraction module structure provided by the application is shown in the schematic diagram;

[0035] Figure 5 The GC block structure provided by the application is shown in the schematic diagram;

[0036] Figure 6 The improved semantic segmentation module structure provided by the application is shown in the schematic diagram;

[0037] Figure 7 The improved feature matching module structure provided by the application is shown in the schematic diagram;

[0038] Figure 8 The bidirectional attention module structure provided by the application is shown in the schematic diagram. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0040] Rao et al. proposed a bidirectional guided attention network for 3D semantic detection of remote sensing images: BGA-Net. The network structure is as follows Figure 2 As shown in Figure 1, it includes five modules: UBM (Unified Backbone Module), BGAM (Bidirectional Guided Attention Module), SSM (Semantic Segmentation Module), FMM (Feature Matching Module) and BFM (Bidirectional Fusion Module). The task of UBM is: First, the left and right images are respectively subjected to feature extraction through the same backbone network to obtain the feature map X. l and X r , these feature maps contain the low-level feature information of the image. Then, X l and X r Perform multi-scale feature fusion to extract richer semantic and contextual information. After UBM processing, the unified feature map X of the left and right images is obtained. l and X r .

[0041] The task of SSM is to: l and X r After the feature conversion, multi-scale features are extracted and information of different scales is fused to obtain the semantic feature map S. Then, the semantic segmentation attention map M' generated by BGAM is used to weight S to highlight the target object and obtain the activated feature map S'. Finally, the size of S' is restored to the input image size through upsampling and the initial segmentation map is generated through the convolution layer. The task of FMM is to: l and X rAfter feature transformation, the similarity of the left and right feature maps at different disparities is calculated to construct a cost volume V. The cost volume is then regularized using 3D convolution and NL blocks (Nonlocal blocks). The attention mechanism guides the matching process to obtain the regularized cost volume V'. Finally, a softmax operation is performed on V' to calculate the probability of each disparity, and the initial disparity map is obtained through weighted summation. The task of BFM is to: concatenate the initial segmentation map, the input image, and the initial disparity map, and fuse them through a residual learning network to obtain a refined segmentation map; and fuse the initial disparity maps to obtain a refined disparity map. Through the above modules, BGA-Net can simultaneously obtain the semantic segmentation map and disparity map of the image, and use the bidirectional attention mechanism and fusion module to improve the quality and robustness of the results.

[0042] SSM consists of three parts: feature conversion, spatial feature learning, and upsampling. The feature conversion part uses a 256-channel and a 128-channel 2D convolution layer to reduce the feature dimension. Then, a 128-channel residual block and a 3×3 filter are used to normalize the feature map. The spatial feature learning part first processes the output of the feature conversion part using 16 128-channel bottleneck blocks and then divides it into different sub-regions using SPP (Spatial Pyramid Pooling). Next, the hierarchical context information and the deep semantic feature map are connected and fused through a 128-channel and a 64-channel 2D convolution operation and a 3×3 filter to obtain a semantic feature map. The semantic feature map is then activated through the attention map to obtain an activated semantic feature map. Although SPP can capture features at different scales, the pooling operation reduces the spatial resolution of the feature map, which may lead to the loss of detailed information. In addition, the step of connecting hierarchical context information and deep semantic feature maps focuses more on combining global features and features at different scales rather than fine local information. Therefore, SSM mainly extracts global features. Although this operation helps to understand the overall structure, it may not be enough to retain the fine details of the image. In summary, the semantic segmentation module of BGANet focuses on global context information. Global context information is crucial for the semantic information of complex scenes, but local context information also plays a key role in preserving rich spatial details. In addition, the NL-block (Nonlocal Block) and GC block (Global Context block) are mainly used in the feature matching module to address the problem of long-distance information transmission and improve long-distance dependencies. However, the computational complexity is large, the memory consumption is large, and it is easy to cause information redundancy.

[0043] Based on this, the present invention improves BGA-Net by using the attention mechanism to increase the connection between semantic segmentation and dense matching tasks, optimizing the semantic information extraction module and comprehensively utilizing global and local context information for semantic information extraction; optimizing the efficiency of the feature matching module and using a simple and efficient stacked hourglass network for cost aggregation to improve the accuracy and efficiency of the multi-task learning network.

[0044] Example 1

[0045] Figure 1 The following is a flow chart of the semantic segmentation and matching method of the image in this embodiment. Figure 1 The method is described in detail and specifically comprises the following steps:

[0046] S1: Replace the semantic segmentation module in the original BGA-Net network with an improved semantic segmentation module, wherein the improved semantic segmentation module includes several convolution branches, each of which includes an attention mechanism module and a decoder module connected in sequence; the output end of the decoder in each convolution branch is connected to the input end of the decoder module in the next convolution branch along the direction of scale reduction; construct an improved BGA-Net network including the unified backbone module, bidirectional guided attention module, improved semantic segmentation module, feature matching module and bidirectional attention module in the original BGA-Net network.

[0047] Figure 3 The improved BGA-Net network structure constructed is shown, which is mainly composed of a feature extraction module, a bidirectional guided attention module, an improved semantic segmentation module, an improved feature matching module, and a bidirectional attention module. First, the feature extraction module is used to extract features from the left and right images respectively. Then, the left image feature is used to share global features with the bidirectional attention module to guide the learning of the semantic segmentation module and the feature matching module, outputting the initial semantic map and the initial disparity map. Finally, the bidirectional attention module is used to calculate the residual of the initial segmentation map and the disparity map, and refine the segmentation map and the disparity map to further improve the accuracy. More specifically:

[0048] Feature extraction module.

[0049] In order to increase the connection between dense matching and semantic segmentation modules, the present invention performs unified feature extraction on the left and right images, which mainly consists of two parts: unary feature extraction part and multi-scale feature fusion part, such as Figure 4As shown, a convolution operation is performed on the first image through a first convolution kernel, the size of the first convolution kernel is 3x3, the channel is 16, and the stride is 2; a convolution operation is performed on the output of the first convolution kernel through a first convolution block, the first convolution block includes three 3×3 convolution kernels, and the channels of the three 3×3 convolution kernels are 16, 16, and 32 respectively; a convolution operation is performed on the output of the first convolution block through a second convolution block, the second convolution block includes three 3×3 convolution kernels, and the channels of the three 3×3 convolution kernels are 16, 16, and 32 respectively. The channels of the 3×3 convolution kernels are all 32; the output of the second convolution block is convolved by the third convolution block, and the third convolution block includes 16 3×3 convolution kernels with 32 channels, 16 3×3 convolution kernels with 32 channels, 16 1×1 convolution kernels with 32 channels, and 1 3×3 convolution kernel with 64 channels; the output of the third convolution block is convolved by the fourth convolution block, and the fourth convolution block includes 3 3×3 convolution kernels, 3 3×3 The channels of the convolution kernel are all 92, and the step sizes are 2, 2, and 1 respectively; the output of the fourth convolution block is convolved by the fifth convolution block, and the fifth convolution block includes three 3×3 convolution kernels, and the channels of the three 3×3 convolution kernels are all 128, and the step sizes are 4, 4, and 1 respectively; the output of the fourth convolution block is convolved by the second convolution kernel, the third convolution kernel, the fourth convolution kernel, and the fifth convolution kernel respectively, to obtain outputs of four scales; among which, the second convolution kernel, the third convolution kernel, the fourth convolution kernel, and the fifth convolution kernel are respectively The size, channels, and stride of the first, fourth, and fifth convolution kernels are {1×1, 16, 1}, {3×3, 16, 6}, {3×3, 16, 12}, and {3×3, 16, 18}, respectively. The outputs of the third, fourth, and fifth convolution blocks, the second, third, fourth, and fifth convolution kernels are combined and convolved with two convolution kernels of 256 and 384, respectively, to obtain the output. The boxes represent the convolution kernel size, output dimension, and stride, respectively. If the stride is not shown, it defaults to 1.

[0050] After feature extraction, further feature transformation is performed, which is applied to the bidirectional attention module, feature matching module, and semantic segmentation module. For the dense matching and bidirectional attention modules, a series of two-dimensional convolution operations are used to reduce the feature dimension. For the semantic segmentation module, a multi-scale feature pyramid is constructed.

[0051] Bidirectional guided attention module.

[0052] The bidirectional guided attention module consists of three parts: feature transformation, global context extraction, and bidirectional attention. The first step of the feature transformation part is to reduce the feature dimension through a series of 2-D convolution operations and use a 64-channel residual block to regularize the feature map. The global context extraction part is to capture global context information through a set of global context information blocks, and the amount of computation in object detection is significantly reduced. The process is as follows: Figure 5As shown, firstly, according to the input feature map, the shape of which is HxWxF, H is the height, W is the width, and F is the number of channels, the input feature map of the shape of HxWxF is converted, and then a weight distribution is obtained by applying a softmax function, so as to form context-related features by calculating a global attention map and sharing for all positions. Secondly, the feature map of the shape of HxWxF is converted into FxHxW, and the feature map converted into FxHxW is weighted by the weight distribution output by the softmax function; and a convolution operation is performed on the weighted feature map to convert the features for capturing channel-dependent relationships. Finally, the result of the convolution operation is fused with the feature map of the input bidirectional guided attention module, and the global context features are fused into the features of all positions, and the output shape is still HxWxF. The last step is to generate a bidirectional attention map for semantic segmentation and dense matching by a sigmoid function and a two-dimensional convolution operation. For semantic segmentation, the semantic attention map will assign more weights to features of ground objects (for example, buildings, roads, and trees). For dense matching, each object is usually regarded as a rigid surface; therefore, the same object has the same or close disparity. The disparity attention map will assign different weights within the disparity range.

[0053] The effect of the bidirectional attention map on the semantic segmentation module is that the semantic weights (such as buildings and roads) in the bidirectional attention map directly enhance the features related to ground objects, so that the segmentation module pays more attention to the class saliency region and suppresses the background interference. The effect on the dense matching module is that the disparity attention map assigns similar weights to the same rigid object (such as the surface disparity of a vehicle), reduces the matching ambiguity, and improves the continuity of disparity prediction. The bidirectional information is complementary to provide scene understanding (global) for the semantic attention map and geometric constraints (local) for the disparity attention map, and the two are further coordinated through the cross-fusion module to ensure the consistency of the segmentation and disparity results in space and semantics.

[0054] The semantic segmentation module.

[0055] The semantic segmentation module applies a series of two-dimensional convolution operations to generate an initial segmentation map. Multi-scale information has been shown to be helpful for semantic segmentation of scene understanding. Therefore, the present invention uses the construction of a feature pyramid to extract multi-scale information. Multi-scale feature extraction performs convolution operations of different scales on the input image to generate feature maps with different resolutions. Top-down path: Starting from a high-resolution feature map, the size of the feature map is gradually increased through upsampling operations, while performing element-level summation or splicing with high-resolution feature maps from lower layers to restore fine-grained information. Horizontal connection: In the top-down path, the upsampled output of each layer is fused with the original resolution feature map of the same layer (i.e., the input of the horizontal connection). This is usually achieved through 1x1 convolution to reduce the number of channels of the feature map to make it consistent with the number of channels of the upsampled feature map, and then perform element-wise summation. Convolution fusion: After completing the lateral connection, a 3x3 convolution operation is performed on the fused feature map to further refine the features and enhance the representation capability of the features.

[0056] The feature map is refined through a multi-scale feature learning module, also known as an attention mechanism module, where the feature map is combined with the upsampled high-level features. After this, the fused features are upsampled accordingly and help refine the low-level features. Finally, a deconvolution layer is applied to upsample the output of the final decoder to match the spatial resolution of the input image, followed by an attention mechanism, and then fed into the final convolution layer to extract semantic information, as shown in Figure 2. Figure 4 As shown, the first attention mechanism module is used to enhance the feature 16x output by the pyramid module, and the enhanced result is input into the first decoder for decoding to obtain the first decoding feature; the second attention mechanism module is used to enhance the feature 8x output by the pyramid module, and the enhanced result and the first decoding feature are merged and input into the second decoder for decoding to obtain the second decoding feature; the third attention mechanism module is used to enhance the feature 4x output by the pyramid module, and the enhanced result and the second decoding feature are merged and input into the third decoder for decoding to obtain the third decoding feature; the fourth attention mechanism module is used to enhance the feature 2x output by the pyramid module, and the enhanced result and the third decoding feature are merged and input into the fourth decoder for decoding to obtain the fourth decoding feature. A bidirectional attention map is added to the fourth decoding feature and fused through a 1×1 convolution layer to obtain the output of the semantic segmentation module.

[0057] Global context information is crucial for the semantic information of complex scenes, and local context information also plays a key role in preserving rich spatial details. The multi-scale feature learning module focuses on global context information and local context information. One branch of the attention mechanism module uses the dot product attention mechanism. First, the feature X is convolved using one-dimensional convolution to obtain the matrix Q. Each column of the transformation result is a one-dimensional sequence of feature channels, and each row is the value of different channels at the same position in the feature map. In addition, the K and V matrices can be obtained by the same method, as shown in the following formula. Calculate the similarity between each row vector and the corresponding elements of other row vectors, and calculate the similarity between each row vector and the corresponding elements of other row vectors. By applying the softmax function to each row of the similarity matrix, the normalized similarity weight of the elements at each position in the feature map relative to all other elements can be generated. The similarity weight is multiplied by the matrix V to generate the self-attention weight matrix:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] in, is the input of the attention mechanism module, C is the number of channels, is the weight matrix of self-attention, and the superscript T represents the transpose operation.

[0063] The first initial feature is sequentially convolved, batch normalized, and activated by the ReLU function through the Conv-BN-ReLU branch: the output of the dot product attention branch is fused with the Conv-BN-ReLU branch to obtain the output of the attention mechanism module.

[0064] Feature matching module.

[0065] After integrating and transforming the second initial features of different scales, a four-dimensional cost volume (height × width × disparity range × feature size) is constructed. The minimum disparity is introduced when constructing the matching cost cube to adapt to different terrain areas and improve the accuracy and efficiency of deep learning dense matching. In order to obtain more contextual information, the present invention uses a stacked hourglass (encoding-decoding) structure, which consists of multiple repeated top-down / bottom-up processes with intermediate layer supervision. The network structure includes 4 3×3×3 convolutional layers and three stacked hourglass networks, which are composed of multiple repeated top-down / bottom-up processes with intermediate layer supervision, and introduce attention information in each stacked hourglass network, such as Figure 7As shown, a three-dimensional convolution operation is performed on the cost volume through the convolution module; a first disparity map is obtained; the first disparity map is gradually encoded and decoded through the first encoding-decoding module, the second encoding-decoding module and the third encoding-decoding module. During the decoding process, dense matching is achieved by introducing the bidirectional attention map output by the bidirectional guided attention module to obtain a dense matching feature map; the dense matching feature map is interpolated through the bilinear interpolation layer, and the interpolation result is regressed through the regression layer to output the initial disparity map.

[0066] Bidirectional attention module.

[0067] The bidirectional attention module is divided into a semantic fusion part and a disparity fusion part. The input information is the left image, the initial disparity map and the initial segmentation map. Except for the last convolution layer, the rest of the two parts are the same, such as Figure 8 As shown in the figure, the output dimension of the last convolutional layer in the semantic fusion part is class, where class represents the object category in the dataset. The semantic residual result of the fusion information is then added to the initial segmentation map to obtain the refined segmentation map. The disparity fusion part is similar to the semantic fusion part. The refined score map is calculated by adding the fusion result and the initial score map, and then the refined segmentation map is obtained through the Argmax operation. To reduce the computational complexity, the output dimension of the last convolutional layer in the disparity fusion part is 1.

[0068] S2: Collect image data to build a dataset, use the dataset to train the improved BGA-Net network, and obtain a semantic segmentation-dense matching model.

[0069] The present invention uses the TensorFlow framework and uses the ADAM optimizer (β1 = 0.9, β2 = 0.999) to train the model in an end-to-end manner. Before training, the input images are normalized and the pixel intensity levels are -1 to 1. The batch size is set to 1 and the disparity is set to 96. The total epoch is set to 40. The initial learning rate is set to 0.001, and as training progresses, the learning rate is reduced to half every 10 epochs. Please note that this training process is performed on a Tesla V100s GPU. The present invention uses the open source dataset US3D dataset, which has a total of 4292 pairs of images. These images are WorldView-3 satellite images. 4000 pairs of image pairs from the Jacksonville and Omaha areas are used for validation and training, and the rest are used for testing.

[0070] On the US3D dataset, after 40 epochs of training, the dense matching accuracy metrics, EPE, and D1, were 1.38 and 8.23%, respectively, outperforming BGANet's 1.62 and 11.29%, respectively. The semantic segmentation accuracy metric, F1 score, was 80%.

[0071] This method can achieve high-precision performance in fewer training rounds (compared to other networks), improving task efficiency. Semantic segmentation accuracy needs to be optimized for satellite imagery classification, and the attention mechanism used needs to be further optimized to improve task efficiency.

[0072] S3: The first image and the second image to be processed are input into the semantic segmentation-dense matching model, and preliminary feature extraction is performed on the first image and the second image through the unified backbone module to obtain first initial features of different scales and second initial features of different scales; context feature extraction and weighting are performed on the first initial features of different scales through the bidirectional guided attention module to obtain a bidirectional attention map; attention weighting operations are performed on the first initial features of different scales through the attention mechanism module of several convolution branches in the semantic segmentation module to obtain semantic enhancement feature maps of different scales; the semantic enhancement feature maps of different scales are decoded through the decoder module to obtain decoding feature maps of different scales; wherein, the input of each decoder module also includes the output from the decoder module of the convolution branch of the previous scale; the initial segmentation map is obtained by adding the bidirectional attention map to the decoding feature map of the minimum scale; the second initial features of different scales are integrated into a cost body through the feature matching module, and the cost body and the bidirectional attention map are densely matched to obtain an initial disparity map; the initial segmentation map and the initial disparity map are cross-fused through the bidirectional attention module to obtain the final segmentation map and the final disparity map.

[0073] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0074] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for semantic segmentation and matching of an image, characterized in that: include: The semantic segmentation module in the original BGA-Net network is replaced with an improved semantic segmentation module, wherein the improved semantic segmentation module includes several convolution branches, each of which includes an attention mechanism module and a decoder module connected in sequence; the output of the decoder in each convolution branch is connected to the input of the decoder module in the next convolution branch along a scale-reducing direction; and an improved BGA-Net network is constructed, which includes a unified backbone module, a bidirectional guided attention module, an improved semantic segmentation module, a feature matching module, and a bidirectional attention module in the original BGA-Net network; Collect image data to build a dataset, use the dataset to train the improved BGA-Net network, and obtain a semantic segmentation-dense matching model; Inputting the first image and the second image to be processed into the semantic segmentation-dense matching model, performing preliminary feature extraction on the first image and the second image through the unified backbone module to obtain first initial features of different scales and second initial features of different scales; The bidirectional guided attention module extracts and weights the contextual features of the first initial features of different scales to obtain a bidirectional attention map; the attention mechanism module of several convolution branches in the improved semantic segmentation module performs attention weighting operations on the first initial features of different scales to obtain semantic enhancement feature maps of different scales; the semantic enhancement feature maps of different scales are decoded by the decoder module to obtain decoding feature maps of different scales; wherein, the input of each decoder module also includes the output of the convolution branch decoder module of the previous scale; the initial segmentation map is obtained by adding the bidirectional attention map to the minimum scale decoding feature map; the second initial features of different scales are integrated into a cost body through the feature matching module, and the cost body and the bidirectional attention map are densely matched to obtain the initial disparity map; the initial segmentation map and the initial disparity map are cross-fused by the bidirectional attention module to obtain the final segmentation map and the final disparity map; The feature matching module is an improved feature matching module, which includes a convolution module, a first encoding-decoding module, a second encoding-decoding module, a third encoding-decoding module, a bilinear interpolation layer, and a regression layer connected in sequence. The specific steps of outputting the initial disparity map by the improved feature matching module include: Performing a three-dimensional convolution operation on the cost volume through a convolution module; obtaining a first disparity map; The first disparity map is gradually encoded and decoded by a first encoding-decoding module, a second encoding-decoding module, and a third encoding-decoding module. During the decoding process, dense matching is achieved by introducing a bidirectional attention map output by a bidirectional guided attention module to obtain a dense matching feature map. The dense matching feature map is interpolated through a bilinear interpolation layer, and the interpolation result is regressed through a regression layer to output the initial disparity map.

2. The image semantic segmentation and matching method according to claim 1, wherein: The attention mechanism module includes a parallel dot product attention branch and a Conv-BN-ReLU branch; The steps of the attention mechanism module to perform attention weighting operations include: The dot product attention branch is used to transform the input of the attention mechanism module to obtain matrices Q, K, and V. The softmax function is used to calculate the normalized similarity weight of the element at each position in the input of the attention mechanism module relative to all other elements, and the similarity weight is multiplied by the matrix V to generate the self-attention weight matrix: in, is the input of the attention mechanism module, C is the number of channels, is the weight matrix of self-attention, and the superscript T represents the transpose operation; The first initial feature is sequentially convolved, batch normalized, and activated by the ReLU function through the Conv-BN-ReLU branch: The output of the dot product attention branch is fused with the Conv-BN-ReLU branch to obtain the output of the attention mechanism module.

3. The image semantic segmentation and matching method according to claim 1, wherein: A feature pyramid module is provided between the unified backbone module and the improved semantic segmentation module. The feature pyramid module enhances the semantic information in the first initial feature through convolution operations of different scales and deconvolution operations of different scales.

4. The image semantic segmentation and matching method according to claim 1, wherein: The dense matching is achieved by introducing a bidirectional attention map output by a bidirectional guided attention module, which is achieved by fusing the bidirectional attention map with the output of the encoding layer in the encoding-decoding module and inputting the result into the decoding layer in the encoding-decoding module for decoding.

5. The image semantic segmentation and matching method according to claim 1, wherein: The specific steps of the bidirectional guidance attention module for context feature extraction and weighting include: After transforming the feature map of the input bidirectional guided attention module into the shape of H×W×F, the softmax function is applied to obtain the weight distribution; The feature map with a shape of H×W×F is converted to F×H×W, and the feature map converted to F×H×W is weighted by the weight distribution output by the softmax function; and the weighted feature map is convolved; The result of the convolution operation is fused with the feature map of the input bidirectional guided attention module to output a bidirectional attention map.

Images