Remote sensing image semantic segmentation method and system

Abstract

This invention provides a method and system for semantic segmentation of remote sensing images, relating to the field of image processing technology. The method includes: establishing an initial semantic segmentation model and training the initial semantic segmentation model using training data to obtain a target model; wherein the target model includes a semantic segmentation module and a correction module; acquiring an initial image and cropping the original image to obtain multiple images to be processed; performing initial semantic segmentation on each of the images to be processed using the semantic segmentation module to obtain initial semantic segmentation results; performing bidirectional feature correction on rows and columns on the initial semantic segmentation results using the correction module to obtain semantic segmentation results corresponding to each image to be processed; and concatenating the semantic segmentation results corresponding to each image to be processed to generate a target semantic segmentation result, thereby addressing the problem of low accuracy in semantic segmentation of remote sensing images in the prior art.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a semantic segmentation method and system for remote sensing images. Background Technology

[0002] With the rapid development of information and internet technologies, the digital age has arrived, and the level of urban informatization has become an important indicator for measuring a city's development. The digital city is an essential stage in the development of urban informatization, moving from independence to integration, from partial to holistic, and from fragmentation to consolidation. It has a positive impact and driving force on urban management, construction, planning, and the urban economy. The operation and construction of digital cities require information technology as support, and spatial data is the most important and fundamental information resource. Utilizing geographic remote sensing information effectively can support spatial data processing and production, playing a crucial role in the development of digital cities. Applying geographic remote sensing information in digital cities can promote the development of surveying and mapping towards real-time, intelligent, and automated methods, reducing the workload of urban data acquisition and having positive significance and important value in promoting digital city development. To utilize remote sensing information in digital cities, it is necessary to employ technical means, such as image semantic segmentation, to analyze remote sensing images and obtain data that is difficult to statistically analyze, such as vegetation coverage, water coverage, building area, and arable land area.

[0003] Semantic segmentation algorithms based on deep learning have emerged rapidly in recent years and achieved good results. However, most existing remote sensing image semantic segmentation methods are designed for specific categories of image analysis, such as land identification and water source identification. The semantic segmentation accuracy is relatively low, especially in terms of handling details, often resulting in numerous category recognition errors. They also often fail to achieve good semantic segmentation results for semantically complex objects. Summary of the Invention

[0004] In order to overcome the above-mentioned technical defects, the purpose of this invention is to provide a remote sensing image semantic segmentation method and system to solve the problem of low accuracy in the semantic segmentation of remote sensing images in the prior art.

[0005] This invention discloses a method for semantic segmentation of remote sensing images, comprising the following steps:

[0006] An initial semantic segmentation model is established, and the initial semantic segmentation model is trained using training data to obtain the target model;

[0007] The target model includes a semantic segmentation module based on a UNET network with overlapping dilated convolutions and a correction module based on a bidirectional LSTM network.

[0008] Obtain an initial image, crop the original image to obtain multiple images to be processed;

[0009] Each of the images to be processed is sequentially input into the target model;

[0010] The semantic segmentation module is used to perform initial semantic segmentation on each of the images to be processed to obtain initial semantic segmentation results;

[0011] A correction module is used to perform bidirectional feature correction on rows and columns on the initial semantic segmentation result to obtain the semantic segmentation result corresponding to each image to be processed;

[0012] The semantic segmentation results corresponding to each image to be processed are concatenated to generate the target semantic segmentation result.

[0013] Preferably, the initial semantic segmentation of each of the images to be processed using the semantic segmentation module to obtain initial semantic segmentation results includes the following:

[0014] For any image to be processed;

[0015] The image to be processed is processed using a first dilated convolutional UNET network to obtain a first result;

[0016] The first result is combined with the image to be processed and input into the second dilated convolutional UNET network to obtain the initial semantic segmentation result; wherein the second dilated convolutional UNET network has the same structure as the first dilated convolutional UNET network.

[0017] Preferably, the step of processing the image to be processed using a first dilated convolutional UNET network to obtain a first result includes the following:

[0018] The image to be processed is subjected to convolution and pooling downsampling to obtain a first feature map;

[0019] A second feature map is obtained by upsampling based on the first feature map using transposed convolution;

[0020] Cropping and fusing low-level feature maps, and repeating downsampling and upsampling to obtain the output feature map;

[0021] The output feature map is downsampled using a parallel dilated convolution group, and after upsampling, it is passed through a softmax layer to output the first processing result. The parallel dilated convolution group contains dilated convolution networks with different dilation rates.

[0022] Preferably, when upsampling based on the first feature map using transposed convolution, the following is included:

[0023] The "same" type padding is used to ensure that the size of the output first feature map is the same as that of the image to be processed.

[0024] Preferably, the step of using the correction module to perform bidirectional feature correction on the rows and columns of the initial semantic segmentation result to obtain the semantic segmentation result corresponding to each image to be processed includes the following:

[0025] Based on each image in the initial semantic segmentation result, it is divided into several columns and several rows of images in the column and row directions, respectively, according to a preset number of samples; wherein, the preset number of samples is consistent with the input of each bidirectional LSTM network of the correction module;

[0026] The aforementioned column images and row images are simultaneously input into a bidirectional LSTM network to generate several feature maps located in the column and row directions, respectively.

[0027] The feature maps located in the column and row directions are combined according to channels to generate a combined feature map;

[0028] The combined feature maps are passed through a preset convolutional network to obtain the semantic segmentation results corresponding to each image to be processed.

[0029] Preferably, the step of training the initial semantic segmentation model with training data to obtain the target model includes the following:

[0030] The initial semantic segmentation model includes an initial semantic segmentation module and an initial correction module;

[0031] The initial semantic segmentation module is trained using the first training data until the UNET network with overlapping dilated convolutions converges and reaches a preset first precision value, and the parameters of the initial semantic segmentation module are fixed.

[0032] The initial correction module is trained using the second training data until the bidirectional LSTM network converges and reaches a preset second precision value; wherein, the output of the initial semantic segmentation module in the first training data is used as input for the second training data;

[0033] The parameters of the initial semantic segmentation module are no longer fixed. The initial semantic segmentation module and the initial correction module are trained as a whole using the third training data until the training is completed and the target model is obtained.

[0034] Preferably, the cropping of the original image to obtain multiple images to be processed includes the following:

[0035] Obtain a preset target quantity, and calculate the target image size based on the preset target quantity;

[0036] Cropping is performed by sliding a first dimension horizontally and a second dimension vertically according to the target image size to obtain multiple images to be processed; wherein, the first dimension is smaller than the width of the target image size by a first preset value; and the second dimension is smaller than the height of the target image size by a second preset value.

[0037] Preferably, the semantic segmentation results corresponding to each image to be processed are concatenated to generate the target semantic segmentation result;

[0038] Based on the cropping rules of the original image, the target semantic segmentation result is generated by performing the reverse operation according to the cropping rules.

[0039] The present invention also provides a remote sensing image semantic segmentation system, comprising:

[0040] The model building module is used to build an initial semantic segmentation model and train the initial semantic segmentation model using training data to obtain the target model;

[0041] The target model includes a semantic segmentation module based on a UNET network with overlapping dilated convolutions and a correction module based on a bidirectional LSTM network.

[0042] The preprocessing module is used to acquire an initial image, crop the original image, and obtain multiple images to be processed.

[0043] The input module is used to sequentially input each of the images to be processed into the target model;

[0044] The semantic segmentation module is used to perform initial semantic segmentation on each of the images to be processed to obtain initial semantic segmentation results;

[0045] The correction module is used to perform bidirectional feature correction on rows and columns on the initial semantic segmentation result to obtain the semantic segmentation result corresponding to each image to be processed;

[0046] The processing module is used to concatenate the semantic segmentation results corresponding to each image to be processed to generate the target semantic segmentation result.

[0047] Preferably, the semantic segmentation module performs the following:

[0048] For any image to be processed;

[0049] The image to be processed is processed using a first dilated convolutional UNET network to obtain a first result;

[0050] The first result is combined with the image to be processed and input into the second dilated convolutional UNET network to obtain the initial semantic segmentation result; wherein the second dilated convolutional UNET network has the same structure as the first dilated convolutional UNET network.

[0051] Compared with existing technologies, the above technical solution has the following advantages:

[0052] In this scheme, a UNET network with dilated convolutions is used in the semantic segmentation module to enhance the recognition and segmentation of small targets in remote sensing images. A correction module based on a bidirectional LSTM network is used to correct edge pixels of the initial segmentation results synchronously in the row and column directions, thereby improving the segmentation effect of object edges, reducing segmentation errors, and solving the problem of low semantic segmentation accuracy of remote sensing images in existing technologies. Attached Figure Description

[0053] Figure 1 This is a flowchart of an embodiment of the remote sensing semantic segmentation method described in this invention;

[0054] Figure 2 This is a flowchart illustrating step-by-step training in Embodiment 1 of the remote sensing semantic segmentation method described in this invention;

[0055] Figure 3 This is a flowchart of the semantic segmentation module processing in Embodiment 1 of the remote sensing semantic segmentation method described in this invention;

[0056] Figure 4 This is a schematic diagram of the network structure of the semantic segmentation model in Embodiment 1 of the remote sensing semantic segmentation method of the present invention;

[0057] Figure 5 This is a flowchart illustrating the use of a first dilated convolutional UNET network to process the image to be processed and obtain a first result in a first embodiment of the remote sensing semantic segmentation method of the present invention.

[0058] Figure 6 This is a flowchart of the correction module processing in Embodiment 1 of the remote sensing semantic segmentation method described in this invention;

[0059] Figure 7 This is a schematic diagram of the network structure of the modified model in Embodiment 1 of the remote sensing semantic segmentation method of the present invention;

[0060] Figure 8 This is a schematic diagram of a module in a second embodiment of the remote sensing semantic segmentation system described in this invention.

[0061] Figure label:

[0062] 7-Remote Sensing Semantic Image Segmentation System; 71-Model Building Module; 72-Preprocessing Module; 73-Input Module; 74-Semantic Segmentation Module; 75-Correction Module; 76-Processing Module Detailed Implementation

[0063] The advantages of the present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments.

[0064] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0065] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0066] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0067] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0068] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0069] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0070] Example 1: This embodiment provides a method for semantic segmentation of remote sensing images. (See attached document.) Figure 1 This includes the following steps:

[0071] S100: Establish an initial semantic segmentation model and train the initial semantic segmentation model using training data to obtain a target model; wherein, the target model includes a semantic segmentation module based on a UNET network with overlapping dilated convolutions and a correction module based on a bidirectional LSTM network; as a supplementary explanation, the semantic segmentation module includes two overlapping UNET networks with dilated convolutions, and the correction module includes two parallel bidirectional LSTM networks, a feature combination layer, and a preset convolutional network.

[0072] Specifically, in the semantic segmentation task of remote sensing images, the UNET network can achieve good results with a relatively small amount of data. Therefore, in this invention, the UNET network is used as the basic architecture of the segmentation network. Specifically, the input image to be segmented is processed by a specially designed Stacked Dilated UNET (i.e., the semantic segmentation module built on the UNET network with overlapping dilated convolutions mentioned above) to obtain preliminary segmentation results. These preliminary segmentation results are then fed into a correction module (i.e., the correction module built on a bidirectional LSTM network mentioned above), and the final segmentation result is output after correction. It should be noted that, to further increase the types of remote sensing images that can be recognized, common objects in remote sensing images, such as houses, streets, vegetation, water sources, and farmland, are included during training, rather than targeting only a single or limited number of objects. The target model is trained using this scheme, enabling it to recognize multiple types of objects in remote sensing images.

[0073] During training, since simultaneously training the initial semantic segmentation model would increase the training difficulty, a step-by-step training approach is adopted. Specifically, the initial semantic segmentation model is trained using training data to obtain the target model, see [link to relevant documentation]. Figure 2 Including the following:

[0074] The initial semantic segmentation model includes an initial semantic segmentation module and an initial correction module;

[0075] As can be seen from the above, the target model includes a semantic segmentation module and a correction module. The initial semantic segmentation model also contains these two modules, the only difference being that the network parameters are inconsistent before and after training.

[0076] S110: Train the initial semantic segmentation module using the first training data until the UNET network with overlapping dilated convolutions converges and reaches a preset first precision value, and fix the parameters of the initial semantic segmentation module.

[0077] In the above steps, the initial semantic segmentation model is trained stepwise, that is, the semantic segmentation module and the correction module are trained separately first, and then the two modules are trained together at the end. It should also be noted that the first training data, the second training data and the third training data described below are all labeled training data, and the first training data and the second training data are subsets of the third training data. The second training data uses the output of the first training data as input.

[0078] S120: The initial correction module is trained using the second training data until the bidirectional LSTM network converges and reaches a preset second precision value; wherein, the second training data uses the output of the initial semantic segmentation module in the first training data as input;

[0079] Specifically, it should be emphasized that the second training data must use the labeled output in the first training data as input. The initial semantic segmentation module and the initial correction module can be trained synchronously, or the initial correction module can be trained first and the parameters fixed before the initial semantic segmentation module is trained. In this embodiment, the initial semantic segmentation module is trained first.

[0080] S130: Unfix the parameters of the initial semantic segmentation module, and use the third training data to train the initial semantic segmentation module and the initial correction module as a whole until the training is completed and the target model is obtained.

[0081] Specifically, the first training data and the second training data are subsets of the third training data. That is, the third training data contains the total data of the two modules. The data is input into the initial semantic segmentation module and the initial correction module, and the output is directly processed by the module. The parameters of the two modules are adjusted synchronously to obtain the target model.

[0082] S200: Obtain an initial image, crop the original image to obtain multiple images to be processed;

[0083] Specifically, the original remote sensing images (i.e., the initial images) have extremely high resolution and cannot be directly sent to the network, and also require a huge amount of memory. Therefore, the original images are cropped into many images of equal size.

[0084] Specifically, the cropping of the original image to obtain multiple images to be processed includes the following:

[0085] S210: Obtain a preset number of targets, and calculate the target image size based on the preset number of targets;

[0086] Specifically, as an example, if we set an initial image of 6000×4000 (W×H) and the preset target quantity is 400, we can crop it into 400 target images of 300×200 (i.e., target image size).

[0087] S220: Based on the target image size, crop by sliding the width of a first dimension horizontally and the height of a second dimension vertically to obtain multiple images to be processed; wherein, the first dimension is smaller than the width of the target image size by a first preset value; and the second dimension is smaller than the height of the target image size by a second preset value.

[0088] The above steps primarily aim to avoid noticeable stitching when reassembling the segmented images into a complete image. During image cropping, the image is slid horizontally to two-thirds of the target image's width and vertically to three-quarters of its height, ensuring that each cropped target image overlaps with other target images. Using this method, 725 images can be obtained in the example above. This demonstrates that this method effectively removes stitching artifacts while increasing the amount of data, thus improving the network's training performance. Finally, the individual target images can be stitched together according to the cropping rules to reconstruct the final segmentation result.

[0089] S300: Input each of the images to be processed into the target model in sequence;

[0090] In step S200 above, the initial image is cropped into several smaller images to be processed, and semantic segmentation is performed on each image to be processed (passing through the semantic segmentation module and the correction module in sequence). Finally, the segmentation results of each image to be processed are stitched together.

[0091] S400: The semantic segmentation module is used to perform initial semantic segmentation on each of the images to be processed to obtain initial semantic segmentation results;

[0092] As described above, the semantic segmentation module includes two dilated UNETs with dilated convolutions (i.e., the first dilated convolutional UNET network and the second dilated convolutional UNET network described below). Specifically, the semantic segmentation module performs initial semantic segmentation on each of the images to be processed to obtain initial semantic segmentation results, see [link to documentation]. Figure 3 and Figure 4 Including the following:

[0093] For any image to be processed;

[0094] S410: The image to be processed is processed using the first dilated convolutional UNET network to obtain a first result;

[0095] It should be noted that the structure of each Dilated UNET with dilated convolution is consistent. Furthermore, its Dilated UNET is formed by replacing the original standard convolutional module with a set of parallel dilated convolutional groups composed of different dilation rates during the convolution process after the last downsampling of the standard UNET network. In the upsampling stage, transposed convolution replaces pooling for upsampling, while other aspects remain unchanged. Therefore, the first dilated convolutional UNET network is used to process the image to obtain the first result. For details, please refer to [link to relevant documentation]. Figure 5 Including the following:

[0096] S411: Perform convolution and pooling downsampling on the image to be processed to obtain a first feature map;

[0097] Specifically, the convolution and pooling downsampling steps mentioned above are processing steps in the standard UNET network. The standard UNET network is the existing general-purpose UNET network. The overall structure of this network is similar to the uppercase English letter U, including a contraction path and an expansion path. The contraction path is mainly used to capture the contextual information in the image (i.e., feature extraction in the first half, convolution and pooling downsampling), while the corresponding expansion path is used to accurately locate the parts of the image that need to be segmented (i.e., upsampling in the second half).

[0098] S412: Upsample the first feature map using transposed convolution to obtain the second feature map;

[0099] Specifically, transposed convolution and regular convolution have the same essence, which is to establish a relationship between values. However, the relationship established by transposed convolution is in the opposite direction to that established by regular convolution.

[0100] When upsampling based on the first feature map using transposed convolution, the following is included:

[0101] The "same" type padding is used to ensure that the size of the output first feature map is the same as that of the image to be processed.

[0102] The use of "same" padding in the above steps is to ensure that the input and output sizes are the same, thereby reducing the possibility of size inconsistencies caused by the transposed convolution process.

[0103] S413: Crop and fuse the low-level feature maps, repeat downsampling and upsampling to obtain the output feature map;

[0104] As described above, the main changes in the first and second dilated convolutional UNET networks in this embodiment lie in the operation after the last set of upsampling, and the use of transposed convolution for upsampling. Therefore, the above steps are the processing steps in the standard UNET network. Lower-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and they contain more noise. Higher-level features have stronger semantic information, but their resolution is very low, and their ability to perceive details is poor. Efficiently fusing the two improves the segmentation effect.

[0105] S414: The output feature map is downsampled using a parallel dilated convolution group, and after upsampling, it is output as a first processing result through a softmax layer. The parallel dilated convolution group contains dilated convolution networks with different dilation rates.

[0106] Specifically, in the above steps, parallel dilated convolutional groups are used to replace the original convolutional layers, which differs from the standard UNET network and results in better segmentation and recognition performance. Dilated convolution involves padding the convolutional kernel with zeros, and there are two implementation methods: first, padding the convolutional kernel with zeros; second, sampling the input at equal intervals. The appropriate implementation method can be selected according to the actual application scenario. Finally, feature classification is performed through a softmax (fully connected) layer, and the output result is given.

[0107] S420: Combine the first result with the image to be processed and input it into the second dilated convolutional UNET network to obtain the initial semantic segmentation result; wherein, the second dilated convolutional UNET network has the same structure as the first dilated convolutional UNET network.

[0108] In the above steps, after inputting the image to be processed into the first Dilated Unet (i.e., the first dilated convolutional UNET network) to obtain a segmentation result, the segmentation result is then combined with the image to be processed and fed into the second Dilated Unet (i.e., the second dilated convolutional UNET network) to fine-tune the segmentation of the first result and obtain the initial semantic segmentation result. It should be noted that since the second dilated convolutional UNET network has the same structure as the first dilated convolutional UNET network, the processing of the first result can refer to the above steps S411-S413.

[0109] S500: The initial semantic segmentation result is corrected in both rows and columns by the correction module to obtain the semantic segmentation result corresponding to each image to be processed;

[0110] Specifically, as described above, the correction module includes two parallel bidirectional LSTM networks to perform bidirectional feature correction in both rows and columns. Its purpose is to effectively utilize the relationships between adjacent pixels, performing corrections both in the column and row directions. Considering that the relationship between adjacent pixels is bidirectional rather than unidirectional, a bidirectional LSTM is used as the processing structure. Therefore, the correction module performs bidirectional feature correction in both rows and columns on the initial semantic segmentation result to obtain the semantic segmentation result corresponding to each image to be processed. (See [reference]). Figure 6 Including the following:

[0111] S510: Based on each image in the initial semantic segmentation result, divide it into several columns and several rows of images in the column and row directions respectively according to a preset number of samples; wherein, the preset number of samples is consistent with the input of each bidirectional LSTM network of the correction module;

[0112] In this embodiment, each image in the initial semantic segmentation result is divided into several columns and several rows according to the number of single samples of 3, 5 and 7, respectively, and then fed into the corresponding bidirectional LSTM to obtain multiple feature maps in the column direction and the row direction.

[0113] S520: The plurality of column images and the plurality of row images are synchronously input into a bidirectional LSTM network to generate a plurality of feature maps located in the column and row directions respectively;

[0114] In the above steps, setting up two bidirectional LSTM networks allows for synchronous processing of rows and columns. Alternatively, a single network could be used for sequential processing, but this would reduce processing efficiency. Therefore, two parallel bidirectional LSTM networks are used. S530: Combine the feature maps located in the column and row directions according to channels to generate a combined feature map;

[0115] As described above, the above steps are performed in the feature combination layer of the correction module. The above channel is for the detection of a certain feature. The strength of a certain value in the channel is the response to the strength of the current feature. Therefore, channel combination is to superimpose the various features in the feature map.

[0116] S540: The combined feature map is passed through a preset convolutional network to obtain the semantic segmentation results corresponding to each image to be processed.

[0117] In the above steps, the pre-set convolutional network is set to 1×1 convolution, which acts as a fully connected layer, so that the features superimposed in the above steps are combined into a segmentation map, that is, the semantic segmentation result corresponding to the image to be processed. The size of the above feature map is H×W×N, where H is the image height, W is the image width, and N is the number of categories of the segmented target.

[0118] The aforementioned correction module primarily adjusts the relationship between adjacent pixels in the initial segmentation result to obtain better edge segmentation. This differs from existing technologies that use multiple CRF iterations to achieve better segmentation results, but this method is complex, its hyperparameters are difficult to adjust, the number of iterations requires experimentation, and it is time-consuming. Therefore, this embodiment uses a recurrent neural network (two synchronized bidirectional LSYM networks) to process the initial segmentation result.

[0119] S600: The semantic segmentation results corresponding to each image to be processed are concatenated to generate the target semantic segmentation result.

[0120] Since the above step S200 cropped the original image to obtain multiple images to be processed, after the semantic segmentation of the images to be processed is completed, they need to be spliced ​​together. Specifically, the semantic segmentation results corresponding to each image to be processed are spliced ​​together to generate the target semantic segmentation result.

[0121] Based on the cropping rules of the original image, the target semantic segmentation result is generated by performing the reverse operation according to the cropping rules.

[0122] Specifically, the cropping rules mentioned above are as follows: the image is slid horizontally to two-thirds of its width and vertically to three-quarters of its height to crop, so that each cropped image overlaps with other images. Therefore, the process is reversed during stitching, that is, the images to be processed are stitched horizontally and the overlapping one-third of the images are eliminated, and the overlapping one-quarter of the images are eliminated vertically to obtain the target semantic segmentation result.

[0123] In this embodiment, overlapping Dilated UNETs (i.e., the first dilated convolutional UNET network and the second dilated convolutional UNET network) are used as semantic segmentation modules, which significantly enhances the recognition and segmentation accuracy in the segmentation task and improves the recognition and segmentation effect of small targets in remote sensing images. A correction module based on bidirectional LSTM is used to correct edge pixels of the initial segmentation results, which significantly improves the segmentation effect of object edges and reduces segmentation error. The target model is obtained by step-by-step training, which reduces the difficulty of training convergence, and step-by-step training can achieve higher accuracy than end-to-end training.

[0124] Example 2: This example provides a remote sensing image semantic segmentation system 7, see reference. Figure 8 ,include:

[0125] The model building module 71 is used to build an initial semantic segmentation model and train the initial semantic segmentation model using training data to obtain a target model. The target model includes a semantic segmentation module based on a UNET network with overlapping dilated convolutions and a correction module based on a bidirectional LSTM network. The semantic segmentation module includes two overlapping UNET networks with dilated convolutions, and the correction module includes two parallel bidirectional LSTM networks, a feature combination layer, and a preset convolutional network.

[0126] It should be noted that during the training process, since the model is large, starting training simultaneously would increase the training difficulty. Therefore, a step-by-step training method is adopted, that is, the semantic segmentation module and the correction module are trained separately and then combined for training again until the target model is obtained.

[0127] Preprocessing module 72 is used to acquire an initial image, crop the original image, and obtain multiple images to be processed;

[0128] Specifically, the original remote sensing images have extremely high resolution, making them unsuitable for direct input into the network due to their high memory consumption. Therefore, the original images are cropped into many images of equal size. For example, to avoid noticeable stitching when reassembling the segmented images into a complete image, the cropping process involves sliding two-thirds of the width of the smaller image horizontally and three-quarters of its height vertically. This ensures that the cropped images have overlapping portions, effectively removing stitching artifacts and increasing the amount of data, thus enhancing the training effect.

[0129] Input module 73 is used to sequentially input each of the images to be processed into the target model;

[0130] The semantic segmentation module 74 is used to perform initial semantic segmentation on each of the images to be processed to obtain initial semantic segmentation results;

[0131] Specifically, the semantic segmentation module performs the following:

[0132] For any image to be processed, the first dilated convolutional UNET network is used to process the image to obtain a first result; the first result is combined with the image to be processed and input into the second dilated convolutional UNET network to obtain an initial semantic segmentation result; wherein, the second dilated convolutional UNET network has the same structure as the first dilated convolutional UNET network.

[0133] Specifically, both the first and second dilated convolutional UNET networks replace the original standard convolutional modules with a set of parallel dilated convolutional groups composed of different dilation rates during the convolution process after the last downsampling of the standard UNET. In the upsampling stage, transposed convolution is used instead of pooling for upsampling, and "same" type padding is used to ensure that the input and output sizes are the same, while other parts remain unchanged.

[0134] The correction module 75 is used to perform bidirectional feature correction on rows and columns on the initial semantic segmentation result to obtain the semantic segmentation result corresponding to each image to be processed;

[0135] Specifically, the correction module performs corrections in both the column and row directions. The image is divided into several columns and rows with single sample sizes of 3, 5, and 7, respectively, and fed into the corresponding bidirectional LSTM network to obtain multiple feature maps in the column and row directions. These feature maps are then combined by channel to form a new feature map, which is then subjected to a 1×1 convolution to obtain the semantic segmentation result.

[0136] The processing module 76 is used to concatenate the semantic segmentation results corresponding to each image to be processed to generate the target semantic segmentation result.

[0137] In this embodiment, an initial semantic segmentation model is established through the model building module 71, and a target model is obtained by training it using a distributed training method. The initial image is cropped by the preprocessing module 72 to obtain several images to be processed, which are then input into the target model by the input module 73. The semantic segmentation module 74 of the target model obtains the initial segmentation result through a specially designed Stacked Dilated UNET (i.e., a UNET network with overlapping dilated convolutions). The initial segmentation result is then sent to the correction module 75, which outputs the semantic segmentation result corresponding to the image to be processed after correction. This result is then stitched together by the processing module 76 to obtain the target semantic segmentation result. Based on the semantic segmentation module 74 and the correction module 75, the overlapping first dilated convolution UNET network and the second dilated convolution UNET network are used as the semantic segmentation module, which significantly enhances the recognition and segmentation accuracy in the segmentation task and improves the recognition and segmentation effect of small targets in remote sensing images. The correction module based on bidirectional LSTM is used to correct the edge pixels of the initial segmentation result, which significantly improves the segmentation effect of object edges in the segmentation, reduces the segmentation error, and solves the problem of low semantic segmentation accuracy of remote sensing images in the prior art.

[0138] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for semantic segmentation of remote sensing images, characterized in that, The method comprises the following steps: An initial semantic segmentation model is established, and the initial semantic segmentation model is trained using training data to obtain a target model; The target model comprises a semantic segmentation module established based on a UNET network with overlapping dilated convolution and a correction module established based on a bidirectional LSTM network; the UNET network with overlapping dilated convolution uses a transposed convolution for upsampling in the UNET network and a parallel dilated convolution group for the last downsampling; An initial image is obtained, and the initial image is cropped to obtain a plurality of to-be-processed images; Each to-be-processed image is sequentially input into the target model; An initial semantic segmentation is performed on each to-be-processed image using the semantic segmentation module to obtain an initial semantic segmentation result; Bidirectional feature correction is performed on the initial semantic segmentation result in the row and column directions using the correction module, the relationship between adjacent pixels in the initial semantic segmentation result is adjusted, edge pixel correction is performed, and a semantic segmentation result corresponding to each to-be-processed image is obtained; The semantic segmentation results corresponding to the to-be-processed images are spliced to generate a target semantic segmentation result.

2. The semantic segmentation method of claim 1, wherein, The initial semantic segmentation is performed on each to-be-processed image using the semantic segmentation module to obtain an initial semantic segmentation result, which comprises the following: For any to-be-processed image; A first result is obtained by processing the to-be-processed image using a first dilated convolution UNET network; The first result and the to-be-processed image are combined and input into a second dilated convolution UNET network to obtain an initial semantic segmentation result; the second dilated convolution UNET network has the same structure as the first dilated convolution UNET network.

3. The semantic segmentation method of claim 2, wherein, The first result is obtained by processing the to-be-processed image using a first dilated convolution UNET network, which comprises the following: The to-be-processed image is convolved and down-sampled by pooling to obtain a first feature map; Upsampling is performed on the first feature map based on the first feature map using a transposed convolution to obtain a second feature map; Low-level feature maps are cropped and fused, and down-sampling and up-sampling are repeated to obtain an output feature map; The output feature map is down-sampled using a parallel dilated convolution group, and after up-sampling, a first processing result is output through a softmax layer; the parallel dilated convolution group comprises dilated convolution networks with different dilated convolution rates.

4. The semantic segmentation method of claim 3, wherein, When upsampling is performed on the first feature map based on the first feature map using a transposed convolution, the following is included: A "same" type of padding is used to make the size of the output first feature map consistent with that of the to-be-processed image.

5. The semantic segmentation method of claim 1, wherein, The bidirectional feature correction is performed on the initial semantic segmentation result in the row and column directions using the correction module to obtain the semantic segmentation result corresponding to each to-be-processed image, which comprises the following: Based on each image in the initial semantic segmentation result, a corresponding number of column images and a corresponding number of row images are divided in the column and row directions according to a preset sample number; the preset sample number is consistent with the input of each bidirectional LSTM network of the correction module; The column images and the row images are synchronously input into a bidirectional LSTM network to generate a plurality of feature maps located in the column and row directions, respectively; Combine the feature maps in the column and row directions by channel to generate a combined feature map; Pass the combined feature map through a preset convolutional network to obtain a semantic segmentation result corresponding to each to-be-processed image.

6. The semantic segmentation method of claim 1, wherein, The training of the initial semantic segmentation model by using the training data to obtain a target model includes the following: The initial semantic segmentation model includes an initial semantic segmentation module and an initial correction module; The initial semantic segmentation module is trained by using first training data until the UNET network with overlapping dilated convolution converges and reaches a preset first accuracy value, and the parameters of the initial semantic segmentation module are fixed; The initial correction module is trained by using second training data until the bidirectional LSTM network converges and reaches a preset second accuracy value; the second training data uses the output of the initial semantic segmentation module in the first training data as input; The fixation of the parameters of the initial semantic segmentation module is cancelled, and the initial semantic segmentation module and the initial correction module are integrally trained by using third training data until the training is completed to obtain the target model.

7. The semantic segmentation method of claim 1, wherein, The cropping of the initial image to obtain a plurality of to-be-processed images includes the following: A preset target number is obtained, and a target image size is calculated according to the preset target number; The initial image is cropped according to the target image size by sliding a width of a first size in a horizontal direction and a height of a second size in a vertical direction to obtain a plurality of to-be-processed images; the first size is less than the width of the target image size by a first preset value, and the second size is less than the height of the target image size by a second preset value.

8. The semantic segmentation method of claim 1, wherein, The semantic segmentation results corresponding to each to-be-processed image are spliced to generate a target semantic segmentation result; The target semantic segmentation result is generated by performing reverse operations according to the cropping rule of the initial image. 9.A remote sensing image semantic segmentation system, characterized in that, It includes: A model establishing module is configured to establish an initial semantic segmentation model, and train the initial semantic segmentation model by using training data to obtain a target model; The target model includes a semantic segmentation module established based on a UNET network with overlapping dilated convolution and a correction module established based on a bidirectional LSTM network; the UNET network with overlapping dilated convolution uses transpose convolution for upsampling in the UNET network and a parallel dilated convolution group for the last downsampling; A preprocessing module is configured to obtain an initial image, crop the initial image to obtain a plurality of to-be-processed images; An input module is configured to sequentially input each to-be-processed image into the target model; A semantic segmentation module is configured to perform initial semantic segmentation on each to-be-processed image to obtain an initial semantic segmentation result; A correction module is configured to perform bidirectional feature correction on rows and columns on the initial semantic segmentation result, adjust the relationship between adjacent pixels in the initial semantic segmentation result, and perform edge pixel correction to obtain a semantic segmentation result corresponding to each to-be-processed image; A processing module is configured to splice the semantic segmentation results corresponding to each to-be-processed image to generate a target semantic segmentation result.

10. The semantic segmentation system of claim 9, wherein, The semantic segmentation module performs the following: For any to-be-processed image; The first result is obtained by processing the image to be processed by using a first dilated convolution UNET network. The first result and the image to be processed are combined and input into a second dilated convolution UNET network to obtain an initial semantic segmentation result, wherein the second dilated convolution UNET network has the same structure as the first dilated convolution UNET network.

Images