Satellite cloud image processing method, device, computer equipment and storage medium

Abstract

The present application provides a satellite cloud layer image processing method, device, computer equipment and storage medium. The method includes performing overlapping segmentation processing on satellite cloud layer images to obtain multiple overlapping cloud layer images, wherein each overlapping cloud layer image has at least one overlapping area; performing convolution classification operation on the same feature layer of two overlapping cloud layer images to obtain the feature probability of each pixel point of each feature layer in the overlapping area; and adjusting the grayscale value of the corresponding pixel point on the output layer according to the feature probability. By performing convolution classification operation on the segmented overlapping cloud layer images, the probability of each pixel point of each segmented image of the satellite cloud layer image in the overlapping area is obtained, and finally, the grayscale value of the corresponding pixel point on the output layer is adjusted according to the probability distribution of the same pixel point of the same feature layer, so as to facilitate the distinction of various background images in the overlapping area, so that the blocking effect in the overlapping area is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud layer image processing, and in particular to a satellite cloud layer image processing method, device, computer equipment and storage medium. Background Art

[0002] With the development of image processing technology, satellite image processing has become an important branch of image processing. For the processing of satellite graphics, the U-net segmentation network model is usually used. The traditional U-net is a segmentation network for medical tissue cell images, which has a good ability to distinguish the semantic information of images.

[0003] However, the receptive field of each feature point in the central part of the traditional U-net is small, and it cannot capture a wide range of semantic information. For some objects with large spatial spans, it is more likely to produce incorrect segmentation results and is not suitable for cloud and cloud shadow detection tasks. Moreover, for satellite images with a large width, the traditional U-net needs to use a block prediction method and then stitch them together to form a complete image. However, when the block image is input into the network model, its boundary semantic information may be incomplete, resulting in a high misjudgment rate of boundary pixels. Finally, the whole image produces a blocking effect, resulting in a decrease in the integrity of the final detection image, and then a decrease in the clarity of the final detection image. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a satellite cloud image processing method, device, computer equipment and storage medium that reduce the probability of blocking effect.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A satellite cloud layer image processing method, the method comprising:

[0007] Performing overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area;

[0008] Performing a convolution classification operation on the same feature layers of the two overlapping cloud images to obtain feature probabilities of each pixel point of each feature layer in the overlapping area;

[0009] The grayscale value of the corresponding pixel point on the output layer is adjusted according to the feature probability.

[0010] In one of the embodiments, the overlapping segmentation processing of the satellite cloud layer images to obtain a plurality of overlapping cloud layer images includes: performing row-wise overlapping segmentation processing on the satellite cloud layer images to obtain a plurality of row-wise overlapping cloud layer images.

[0011] In one of the embodiments, the overlapping segmentation processing of the satellite cloud layer images to obtain a plurality of overlapping cloud layer images includes: performing column-wise overlapping segmentation processing on the satellite cloud layer images to obtain a plurality of column-wise overlapping cloud layer images.

[0012] In one of the embodiments, performing a convolution classification operation on the same feature layer of the two overlapping cloud images includes: performing a residual operation on the feature layer of each overlapping cloud image.

[0013] In one of the embodiments, the residual operation on the feature layers of each overlapping cloud image includes: performing downsampling residual processing and upsampling residual processing on the feature layers respectively to form a plurality of compression path layers and a plurality of expansion path layers.

[0014] In one of the embodiments, the downsampling residual processing and the upsampling residual processing are performed on the feature layer respectively, and then the method further includes: performing a feature fusion operation on the compression path layer and the corresponding extension path layer.

[0015] In one of the embodiments, the convolution classification operation on the same feature layer of the two overlapping cloud images also includes: performing a bidirectional asymmetric dilated convolution operation on the minimum-sized compression path layer on the compression path to output an initial expansion path layer on the expansion path.

[0016] A satellite cloud layer image processing device, the device comprising:

[0017] A first processing module is used to perform overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area;

[0018] The second processing module is used to perform a convolution classification operation on the same feature layers of the two overlapping cloud layer images to obtain the feature probability of each pixel point of each feature layer in the overlapping area;

[0019] The grayscale adjustment module is used to adjust the grayscale value of the corresponding pixel point on the output layer according to the feature probability.

[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Performing overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area;

[0022] Performing a convolution classification operation on the same feature layers of the two overlapping cloud images to obtain feature probabilities of each pixel point of each feature layer in the overlapping area;

[0023] The grayscale value of the corresponding pixel point on the output layer is adjusted according to the feature probability.

[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0025] Performing overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area;

[0026] Performing a convolution classification operation on the same feature layers of the two overlapping cloud images to obtain feature probabilities of each pixel point of each feature layer in the overlapping area;

[0027] The grayscale value of the corresponding pixel point on the output layer is adjusted according to the feature probability.

[0028] Compared with the prior art, the present invention has at least the following advantages:

[0029] By performing convolution classification operations on the segmented overlapping cloud images, the probability of each pixel point in the overlapping area of ​​each segmented image of the satellite cloud image is obtained, which is used to display the probability distribution of the pixel points in the overlapping area. Finally, the grayscale value of the corresponding pixel point on the output layer is adjusted according to the probability distribution of the same pixel point of the same feature layer, which makes it easier to distinguish various background images in the overlapping area, reduces the blocking effect in the overlapping area, and improves the clarity of the detection output image of the satellite cloud image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flow chart of a satellite cloud image processing method in one embodiment;

[0032] Figure 2 4 is a structural diagram of a pyramid pooling module based on a convolution kernel in one embodiment;

[0033] Figure 3 for Figure 2Schematic diagram of volume distribution of bidirectional asymmetric hole convolution based on convolution kernel pyramid pooling module shown;

[0034] Figure 4 1 is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thoroughly understood.

[0036] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only and do not represent the only implementation method.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0038] The present invention relates to a satellite cloud layer image processing method. In one embodiment, the satellite cloud layer image processing method includes performing overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each overlapping cloud layer image has at least one overlapping area; performing convolution classification operation on the same feature layer of the two overlapping cloud layer images to obtain the feature probability of each pixel point of each feature layer in the overlapping area; and adjusting the gray value of the corresponding pixel point on the output layer according to the feature probability. By performing convolution classification operation on the segmented overlapping cloud layer images, the probability of each pixel point of each segmented image of the satellite cloud layer image in the overlapping area is obtained, which is used to show the probability distribution of the occurrence of the pixel point in the overlapping area. Finally, the gray value of the corresponding pixel point on the output layer is adjusted according to the probability distribution of the same pixel point of the same feature layer, so as to facilitate the distinction of various background images in the overlapping area, so that the blocking effect in the overlapping area is reduced, and the clarity of the detection output image of the satellite cloud layer image is improved.

[0039] See also Figure 1 , which is a flow chart of a satellite cloud layer image processing method according to an embodiment of the present invention. The satellite cloud layer image processing method includes part or all of the following steps.

[0040] S100: performing overlapping segmentation processing on satellite cloud layer images to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area.

[0041] In this embodiment, the satellite cloud layer image is a collection of GF-1WFV satellite images, and can also be considered as using GF-1WFV satellite images as the satellite cloud layer image. The satellite cloud layer image includes three images, namely, there are three images of clouds, cloud shadows and background in the satellite cloud layer image. Since the satellite cloud layer image is a satellite image of the earth taken by the Gaofen-1 satellite, the specific position of the cloud layer is mainly collected in real time to distinguish the cloud layer from the mountains, waters, ice and snow on the ground. Due to the different shooting time and angles of the satellite, there will be a certain misalignment between the cloud layer and the cloud shadow. In order to distinguish the above three images, it is necessary to segment the satellite image with a larger width, that is, the overlapping segmentation processing, so that the large-size satellite cloud layer image is divided into multiple sub-images, which is convenient for image processing of the satellite cloud layer image. Among them, in order to reduce the cutting of the boundary semantics of each sub-image, that is, to reduce the probability of the blocking effect, the overlapping segmentation processing of the satellite cloud layer image is to obtain an overlapping cloud layer image with overlapping areas. Moreover, these overlapping areas are all located at the edges of the overlapping cloud images, that is, two adjacent overlapping cloud images overlap at the edge, and the two complement each other's semantics, so that the semantics at the edge remains intact, which facilitates the subsequent probabilistic processing of the pixels in the overlapping area, and then facilitates the subsequent distinction between the three different images in the overlapping area, so as to reduce the probability of blocking effect in the boundary area.

[0042] S200: performing a convolution classification operation on the same feature layers of the two overlapping cloud images to obtain feature probabilities of each pixel point of each feature layer in the overlapping area.

[0043] In this embodiment, the feature layer corresponds to the overlapping area, and the feature layer is a layer formed after feature classification of the overlapping cloud layer images, that is, the feature layer is a layer formed after feature classification operation is performed on the overlapping area of ​​the two overlapping cloud layer images. The convolution classification operation is performed on the same feature layer of the two overlapping cloud layer images, which is to classify the feature layer in the overlapping area at a single point position, so that the probability of each pixel point of the feature layer in the overlapping area can be displayed, which is convenient for outputting the feature probability of each pixel point of each feature layer in the overlapping area, thereby facilitating the reflection of the probability distribution of each pixel point in the overlapping area, and then facilitating the subsequent use as a basis for judging the output grayscale of each pixel point in the overlapping area.

[0044] S300: adjusting the grayscale value of the corresponding pixel point on the output layer according to the feature probability.

[0045] In this embodiment, the feature probability is the occurrence probability of each pixel point in the overlapping area. On the same feature layer of two overlapping cloud images, there are the same pixels in the overlapping area. These same pixels have their own feature probabilities. The feature probabilities of the same pixel point of the same feature layer are fused to facilitate the analysis of the appearance of the image of this pixel point. For example, the feature probability of this pixel point on the feature layer of cloud shadow is the largest, and the grayscale value output at the same pixel point on the output layer is 128, that is, gray; for another example, the feature probability of this pixel point on the feature layer of cloud layer is the largest, and the grayscale value output at the same pixel point on the output layer is 255, that is, white; for another example, the feature probability of this pixel point on the feature layer of mountain, water, ice and snow background is the largest, and the grayscale value output at the same pixel point on the output layer is 0, that is, black.

[0046] In the above embodiments, by performing a convolution classification operation on the segmented overlapping cloud images, the probability of each pixel point of each segmented image of the satellite cloud image in the overlapping area is obtained, which is used to display the probability distribution of the pixel points appearing in the overlapping area. Finally, the grayscale value of the corresponding pixel point on the output layer is adjusted through the probability distribution of the same pixel point of the same feature layer, so as to facilitate the distinction of various background images in the overlapping area, reduce the blocking effect in the overlapping area, and improve the clarity of the detection output image of the satellite cloud image.

[0047] In one of the embodiments, the overlapping segmentation processing of the satellite cloud layer image to obtain multiple overlapping cloud layer images includes: performing row-wise overlapping segmentation processing on the satellite cloud layer image to obtain multiple row-wise overlapping cloud layer images. In this embodiment, the satellite cloud layer image is a satellite image with a large width, that is, the size of the satellite cloud layer image is large. In order to improve the efficiency of image processing, the satellite cloud layer image is segmented into multiple overlapping cloud layer images for separate processing, which not only reduces the amount of data for processing the image at one time, but also improves the processing efficiency of the satellite cloud layer image. Here, the row-wise overlapping segmentation processing of the satellite cloud layer image is to arrange the separated multiple overlapping cloud layer images in the row direction, and the overlapping area between the two adjacent overlapping cloud layer images is in the column direction of the image, that is, the overlapping area is on both sides of the overlapping cloud layer image, so that the semantics of the side images of the overlapping cloud layer image remain intact, which is convenient for reducing the probability of the subsequent block effect after the overlapping cloud layer images are spliced.

[0048] In one of the embodiments, the overlapping segmentation processing of the satellite cloud layer image to obtain a plurality of overlapping cloud layer images includes: performing column-wise overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of column-wise overlapping cloud layer images. In this embodiment, the satellite cloud layer image is a satellite image with a large width, that is, the size of the satellite cloud layer image is large. In order to improve the efficiency of image processing, the satellite cloud layer image is segmented into a plurality of overlapping cloud layer images for separate processing, which not only reduces the amount of data of the image processed at one time, but also improves the processing efficiency of the satellite cloud layer image. Here, the column-wise overlapping segmentation processing of the satellite cloud layer image is to arrange the separated multiple overlapping cloud layer images in the column direction, and the overlapping area between the two adjacent overlapping cloud layer images is in the row direction of the image, that is, the overlapping area is at both ends of the overlapping cloud layer image, so that the semantics of the end images of the overlapping cloud layer image remain intact, which is convenient for reducing the probability of the block effect after the overlapping cloud layer images are subsequently spliced. In another embodiment, the overlapping segmentation processing includes row-wise overlapping segmentation processing and column-wise overlapping segmentation processing, so that each side of the overlapping cloud layer image has an overlapping area, further reducing the probability of blocking effects in the edge overlapping areas of each of the overlapping cloud layer images.

[0049] In one of the embodiments, the convolution classification operation is performed on the same feature layers of the two overlapping cloud images, including: performing a residual operation on the feature layers of each overlapping cloud image. In this embodiment, the feature layers of the overlapping cloud images are subjected to network classification by a residual module, and the residual module is a multi-layer convolution classification of overlapping cloud images, that is, the number of residual modules used is multiple, and the number of channels of the residual module is specifically adjusted according to actual needs, so as to facilitate the transmission of semantic information on different feature layers. The residual module has the function of loading the input semantic information into the output semantic information, so that the residual module effectively transmits the shallow network information to the deep network information, which is convenient for maintaining and transmitting the semantic information completely, and reducing the loss of semantic space detail information.

[0050] Furthermore, the residual operation on the feature layers of each overlapping cloud image includes: performing downsampling residual processing and upsampling residual processing on the feature layers, respectively, to form multiple compression path layers and multiple expansion path layers. In this embodiment, the downsampling residual processing is to perform a downsampling operation on the feature layers of the overlapping cloud images, and to perform image compression on the feature layers of the overlapping cloud images, and the upsampling residual processing is to perform an upsampling operation on the output layers of the downsampling residual processing, and to perform image expansion on the output layers of the downsampling residual processing. In this way, under the action of the residual module, in addition to the role of reducing / raising the dimension in the downsampling and upsampling processes, the integrity of the semantic information is also ensured, and the loss of semantic information in the overlapping area is reduced.

[0051] Furthermore, the feature layer is subjected to downsampling residual processing and upsampling residual processing respectively, and then further includes: performing feature fusion operation on the compression path layer and the corresponding extension path layer. In the present embodiment, the feature fusion operation is the fusion of each feature layer. In the process of downsampling, each downsampling residual module outputs a compression path layer, and in the process of upsampling, each upsampling residual module outputs an extension path layer. The compression path layer and the corresponding extension path layer are feature fused. For example, the number of downsampling residual modules is equal to the number of upsampling residual modules. The compression path layer output by the first downsampling residual module is fused with the extension path layer output by the last upsampling residual module. The compression path layer output by the second downsampling residual module is fused with the extension path layer output by the second to last upsampling residual module. By analogy, each compression path layer is feature fused with an extension path layer to ensure the integrity of the semantic information of the overlapping area.

[0052] Furthermore, the convolution classification operation on the same feature layer of the two overlapping cloud images also includes: performing a bidirectional asymmetric dilated convolution operation on the compression path layer of the minimum size on the compression path to output the initial extended path layer on the extended path. In the present embodiment, the compression path layer of the minimum size on the compression path is the feature layer finally output by each downsampling residual module on the compression path, and is used to reduce the dimension of the overlapping cloud image to a small-sized layer. The bidirectional asymmetric dilated convolution operation is a convolution sublayer of a pyramid pooling module based on a convolution kernel. The pyramid pooling module based on a convolution kernel is an improved model of a dilated space pyramid module. The main improvement is the improvement of the convolution model, that is, a bidirectional asymmetric dilated convolution (DADC, Double-side Asymmetric Dilated Convolution) model is adopted, wherein the model includes a bidirectional asymmetric dilated convolution module combined in series and parallel, a global average pooling module and a 1*1 convolution module. The structural diagram of the pyramid pooling module based on a convolution kernel is shown in detail. Figure 2 While ensuring that the spatial resolution of the features remains unchanged, the receptive field of the central part of the network is greatly expanded, and features of different depths and widths are fused at the same time, so that the final feature map has both a large enough receptive field and rich multi-dimensional semantic information, and the feature scale remains unchanged without losing relative information in space.

[0053] The bidirectional asymmetric dilated convolution formula is defined as follows:

[0054]

[0055] Among them, the convolution kernel size is K1×K2, i and j are independent variables, that is, the number of rows and columns corresponding to the matrix Y, X is the input feature of each pixel of the feature layer, r1 and r2 represent the expansion rate of the two dimensions of the feature space respectively, w represents the weight value, k1 takes any value between 0 and K1, and k2 takes any value between 0 and K2. Moreover, set K1=K2, r1=1 or r2=1, there are two complementary convolution kernels in the bidirectional asymmetric hole convolution module. When the expansion rate of one convolution kernel is r1=1, r2=r, the expansion rate of the other convolution kernel is r1=r, r2=1, and the output features of the two complementary convolution kernels are fused and added to extract richer spatial relationship information. For specific convolution distribution, please refer to Figure 3 .

[0056] The bidirectional asymmetric hole convolution of series and parallel, on the one hand, fuses and adds the dilated convolutions of different dimensions, multiplying the network receptive field without reducing the resolution of the feature map to capture richer global semantic information. On the other hand, the series-parallel structure fuses the feature information of different depths and widths together in a stacking manner to achieve multi-scale feature fusion. Among them, the decoder part uses transposed convolution for upsampling to restore the semantic label map of the same scale as the original image. The cross-connection part is different from the U-net network in splicing and fusing the feature map of the encoder with the feature map of the decoder. This application uses the addition and fusion of corresponding feature points, which can reduce model parameters and reduce the amount of image processing parameters, thereby improving the efficiency of image processing. Moreover, cross entropy loss is a commonly used target optimization function in semantic segmentation tasks. Cross entropy loss is a loss criterion for semantic information in overlapping segmentation processing, that is, it is used to reflect the degree of semantic loss in overlapping areas. Its expression is as follows:

[0057]

[0058] Among them, l is the true label of each pixel, l:Ω→{1,2,…K}, and w represents the weight value, which is used to measure the importance of different pixels. This application will give the cloud shadow category a larger penalty weight to make up for its insufficient number in the training data. P is the result of the softmax normalization of the feature map of the last layer of the network. The specific expression is as follows:

[0059]

[0060] Among them, a k' (x) is the response value of the pixel x at the k'th layer of the feature map, and K is the number of categories.

[0061] Before training the model, data enhancement methods are generally used to expand the number of samples and improve the generalization ability of the model. Since satellite cloud images are "isotropic" and there is no so-called up, down, left, or right distinction, data can be effectively amplified by rotating and flipping the image. At the same time, since satellite cloud images are taken in a bird's-eye view, most objects can be stretched and extended while still keeping their semantics unchanged. Moreover, satellite cloud images will have some changes in brightness due to different shooting times, and the surface coverage of each location is quite different. Therefore, this application will also use color amplification methods to enhance the input samples.

[0062] In the actual processing process, the image width of satellite cloud images is relatively large. For example, a panoramic image of GF-1 has a size of about 15000×15000. The specific processing method is as follows:

[0063] 1. Segment the satellite cloud image into multiple 512×512 overlapping cloud images with overlapping areas;

[0064] 2. Pass the overlapping cloud images through multiple down-sampling residual modules, and fuse the output feature layer of each down-sampling residual module with the corresponding output feature layer of the up-sampling residual module;

[0065] 3. Input the feature layer output by the last down-sampling residual module into the bidirectional asymmetric dilated convolution model to obtain three feature layers of each overlapping cloud image, namely, cloud feature layer, cloud shadow feature layer and background feature layer. The background feature layer includes ice and snow layer, water layer and terrain shadow layer.

[0066] 4. Obtain the feature probability of each pixel of each feature layer after softmax normalization, add the feature probabilities of the same pixel of different overlapping cloud images on the same feature layer, and obtain the probability of each pixel on different feature layers;

[0067] 5. Select the category with the maximum probability on each feature layer as the judgment category for each pixel in the overlapping area. For example, if the probability of a pixel on the feature layer of cloud shadow is the highest, the grayscale value output at the same pixel on the output layer is 128, that is, gray; for another example, if the probability of a pixel on the feature layer of cloud layer is the highest, the grayscale value output at the same pixel on the output layer is 255, that is, white; for another example, if the probability of a pixel on the feature layer of mountain, water, ice and snow background is the highest, the grayscale value output at the same pixel on the output layer is 0, that is, black.

[0068] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0069] The present application also provides a satellite cloud layer image processing device, which is implemented by the satellite cloud layer image processing method described in any of the above embodiments. In one embodiment, the satellite cloud layer image processing device has a functional module corresponding to each step of the satellite cloud layer image processing method. The satellite cloud layer image processing device includes a first processing module, a second processing module and a grayscale adjustment module, wherein:

[0070] A first processing module is used to perform overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area;

[0071] The second processing module is used to perform a convolution classification operation on the same feature layers of the two overlapping cloud layer images to obtain the feature probability of each pixel point of each feature layer in the overlapping area;

[0072] The grayscale adjustment module is used to adjust the grayscale value of the corresponding pixel point on the output layer according to the feature probability.

[0073] In this embodiment, the second processing module performs a convolution classification operation on the segmented overlapping cloud layer images, and obtains the probability of each pixel point of each segmented image of the satellite cloud layer image in the overlapping area, which is used to display the probability distribution of the pixel points appearing in the overlapping area. Finally, the grayscale adjustment module adjusts the grayscale value of the corresponding pixel point on the output layer through the probability distribution of the same pixel point of the same feature layer, so as to distinguish various background images in the overlapping area, reduce the blocking effect in the overlapping area, and improve the clarity of the detection output image of the satellite cloud layer image.

[0074] In one embodiment, the first processing module is also used to perform row-wise overlapping separation processing on the satellite cloud layer image to obtain multiple row-wise overlapping cloud layer images. In this embodiment, the satellite cloud layer image is a satellite image with a large width, that is, the size of the satellite cloud layer image is large. In order to improve the efficiency of image processing, the first processing module adopts the method of segmenting the image to divide the satellite cloud layer image into multiple overlapping cloud layer images for separate processing, which not only reduces the amount of data of the image processed at one time, but also improves the processing efficiency of the satellite cloud layer image. Here, the first processing module performs row-wise overlapping separation processing on the satellite cloud layer image, which is to arrange the separated multiple overlapping cloud layer images in the row direction, and the overlapping area between two adjacent overlapping cloud layer images is in the column direction of the image, that is, the overlapping area is on both sides of the overlapping cloud layer image, so that the semantics of the side image of the overlapping cloud layer image remains intact, which is convenient for reducing the probability of the subsequent block effect after the overlapping cloud layer images are spliced.

[0075] In one embodiment, the first processing module is also used to perform column-wise overlapping separation processing on the satellite cloud layer image to obtain multiple column-wise overlapping cloud layer images. In this embodiment, the satellite cloud layer image is a satellite image with a large width, that is, the size of the satellite cloud layer image is large. In order to improve the efficiency of image processing, the first processing module adopts the method of segmenting the image to divide the satellite cloud layer image into multiple overlapping cloud layer images for separate processing, which not only reduces the amount of data of the image processed at one time, but also improves the processing efficiency of the satellite cloud layer image. Here, the first processing module performs column-wise overlapping separation processing on the satellite cloud layer image, which is to arrange the separated multiple overlapping cloud layer images in the column direction, and the overlapping area between two adjacent overlapping cloud layer images is in the row direction of the image, that is, the overlapping area is at both ends of the overlapping cloud layer image, so that the semantics of the end images of the overlapping cloud layer image remain intact, which is convenient for reducing the probability of the subsequent block effect after the overlapping cloud layer images are spliced. In another embodiment, the first processing module includes a row-wise overlapping separation processing module and a column-wise overlapping separation processing module, so that each side of the overlapping cloud layer image has an overlapping area, further reducing the probability of a blocking effect in the edge overlapping area of ​​each overlapping cloud layer image.

[0076] In one of the embodiments, the second processing module is also used to perform residual operations on the feature layers of each of the overlapping cloud images. In this embodiment, the second processing module performs network classification on the feature layers of the overlapping cloud images through the residual module, and the residual module is a multi-layer convolution classification of the overlapping cloud images, that is, the number of residual modules used is multiple, and the number of channels of the residual module is specifically adjusted according to actual needs to facilitate the transmission of semantic information on different feature layers. The residual module has the function of loading the input semantic information into the output semantic information, so that the residual module effectively transmits the shallow network information to the deep network information, which is convenient for maintaining and transmitting the semantic information completely, and reducing the loss of semantic space detail information.

[0077] Furthermore, the second processing module is also used to perform downsampling residual processing and upsampling residual processing on the feature layer, respectively, to form multiple compression path layers and multiple expansion path layers. In this embodiment, the downsampling residual processing is to perform a downsampling operation on the feature layer of the overlapping cloud layer image, and to perform image compression on the feature layer of the overlapping cloud layer image, and the upsampling residual processing is to perform an upsampling operation on the output layer of the downsampling residual processing, and to perform image expansion on the output layer of the downsampling residual processing. In this way, under the action of the residual module, in addition to the role of reducing / raising the dimension in the downsampling and upsampling processes, the integrity of the semantic information is also ensured, and the loss of semantic information in the overlapping area is reduced.

[0078] Furthermore, the second processing module is also used to perform feature fusion operation on the compression path layer and the corresponding extension path layer. In the present embodiment, the feature fusion operation adopted by the second processing module is the fusion of each feature layer. In the process of downsampling, each downsampling residual module outputs a compression path layer, and in the process of upsampling, each upsampling residual module outputs an extension path layer. The compression path layer and the corresponding extension path layer are feature fused. For example, the number of downsampling residual modules is equal to the number of upsampling residual modules. The compression path layer output by the first downsampling residual module is fused with the extension path layer output by the last upsampling residual module. The compression path layer output by the second downsampling residual module is fused with the extension path layer output by the second to last upsampling residual module. And so on. Each compression path layer is feature fused with an extension path layer to ensure the integrity of the semantic information of the overlapping area.

[0079] Furthermore, the second processing module is also used to perform a bidirectional asymmetric dilated convolution operation on the compression path layer of the minimum size on the compression path to output the initial extension path layer on the extension path. In the present embodiment, the compression path layer of the minimum size on the compression path of the second processing module is the feature layer finally output by each downsampling residual module on the compression path, and is used to reduce the dimension of the overlapping cloud image to a small-sized layer. The bidirectional asymmetric dilated convolution operation adopted by the second processing module is a convolution sublayer of the pyramid pooling module based on the convolution kernel. The pyramid pooling module based on the convolution kernel is an improved model of the dilated space pyramid module. The main improvement is the improvement of the convolution model, that is, the bidirectional asymmetric dilated convolution (DADC, Double-side Asymmetric DilatedConvolution) model is adopted, wherein the model includes a bidirectional asymmetric dilated convolution module combined in series and parallel, a global average pooling module and a 1*1 convolution module. The structural diagram of the pyramid pooling module based on the convolution kernel is shown in detail. Figure 2 While ensuring that the spatial resolution of the features remains unchanged, the receptive field of the central part of the network is greatly expanded, and features of different depths and widths are fused at the same time, so that the final feature map has both a large enough receptive field and rich multi-dimensional semantic information, and the feature scale remains unchanged without losing relative information in space.

[0080] The bidirectional asymmetric dilated convolution formula is defined as follows:

[0081]

[0082] Among them, the convolution kernel size is K1×K2, i and j are independent variables, that is, the number of rows and columns corresponding to the matrix Y, X is the input feature of each pixel of the feature layer, r1 and r2 represent the expansion rate of the two dimensions of the feature space respectively, w represents the weight value, k1 takes any value between 0 and K1, and k2 takes any value between 0 and K2. Moreover, set K1=K2, r1=1 or r2=1, there are two complementary convolution kernels in the bidirectional asymmetric hole convolution module. When the expansion rate of one convolution kernel is r1=1, r2=r, the expansion rate of the other convolution kernel is r1=r, r2=1, and the output features of the two complementary convolution kernels are fused and added to extract richer spatial relationship information. For specific convolution distribution, please refer to Figure 3 .

[0083] The bidirectional asymmetric hole convolution of series and parallel, on the one hand, fuses and adds the dilated convolutions of different dimensions, multiplying the network receptive field without reducing the resolution of the feature map to capture richer global semantic information. On the other hand, the series-parallel structure fuses the feature information of different depths and widths together in a stacking manner to achieve multi-scale feature fusion. Among them, the decoder part uses transposed convolution for upsampling to restore the semantic label map of the same scale as the original image. The cross-connection part is different from the U-net network in splicing and fusing the feature map of the encoder with the feature map of the decoder. This application uses the addition and fusion of corresponding feature points, which can reduce model parameters and reduce the amount of image processing parameters, thereby improving the efficiency of image processing. Moreover, cross entropy loss is a commonly used target optimization function in semantic segmentation tasks. Cross entropy loss is a loss criterion for semantic information in overlapping segmentation processing, that is, it is used to reflect the degree of semantic loss in overlapping areas. Its expression is as follows:

[0084]

[0085] Among them, l is the true label of each pixel, l:Ω→{1,2,…K}, and w represents the weight value, which is used to measure the importance of different pixels. This application will give the cloud shadow category a larger penalty weight to make up for its insufficient number in the training data. P is the result of the softmax normalization of the feature map of the last layer of the network. The specific expression is as follows:

[0086]

[0087] Among them, a k' (x) is the response value of the pixel x at the k'th layer of the feature map, and K is the number of categories.

[0088] Before training the model, data enhancement methods are generally used to expand the number of samples and improve the generalization ability of the model. Since satellite cloud images are "isotropic" and there is no so-called up, down, left, or right distinction, data can be effectively amplified by rotating and flipping the image. At the same time, since satellite cloud images are taken in a bird's-eye view, most objects can be stretched and extended while still keeping their semantics unchanged. Moreover, satellite cloud images will have some changes in brightness due to different shooting times, and the surface coverage of each location is quite different. Therefore, this application will also use color amplification methods to enhance the input samples.

[0089] In the actual processing process, the image width of satellite cloud images is relatively large. For example, a panoramic image of GF-1 has a size of about 15000×15000. The specific processing method is as follows:

[0090] 1. Segment the satellite cloud image into multiple 512×512 overlapping cloud images with overlapping areas;

[0091] 2. Pass the overlapping cloud images through multiple down-sampling residual modules, and fuse the output feature layer of each down-sampling residual module with the corresponding output feature layer of the up-sampling residual module;

[0092] 3. Input the feature layer output by the last down-sampling residual module into the bidirectional asymmetric dilated convolution model to obtain three feature layers of each overlapping cloud image, namely, cloud feature layer, cloud shadow feature layer and background feature layer. The background feature layer includes ice and snow layer, water layer and terrain shadow layer.

[0093] 4. Obtain the feature probability of each pixel of each feature layer after softmax normalization, add the feature probabilities of the same pixel of different overlapping cloud images on the same feature layer, and obtain the probability of each pixel on different feature layers;

[0094] 5. Select the category with the maximum probability on each feature layer as the judgment category for each pixel in the overlapping area. For example, if the probability of a pixel on the feature layer of cloud shadow is the highest, the grayscale value output at the same pixel on the output layer is 128, that is, gray; for another example, if the probability of a pixel on the feature layer of cloud layer is the highest, the grayscale value output at the same pixel on the output layer is 255, that is, white; for another example, if the probability of a pixel on the feature layer of mountain, water, ice and snow background is the highest, the grayscale value output at the same pixel on the output layer is 0, that is, black.

[0095] For the specific definition of the satellite cloud layer image processing device, please refer to the definition of the satellite cloud layer image processing method above, which will not be repeated here. Each module in the above-mentioned satellite cloud layer image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0096] The present application also provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in Figure 4 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a satellite cloud layer image processing method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0097] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0098] In one of the embodiments, the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0099] In one of the embodiments, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0100] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0101] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A satellite cloud image processing method, characterized in that: include: Performing overlapping segmentation processing on the satellite cloud layer image to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area; Performing a convolution classification operation on the same feature layers of the two overlapping cloud images to obtain feature probabilities of each pixel point of each feature layer in the overlapping area; Adjust the grayscale value of the corresponding pixel point on the output layer according to the feature probability; The overlapping segmentation process of the satellite cloud layer images to obtain a plurality of overlapping cloud layer images includes: Performing row-wise overlapping separation processing on the satellite cloud layer image to obtain a plurality of row-wise overlapping cloud layer images; Furthermore, the satellite cloud layer images are subjected to column-wise overlapping separation processing to obtain a plurality of column-wise overlapping cloud layer images.

2. The satellite cloud image processing method according to claim 1, characterized in that: The performing a convolution classification operation on the same feature layer of the two overlapping cloud layer images includes: A residual operation is performed on the feature layers of each of the overlapping cloud layer images.

3. The satellite cloud image processing method according to claim 2, characterized in that: The performing a residual operation on the feature layers of each of the overlapping cloud layer images includes: Downsampling residual processing and upsampling residual processing are performed on the feature layers respectively to form multiple compression path layers and multiple expansion path layers.

4. The satellite cloud image processing method according to claim 3, characterized in that: The down-sampling residual processing and up-sampling residual processing are respectively performed on the feature layer, and then further includes: A feature fusion operation is performed on the compression path layer and the corresponding expansion path layer.

5. The satellite cloud image processing method according to claim 3, characterized in that: The performing of the convolution classification operation on the same feature layers of the two overlapping cloud layer images further includes: A bidirectional asymmetric dilated convolution operation is performed on the minimum-sized compression path layer on the compression path to output an initial expansion path layer on the expansion path.

6. A satellite cloud image processing device, characterized in that: The device comprises: The first processing module is used to perform overlapping segmentation processing on the satellite cloud layer images to obtain a plurality of overlapping cloud layer images, wherein each of the overlapping cloud layer images has at least one overlapping area, and the satellite cloud layer images are overlapped and segmented to obtain a plurality of overlapping cloud layer images: Performing row-wise overlapping separation processing on the satellite cloud layer image to obtain a plurality of row-wise overlapping cloud layer images; and performing column-wise overlapping separation processing on the satellite cloud layer image to obtain a plurality of column-wise overlapping cloud layer images; The second processing module is used to perform a convolution classification operation on the same feature layers of the two overlapping cloud layer images to obtain the feature probability of each pixel point of each feature layer in the overlapping area; The grayscale adjustment module is used to adjust the grayscale value of the corresponding pixel point on the output layer according to the feature probability.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images