Cloud image processing method and device, electronic equipment and storage medium

Abstract

The application discloses a cloud image processing method and device, electronic equipment and storage medium, and belongs to the technical field of image processing. The cloud image processing method comprises the following steps: acquiring a to-be-processed cloud image; performing semantic segmentation on the to-be-processed cloud image by using a deep learning network to obtain a segmented image, wherein the segmented image comprises a first region, and the first region comprises a cloud cluster and a sun; performing binaryzation processing on the to-be-processed cloud image to obtain a binary image; determining a second region corresponding to a sun contour according to a white region in the binary image; and remapping the second region to the first region in the segmented image to obtain a target image. The cloud cluster region is segmented in the same region as the sun by using the deep learning mode, and the segmented image is obtained; then the sun region with obvious features is extracted by directly processing the image; finally, the cloud cluster and the sun in the segmented image are segmented by using the sun region, so that the cloud cluster, the sun and the region outside the two can be more accurately segmented.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a cloud image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Clouds cover more than 50% of the Earth's surface and play a vital role in the Earth's water cycle and energy balance. Cloud images have significant applications in short-term weather forecasting, precipitation prediction, real-time rainfall forecasting, air traffic control, artificial rain enhancement, and photovoltaic power generation.

[0003] The main elements in a cloud image include the sun, clouds, and the sky beyond the sun and clouds. Because the shape of clouds in an image is easily affected by wind, lighting, and other factors, it is difficult to segment these elements. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a cloud image processing method, apparatus, electronic device, and storage medium, wherein the cloud image processing method can more accurately segment clouds, the sun, and areas outside of both.

[0005] In a first aspect, this application provides a cloud image processing method, comprising: acquiring a cloud image to be processed; using a deep learning network to perform semantic segmentation on the cloud image to be processed to obtain a segmented image, the segmented image including a first region, the first region including clouds and the sun; performing binarization processing on the cloud image to be processed to obtain a binary image; determining a second region corresponding to the sun's outline based on the white region in the binary image; and remapping the second region onto the first region in the segmented image to obtain a target image.

[0006] According to the cloud image processing method of this application, a segmented image is obtained by using deep learning to segment cloud regions with diverse shapes and sizes into the same category as the sun; then, the sun region with significant features is extracted by directly processing the image; finally, the sun region is used to segment the cloud and sun in the segmented image, thereby more accurately segmenting the cloud, the sun and the regions outside of them.

[0007] According to one embodiment of this application, determining a second region corresponding to the sun's outline based on a white region in a binary image includes: performing edge detection on the white region in the binary image to obtain edge features; performing circular detection on the edge features to obtain circular features; and combining the circular features and the edge features and taking their intersection to obtain a second region corresponding to the sun's outline.

[0008] According to one embodiment of this application, performing circular detection on edge features to obtain circular features includes: filtering the edge features to obtain filtered edge features; performing circular detection on the filtered edge features to obtain circular features; and combining the circular features with the edge features and taking their intersection, which includes: combining the circular features with the filtered edge features and taking their intersection.

[0009] According to one embodiment of this application, binarization processing of the cloud image to be processed includes: performing erosion processing on the cloud image to be processed to obtain an eroded image; and sequentially performing grayscale and binarization processing on the eroded image.

[0010] According to one embodiment of this application, semantic segmentation of a cloud image to be processed using a deep learning network includes: extracting features from the cloud image to be processed using a convolutional neural network and outputting a feature map with the same scale as the cloud image to be processed; classifying each pixel in the feature map to identify a first region including clouds and the sun, thereby obtaining a segmented image.

[0011] According to one embodiment of this application, classifying each pixel in a feature map to identify a first region including clouds and the sun includes: labeling each pixel in the feature map with a first label and a second label to obtain a labeled feature map, wherein the first label corresponds to cloud and sun features, and the second label corresponds to features other than cloud and sun features; and performing binary classification on each pixel in the labeled feature map to obtain a segmented image.

[0012] According to one embodiment of this application, the method of using a convolutional neural network to extract features from a cloud image to be processed and outputting a feature map with the same scale as the cloud image to be processed includes: extracting image features from the cloud image to be processed using a residual convolutional layer to obtain a first feature map; extracting features from the first feature map using a dilated spatial pyramid pooling layer and concatenating the extracted features in the depth dimension to obtain a second feature map; processing the second feature map sequentially using a transposed convolutional layer and a dilated convolutional layer to obtain a third feature map; and transmitting the third feature map to the output layer to output a feature map with the same scale as the cloud image to be processed.

[0013] According to one embodiment of this application, obtaining a cloud map to be processed includes: obtaining a foundation cloud map; and cropping a target area in the foundation cloud map based on a cropping box to obtain the cloud map to be processed.

[0014] Secondly, this application provides a cloud image processing apparatus, comprising: an image acquisition module for acquiring a cloud image to be processed; a neural network module for performing semantic segmentation on the cloud image to be processed using a deep learning network to obtain a segmented image, the segmented image including a first region, the first region including clouds and the sun; a binary module for performing binarization processing on the cloud image to be processed to obtain a binary image; a segmentation module for determining a second region corresponding to the sun's outline based on the white region in the binary image; and a mapping module for remapping the second region to the first region in the segmented image to obtain a target image.

[0015] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the cloud image processing methods described in the above embodiments.

[0016] Fourthly, this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cloud image processing method as described in any of the above embodiments.

[0017] According to the cloud image processing apparatus, electronic device, and non-transitory computer-readable storage medium of this application, a segmented image is obtained by using deep learning to segment cloud regions with diverse shapes and forms into the same type of region as the sun; then, the sun region with significant features is extracted by directly processing the image; finally, the sun region is used to segment the cloud and sun in the segmented image, thereby more accurately segmenting the cloud, the sun, and the regions outside of both.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0020] Figure 1 This is a schematic flowchart of the cloud image processing method provided in the embodiments of this application;

[0021] Figure 2 This is a cloud map to be processed provided in an embodiment of this application;

[0022] Figure 3 This is a segmented image provided in an embodiment of this application;

[0023] Figure 4 This is one of the binary images provided in the embodiments of this application;

[0024] Figure 5 This is a target image provided in an embodiment of this application;

[0025] Figure 6 This is a schematic flowchart of the solar region extraction method provided in the embodiments of this application;

[0026] Figure 7 This is a second type of binary image provided in the embodiments of this application;

[0027] Figure 8 This is one of the edge detection maps provided in the embodiments of this application;

[0028] Figure 9 This is a second edge detection map provided in the embodiments of this application;

[0029] Figure 10 This is a circular detection pattern provided in an embodiment of this application;

[0030] Figure 11 This is a schematic flowchart of the binarization processing method provided in the embodiments of this application;

[0031] Figure 12 This is a corrosion treatment diagram provided in an embodiment of this application;

[0032] Figure 13 This is one of the flowcharts illustrating the deep learning network processing method provided in the embodiments of this application;

[0033] Figure 14 This is a second schematic flowchart of the deep learning network processing method provided in the embodiments of this application;

[0034] Figure 15 This is a flowchart illustrating the method for obtaining a cloud image to be processed provided in an embodiment of this application;

[0035] Figure 16 This is a ground cloud map provided in an embodiment of this application;

[0036] Figure 17 This is a schematic diagram of the cloud image processing device provided in the embodiments of this application;

[0037] Figure 18 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0039] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0040] In related technologies, ground-based cloud image segmentation mainly takes three approaches: First, it calculates the ratio of the R channel to the B channel in the RGB color space, and then selects an appropriate threshold based on the ratio to determine the regions representing the sun, clouds, and non-sun / cloud areas. Second, it adaptively obtains a series of superpixels based on the characteristics of the sun and clouds, then calculates a local threshold for each superpixel to determine its category, and finally sets a threshold matrix for the entire image. Third, it normalizes the RB value, and then determines whether the ratio map is unimodal or bimodal based on the standard deviation. Finally, unimodal and bimodal images are processed by fixed and least cross-entropy (MCE) thresholding algorithms, respectively. However, these methods are quite sensitive to the selection of algorithm parameters, lack robustness, and require researchers to perform extensive feature engineering to achieve good results when the foreground and background of the image are complex.

[0041] This application provides a cloud image processing method, cloud image processing device, electronic device, and non-transitory computer-readable storage medium. By combining traditional image processing and deep learning, deep learning is used to segment cloud regions with varying shapes, wherein the cloud and sun in the segmented image obtained by deep learning are located in the same region; then, the sun region is segmented using traditional image processing methods, and finally, the sun region is used to segment the cloud and sun in the segmented image, thereby more accurately segmenting the cloud, the sun, and the regions outside of them.

[0042] Reference Figure 1 In some embodiments of this application, the cloud map processing method may include:

[0043] Step 100: Obtain the cloud image to be processed;

[0044] Step 200: Use a deep learning network to perform semantic segmentation on the cloud image to be processed to obtain a segmented image. The segmented image includes a first region, which includes clouds and the sun.

[0045] Step 300: Perform binarization on the cloud image to be processed to obtain a binary image;

[0046] Step 400: Determine the second region corresponding to the sun's outline based on the white region in the binary image;

[0047] Step 500: Remap the second region onto the first region in the segmented image to obtain the target image.

[0048] The execution subject of the cloud map processing method provided in this application embodiment can be an electronic device or a functional module or functional entity in an electronic device that can implement the cloud map processing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile terminals, computers or servers. The cloud map processing method provided in this application embodiment is described below using an electronic device as the execution subject as an example.

[0049] It should be noted that the cloud image to be processed can be a ground-based cloud image. Ground-based cloud images are typically obtained by taking pictures of the sky using an all-sky ground imager. An all-sky ground imager can be a high-definition camera, also known as a fisheye camera. Ground-based cloud images usually include the sun, clouds, and the sky excluding the sun and clouds, and may also include buildings, animals, or aircraft exposed within the range of the fisheye camera's view.

[0050] In some embodiments, the electronic device is connected to an all-sky ground imager, which can capture ground-based cloud images at set time intervals and send the captured ground-based cloud images to the electronic device. The electronic device receives the ground-based cloud images sent by the all-sky ground imager to obtain a cloud image to be processed.

[0051] Semantic segmentation of cloud images involves dividing pixels within the image based on their respective representations, thereby segmenting the different content within the cloud image. Since clouds vary greatly in shape, deep learning networks can be used to segment them more accurately.

[0052] In this embodiment, the deep learning network segments the clouds and the sun within the same content. Since the sun is surrounded by a halo, and semantic segmentation struggles to distinguish between the sun and clouds when they partially overlap, segmenting them within the same content improves the accuracy of semantic segmentation.

[0053] Reference Figure 2 and Figure 3 , Figure 2 A cloud map to be processed is shown. Figure 3 An image segmentation is shown, in which Figure 2 and Figure 3 All images can be in color. In the absence of other interference, the cloud image to be processed includes the sky, clouds, and the sun. The sky is typically blue and its features are more distinct than those of the clouds and the sun, making it easier to segment.

[0054] In some embodiments, the segmented image typically includes two regions, one of which includes clouds and the sun, and the other is the sky. Figure 3 The white area represents the first region, which includes clouds and the sun. Of course, deep learning networks can also use other colors to distinguish these two regions, such as blue and white, black and white, or red and green.

[0055] In some embodiments, when training a deep learning network, clouds and the sun in a sample image are labeled as a first label, and regions other than clouds and the sun are labeled as a second label, so that the trained deep learning network can segment the cloud image to be processed into two regions.

[0056] In some embodiments, the cloud image to be processed is a color image. Therefore, the process of binarizing the cloud image to be processed includes: first converting the cloud image to grayscale, and then binarizing the grayscale image. Taking the color image as an example in RGB format, grayscale conversion can be performed according to the following formula:

[0057] Gray=R*0.299+G*0.587+B*0.114

[0058] Where Gray is the grayscale value, R is the value of the pixel's R channel, G is the value of the pixel's G channel, and B is the value of the pixel's B channel. Of course, the weight values ​​in the above formula are merely examples and can be set to other values ​​as needed; this implementation does not impose any restrictions on this.

[0059] In some implementations, the conversion from a grayscale image to a binary image can also be performed using the following formula:

[0060]

[0061] Wherein, Binary represents the binary value of a pixel. In this formula, a grayscale value of 128 is used as the threshold to convert the grayscale image. When the grayscale value of a pixel is less than or equal to 128, the binary value of that pixel is 0; when the grayscale value of a pixel is greater than 128, the binary value of that pixel is 1. Of course, this threshold can be set to other values ​​as needed, such as 50, 100, or 150; this implementation does not impose any restrictions on this.

[0062] Reference Figure 4 , Figure 4 A binary image is shown. The white area in the binary image is essentially equivalent to the sun area. Since the pixels of the sun area are close to white, the clouds are grayish, and the sky is bluish, the sun area can be separated from other areas after binary processing by controlling the binarization threshold.

[0063] In some embodiments, determining the second region corresponding to the sun's outline based on the white region in the binary image can be achieved by filtering the binary image to remove burrs and outliers, and using the white region in the processed binary image as the second region, the outline of which is the outline of the sun region.

[0064] It is understandable that the interference from the solar halo may cause pixel values ​​near the sun in the cloud image to be similar to those of the sun, resulting in discrete white spots or spikes in the binary image. Further optimization of the binary image can make the white areas in the binary image more closely resemble the actual sun area.

[0065] Remapping refers to the process of placing a pixel at a specific location in one image onto a specified location in another image. By remapping a second region onto a first region in a segmented image, where the sun region is identified, the sun region and the cloud region are separated.

[0066] Reference Figure 5 , Figure 5 A target image is shown. The target image is formed by further subdividing the first region in the segmented image, using the segmented image as a reference. In the target image, region a is the sun region, region b is the cloud region, and region c is the sky region.

[0067] According to the cloud image processing method of this application, a segmented image is obtained by using deep learning to segment cloud regions with diverse shapes and sizes into the same category as the sun; then, the sun region with significant features is extracted by directly processing the image; finally, the sun region is used to segment the cloud and sun in the segmented image, thereby more accurately segmenting the cloud, the sun and the regions outside of them.

[0068] Reference Figure 6 In some embodiments, step 400 may include:

[0069] Step 410: Perform edge detection on the white regions in the binary image to obtain edge features;

[0070] Step 420: Perform circular detection on the edge features to obtain circular features;

[0071] Step 430: Combine the circular feature and the edge feature and take their intersection to obtain the second region corresponding to the sun's outline.

[0072] In this embodiment, by first extracting the edges of the white area of ​​the binary image, and then performing circular detection on the edges, the circular detection structure and the edges are recombined to more easily and accurately separate the solar area.

[0073] Reference Figure 7and Figure 8 , Figure 7 A binary image is shown. Figure 8 An edge detection result image is shown. This is achieved by employing an edge detection algorithm... Figure 7 The binary image shown can be used for detection, and the edges of the white areas can be extracted. The result is shown in the image below. Figure 8 As shown.

[0074] In some embodiments, the process of performing circular detection on edge features to obtain circular features may include: filtering the edge features to obtain filtered edge features; and performing circular detection on the filtered edge features to obtain circular features. Combining the circular features and edge features and taking their intersection may include: combining the circular features and the filtered edge features and taking their intersection.

[0075] Reference Figure 9 , Figure 9 A filtered edge feature map is shown. Compared to... Figure 8 , Figure 10 The edge lines of each part in the image are smoother. Filtering can smooth the edge lines of each part of the edge feature, thereby improving the accuracy of circle detection and more accurately determining the true solar region.

[0076] In other embodiments, the binary image can be filtered first, and then edge detection can be performed on the filtered binary image to obtain smoother edge features.

[0077] Reference Figure 10 , Figure 10 The image shows the result of a circular detection method. Circular detection refers to fitting a portion of the edge to a circular arc, forming a complete circle with that arc. Since at least part of the extracted edges belongs to the solar region, and the solar region is typically circular, fitting the edges to a circle is equivalent to the actual solar region. Furthermore, edges outside the solar region are mostly discrete, random curves, which do not affect the circular detection.

[0078] Continue to refer to Figure 4 , Figure 4 It can also represent a result image combining circular and edge features. Since the solar region may be partially obscured by clouds in actual cloud images, the circular feature may not necessarily correspond to the solar region in the actual cloud image. By combining the circular and edge features and taking their intersection, a second region that matches the solar region in the cloud image can be obtained. The portion of the circular feature that extends beyond the edge feature represents the portion of the sun obscured by clouds.

[0079] Reference Figure 11 In some embodiments, step 300 may include:

[0080] Step 310: Perform erosion processing on the cloud image to be processed to obtain an eroded image;

[0081] Step 320: Perform grayscale and binarization processing on the erosion image sequentially.

[0082] Understandably, erosion operations on images can eliminate boundary points of objects, causing the boundaries to shrink inward, and can remove objects smaller than the structuring element. They can also separate two objects with small connections, and are used to remove burrs, small protrusions, etc.

[0083] Reference Figure 12 , Figure 12 It shows the Figure 2 The result of the etching operation. Compared to... Figure 2 , Figure 12 The highlighted area surrounding the sun in the cloud image is relatively small. In the cloud image, a halo exists around the sun region, and the pixels in the halo area are brighter, which may affect the extraction of the sun region. Therefore, erosion processing of the cloud image can be used to reduce the range of the desired height area.

[0084] In some embodiments, the etching operation can be performed according to the following formula:

[0085]

[0086] Where element is the structuring element, (x, y) is the position of anchor point O, x' and y' are the position offsets of pixels with a structuring element value of 1 relative to anchor point O, src represents the original image, and dst represents the result image.

[0087] Reference Figure 13 In some embodiments, step 200 may include:

[0088] Step 210: Use a convolutional neural network to extract features from the cloud map to be processed, and output a feature map with the same scale as the cloud map to be processed;

[0089] Step 220: Classify each pixel in the feature map to identify the first region, which includes clouds and the sun, to obtain a segmented image.

[0090] Convolutional neural networks (CNNs) are a type of feedforward neural network that includes convolutional computations and has a deep structure. They perform convolutional operations on the cloud image to be processed using multiple convolutional kernels and output a feature map. The basic structure and principles of CNNs are well-established technologies, and will not be elaborated upon here.

[0091] In this embodiment, the features output by the convolutional neural network have the same scale as the cloud map to be processed. To ensure that the remapped solar region is not distorted, it is necessary to ensure that the mapped objects have the same size. The relevant details of remapping can be found in the foregoing embodiments.

[0092] Understandably, convolutional kernels are mostly 3x3 or 5x5 in size. Convolutional kernels typically perform downsampling on the input, meaning the output feature map is often smaller than the input. Therefore, upsampling can be performed before the convolutional neural network outputs the data to restore the original image size.

[0093] Reference Figure 14 In some embodiments, step 210 may include:

[0094] Step 211: Extract image features from the cloud map to be processed using a residual convolutional layer to obtain the first feature map;

[0095] Step 212: Use a hollow spatial pyramid pooling layer to extract features from the first feature map, and then concatenate the extracted features in the depth dimension to obtain the second feature map;

[0096] Step 213: Process the second feature map sequentially using a transposed convolutional layer and a dilated convolutional layer to obtain the third feature map;

[0097] Step 214: Transmit the third feature map to the output layer to output a feature map with the same scale as the cloud map to be processed.

[0098] In this embodiment, a convolutional neural network with residual structure is used as the backbone grid, where residual convolutional layers are used to extract image features from the cloud map to be processed. A hollow spatial pyramid pooling layer can be connected in parallel with multiple convolutional kernels of different sizes to achieve multi-scale feature extraction; then, the multi-scale features are concatenated in the depth dimension to output a high-dimensional second feature map.

[0099] Understandably, in a conventional convolutional neural network, the size of the input image is fixed given a given structure. By adding a dilated spatial pyramid pooling layer, pyramid pooling can be performed on the feature map, extracting a fixed-length feature vector. With the addition of this layer, the size of the input image becomes unrestricted.

[0100] As is understandable, transposed convolution, also known as "deconvolution," is represented in deep learning as the inverse process of convolution. It can restore the image size before convolution based on the kernel size and the output size. Transposed convolution achieves upsampling to restore the size of the feature map.

[0101] Dilated convolution controls the receptive field without changing the feature map size, thereby extracting multi-scale information. After restoring the feature map size, dilated convolution is used for further feature extraction to obtain multi-scale information and improve the accuracy of image segmentation. The output of the dilated convolution is connected to the convolutional output layer of the backbone network for output. Furthermore, since dilated convolution is used to achieve downsampling, its output can be transposed and output from the output layer. The detailed structures of the residual convolutional layer, dilated spatial pyramid pooling layer, transposed convolutional layer, and dilated convolutional layer involved in the above process are well-established technologies and will not be elaborated upon in this implementation.

[0102] In some embodiments, step 220 may include:

[0103] Step 221: Label each pixel in the feature map using a first label and a second label to obtain a labeled feature map. The first label corresponds to cloud and sun features, and the second label corresponds to features other than cloud and sun features.

[0104] Step 222: Perform binary classification on each pixel in the labeled feature map to obtain the segmented image.

[0105] In this embodiment, a deep learning network is used to segment the cloud image to be processed into two parts: one part consists of clouds and the sun, and the other part consists of the sky. Therefore, the depth dimension of the output feature map is 2, and each pixel can be labeled with a first label or a second label, with each label corresponding to a different region.

[0106] In some embodiments, the first label can be 0 and the second label can be 1. Each pixel in the feature map is labeled either 0 or 1 based on its feature value.

[0107] It should be noted that binary classification of each pixel in the labeled feature map refers to classifying each pixel based on its first or second label, and displaying the classification results accordingly. For example, pixels corresponding to the first label are displayed as white, and pixels corresponding to the second label are displayed as blue. An example of image segmentation can be found in [reference needed]. Figure 3 .

[0108] Reference Figure 15 In some embodiments, step 100 may include:

[0109] Step 110: Obtain the foundation cloud map;

[0110] Step 120: Based on the capture box, capture the target area in the foundation cloud map to obtain the cloud map to be processed.

[0111] It should be noted that the target area refers to the ROI (region of interest), which in machine vision and image processing is the region of the image to be processed that needs to be delineated. After receiving the ground-based cloud image sent by the all-sky ground imager, the electronic device can also crop the ROI region in the ground-based cloud image and use the cropped image as the cloud image to be processed.

[0112] In some embodiments, the center of the cropping frame may coincide with the center of the foundation cloud map, and the target area to be cropped is the central area of ​​the foundation cloud map. Of course, other parts of the foundation cloud map can also be cropped as the cloud map to be processed. The size and shape of the cropping frame can be set according to requirements, and this embodiment does not impose any limitations on them.

[0113] Reference Figure 16 , Figure 16 An image from an all-sky ground imager is shown. The image from an all-sky ground imager is typically in color, therefore... Figure 16 The image is actually in color. Since the sky captured by the all-sky ground imager extends for several kilometers, the outer areas may contain ground-based information, and the accuracy of solar irradiance information at distant locations is relatively low. Irradiance analysis based on cloud images often focuses on a small area of ​​cloud cover and sunlight in the central region of the cloud image. Therefore, cropping the central region of the ground-based cloud image as the Region of Interest (ROI) can improve the accuracy of subsequent analysis and reduce the processing difficulty of the cloud image.

[0114] Reference Figure 17 An embodiment of this application also provides a cloud image processing apparatus 600. The cloud image processing apparatus 600 can implement the cloud image processing methods in the above embodiments.

[0115] In this embodiment, the cloud image processing device 600 includes: an image acquisition module 610, a neural network module 620, a binary module 630, a segmentation module 640, and a mapping module 650; the image acquisition module 610 is used to acquire a cloud image to be processed; the neural network module 620 is used to perform semantic segmentation on the cloud image to be processed using a deep learning network to obtain a segmented image, the segmented image including a first region, the first region including clouds and the sun; the binary module 630 is used to perform binarization processing on the cloud image to be processed to obtain a binary image; the segmentation module 640 is used to determine a second region corresponding to the sun's outline based on the white region in the binary image; the mapping module 650 is used to remap the second region to the first region in the segmented image to obtain a target image.

[0116] The cloud image processing device 600 in this embodiment can be an electronic device or a component within an electronic device, such as a chip. The electronic device can be a mobile terminal, computer, or server, etc., and this embodiment does not impose specific limitations.

[0117] According to the cloud image processing apparatus of this application, a segmented image is obtained by using deep learning to segment cloud regions with diverse shapes and sizes into the same category as the sun; then, the sun region with significant features is extracted by directly processing the image; finally, the sun region is used to segment the cloud and sun in the segmented image, thereby more accurately segmenting the cloud, the sun and the regions outside of both.

[0118] In some embodiments, the segmentation module 640 is further configured to perform edge detection on the white region in the binary image to obtain edge features; perform circular detection on the edge features to obtain circular features; and combine the circular features and the edge features and take their intersection to obtain a second region corresponding to the sun outline.

[0119] In some embodiments, the segmentation module 640 is further configured to filter the edge features to obtain filtered edge features; perform circular detection on the filtered edge features to obtain circular features; and combine the circular features with the edge features and take the intersection, including: combining the circular features with the filtered edge features and taking the intersection.

[0120] In some embodiments, the binarization module 630 is further configured to perform erosion processing on the cloud image to be processed to obtain an eroded image; and to perform grayscale and binarization processing on the eroded image in sequence.

[0121] In some embodiments, the neural network module 620 is further configured to use a convolutional neural network to extract features from the cloud image to be processed and output a feature map with the same scale as the cloud image to be processed; classify each pixel in the feature map to identify a first region including clouds and the sun, and obtain a segmented image.

[0122] In some embodiments, the neural network module 620 is further configured to label each pixel in the feature map with a first label and a second label to obtain a labeled feature map, wherein the first label corresponds to cloud and sun features, and the second label corresponds to features other than cloud and sun features; and to perform binary classification on each pixel in the labeled feature map to obtain a segmented image.

[0123] In some embodiments, the neural network module 620 is further configured to extract image features from the cloud map to be processed using a residual convolutional layer to obtain a first feature map; extract features from the first feature map using a dilated spatial pyramid pooling layer, and concatenate the extracted features in the depth dimension to obtain a second feature map; process the second feature map using a transposed convolutional layer and a dilated convolutional layer in sequence to obtain a third feature map; and transmit the third feature map to the output layer to output a feature map with the same scale as the cloud map to be processed.

[0124] In some embodiments, the image acquisition module 610 is further configured to acquire a ground cloud map; and to crop the target area in the ground cloud map based on the cropping box to obtain a cloud map to be processed.

[0125] The cloud image processing device 600 provided in this application embodiment can implement the various processes implemented in the above method embodiments. The relevant content can be referred to the foregoing embodiments. To avoid repetition, it will not be repeated here.

[0126] Reference Figure 18 An embodiment of this application also provides an electronic device 700. The electronic device 700 includes a memory 710, a processor 720, and a computer program stored in the memory 720 and executable on the processor 710. When the processor 720 executes the program, it implements any of the cloud image processing methods described in the above embodiments. Furthermore, it achieves the same technical effects, and to avoid repetition, it will not be described again here.

[0127] One embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the cloud image processing method as described in any of the above embodiments. It achieves the same technical effect, and to avoid repetition, will not be described further here.

[0128] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0131] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0132] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0133] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A cloud image processing method, characterized in that, include: Obtain the cloud map to be processed; A deep learning network is used to perform semantic segmentation on the cloud image to be processed to obtain a segmented image. The segmented image includes a first region, which includes clouds and the sun. The cloud image to be processed is binarized to obtain a binary image; The second region corresponding to the sun's outline is determined based on the white region in the binary image; The second region is remapped onto the first region in the segmented image to obtain the target image; Determining the second region corresponding to the sun's outline based on the white region in the binary image includes: Edge detection is performed on the white areas in the binary image to obtain edge features; Perform circular detection on the edge features to obtain circular features; By combining the circular feature and the edge feature and taking their intersection, a second region corresponding to the sun's outline is obtained; The step of using a deep learning network to perform semantic segmentation on the cloud image to be processed includes: A convolutional neural network is used to extract features from the cloud map to be processed, and a feature map with the same scale as the cloud map to be processed is output. Each pixel in the feature map is labeled using a first label and a second label to obtain a labeled feature map. The first label corresponds to cloud and sun features, and the second label corresponds to features other than cloud and sun features. Each pixel in the labeled feature map is classified into two categories to obtain a segmented image.

2. The cloud image processing method according to claim 1, characterized in that, The step of performing circular detection on the edge features to obtain circular features includes: The edge features are filtered to obtain filtered edge features; Perform circular detection on the filtered edge features to obtain circular features; The step of combining the circular feature and the edge feature and taking their intersection includes: The circular feature is combined with the filtered edge feature, and their intersection is taken.

3. The cloud image processing method according to claim 1 or 2, characterized in that, The binarization process of the cloud image to be processed includes: The cloud image to be processed is subjected to erosion processing to obtain an eroded image; The erosion image is then processed by grayscale conversion and binarization sequentially.

4. The cloud image processing method according to claim 1 or 2, characterized in that, The step of using a convolutional neural network to extract features from the cloud map to be processed and outputting a feature map with the same scale as the cloud map to be processed includes: The image features in the cloud map to be processed are extracted using a residual convolutional layer to obtain a first feature map; The first feature map is extracted using a hollow spatial pyramid pooling layer, and the extracted features are then concatenated along the depth dimension to obtain the second feature map. The second feature map is processed sequentially using a transposed convolutional layer and a dilated convolutional layer to obtain the third feature map; The third feature map is transmitted to the output layer to output a feature map with the same scale as the cloud map to be processed.

5. The cloud image processing method according to claim 1 or 2, characterized in that, The acquisition of the cloud map to be processed includes: Obtain a foundation cloud map; The target area in the foundation cloud map is captured by the capture box to obtain the cloud map to be processed.

6. A cloud image processing device, characterized in that, include: Image acquisition module, used to acquire the cloud image to be processed; The neural network module is used to perform semantic segmentation on the cloud image to be processed using a deep learning network to obtain a segmented image, wherein the segmented image includes a first region, and the first region includes clouds and the sun; The binary module is used to perform binarization processing on the cloud image to be processed to obtain a binary image; A segmentation module is used to determine a second region corresponding to the sun's outline based on the white region in the binary image; The mapping module is used to remap the second region to the first region in the segmented image to obtain the target image; The segmentation module is further configured to perform edge detection on the white region in the binary image to obtain edge features; perform circular detection on the edge features to obtain circular features; and combine the circular features with the edge features and take their intersection to obtain a second region corresponding to the sun outline. The neural network module is further configured to use a convolutional neural network to extract features from the cloud image to be processed and output a feature map with the same scale as the cloud image to be processed; and to label each pixel in the feature map with a first label and a second label to obtain a labeled feature map, wherein the first label corresponds to cloud and solar features, and the second label corresponds to features other than cloud and solar features. Each pixel in the labeled feature map is classified into two categories to obtain a segmented image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cloud image processing method as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the cloud map processing method as described in any one of claims 1-5.

Images