A method for ground-based cloud shape recognition based on lightweight neural network

Abstract

The present application discloses a ground-based cloud recognition method based on a lightweight neural network, which relates to the field of image processing technology. The method aims at the phenomenon of uneven distribution of sample data of different cloud categories, and combines the characteristics of multi-label recognition scenarios in which a single sample full-sky cloud map contains multiple cloud labels with correlation. It proposes a method for balancing the number of samples of different cloud label combinations by calculating the cloud label value of the entire sample full-sky cloud map and then realizing oversampling operation. In addition, a new lightweight cloud recognition network is built, and the Inception module, SRM module and residual module are introduced, which can effectively enhance the feature extraction capability of the cloud recognition network for the edge and texture of the cloud. The preprocessing operation of the data set used for model training is combined with the optimization of the network structure, so that the method can improve the accuracy of cloud recognition, while ensuring the operation speed, and can run on devices with limited resources.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a ground-based cloud shape recognition method based on a lightweight neural network. Background Art

[0002] Clouds play a very important role in atmospheric radiation transmission, global water cycle, global energy balance, etc. Studying the characteristics of clouds is also helpful to explore ice accumulation in the troposphere and strong disturbances in the stratosphere. Therefore, cloud observation is of great significance in weather forecasting, artificial weather modification, flight support, etc. In meteorological operations, the three elements of cloud observation are: cloud amount, cloud shape and cloud height. At present, various observation equipment such as satellites, all-sky imagers, laser cloud meters or millimeter-wave cloud radars have been used in meteorological operations to carry out automatic observations of cloud amount and cloud height. However, the observation of cloud shape has always been subjective, uncertain, with large errors, and time-consuming and labor-intensive. Therefore, cloud shape observation technology has received more and more attention in recent years.

[0003] With the development of ground observation equipment and imaging technology, ground-based cloud observation has gradually changed from manual visual recording to automated observation recording. At present, the research on cloud shape recognition mainly relies on two automated observation methods: (1) cloud shape classification and recognition based on cloud images. (2) classification and recognition based on cloud reflectance characteristics. Most of the operations of the cloud reflectance feature-based method are still mainly implemented manually, which is inefficient and not suitable for the development of automated observation and forecasting services. Therefore, the main methods used in existing research are cloud shape classification and recognition based on cloud images.

[0004] Cloud classification and recognition based on cloud images mainly adopts image classification and recognition methods, and with the development of technology, methods based on deep learning are gradually used for cloud classification, that is, first extract cloud image features, and then use deep learning models to classify based on cloud image features to achieve cloud recognition. The training of deep learning models depends on cloud sample data. The China Meteorological Administration has established cloud classification standards for three families, ten genera and twenty-nine categories in the ground meteorological observation standard clouds, but some of the clouds are very rare, resulting in very few sample data for such clouds. Therefore, the obtained cloud sample data is often unevenly distributed and generally obeys a long-tail distribution, which leads to the trained deep learning model being able to recognize fewer cloud categories. At present, most studies can usually only achieve 4 to 10 categories of coarse-grained cloud classification and recognition, which affects the accuracy and effect of current cloud recognition. Summary of the invention

[0005] In view of the above problems and technical requirements, this application proposes a ground cloud shape recognition method based on a lightweight neural network. The technical solution of this application is as follows:

[0006] A ground-based cloud shape recognition method based on a lightweight neural network, the ground-based cloud shape recognition method comprising:

[0007] Obtain an initial sample set including several sample full-sky cloud images;

[0008] According to the label value of the cloud shape label contained in each sample full-sky cloud image in the initial sample set, the cloud image label value of the sample full-sky cloud image is determined, and the label value of each type of cloud shape label is different and unique;

[0009] According to the distribution of the cloud image label values ​​of each sample full sky cloud image, an oversampling operation is performed on the initial sample set to balance the number of different cloud shape label combinations and obtain a balanced sample set;

[0010] The cloud recognition network is trained using a balanced sample set. The constructed cloud recognition network includes an Inception module, a residual module, and an SRM module. The Inception module is used for multi-scale feature extraction, and the SRM module adaptively recalibrates the intermediate feature map by using the style of the intermediate feature map.

[0011] The trained cloud shape recognition network is used to perform ground-based cloud shape recognition on the all-sky cloud image to be identified, so as to output the cloud shape recognition result of the all-sky cloud image to be identified.

[0012] Its further technical solution is that the sample full sky cloud image in the initial sample set contains 31 different cloud shape labels, including cloud-free labels, visibility obstruction labels and 29 different cloud shape classification labels, each of which corresponds to a cloud type of ground-based clouds;

[0013] Each sample all-sky cloud map contains only a cloud-free label, or only a visibility obstruction label, or at least one cloud shape classification label; wherein the cloud-free label indicates that the sample all-sky cloud map does not contain ground-based clouds, and the visibility obstruction label indicates that the cloud shape category of the ground-based clouds contained in the sample all-sky cloud map cannot be determined due to visibility obstruction.

[0014] Its further technical solution is that the cloud recognition network includes an Inception module and multiple residual modules cascaded in sequence, and the multiple residual modules are used to realize the extraction of deep features; each residual module is constructed based on the Ghost module and the CA attention mechanism, and the residual connection is used to realize feature reuse.

[0015] Its further technical solution is that the residual module includes a convolutional layer, a Ghost module, a CA attention mechanism, a 1*1 convolutional layer and an activation function connected in sequence. The feature map F1 input to the residual module passes through the convolutional layer to obtain a feature map F2 and is input to the Ghost module. The feature map F2 is subjected to parallel processing of identity mapping and linear transformation in the Ghost module to realize channel splicing to obtain a feature map F3. The feature map F3 passes through the CA attention mechanism and the 1*1 convolutional layer in sequence to obtain a feature map F4. The feature map F4 is residually connected with the feature map F1, passes through the activation function and outputs the residual module.

[0016] Its further technical solution is that the cloud recognition network includes several convolutional layers, Inception modules, multiple residual modules, flattening layers and linear layers cascaded in sequence, a maximum pooling layer is added after each convolutional layer and after the Inception module, an SRM module is inserted between the convolutional layers, an SRM module is inserted before the Inception module, an SRM module is inserted before each residual module and a maximum pooling layer is added after the inserted SRM module, and an average pooling layer is added before the flattening layer.

[0017] Its further technical solution is that the feature map f1 input to the Inception module passes through the 3*3 convolution layer and the 1*1 convolution layer in sequence to obtain the feature map f2 and output it to the concat module, the feature map f1 also passes through the 1*1 convolution layer and the 5*5 convolution layer in sequence to obtain the feature map f3 and output it to the concat module, the feature map f1 also passes through the 1*1 convolution layer and the 3*3 convolution layer in sequence to obtain the feature map f4 and output it to the concat module, the feature map f1 also passes through the 1*1 convolution layer to obtain the feature map f5 and output it to the concat module; the concat module performs channel splicing on the feature map f2, the feature map f3, the feature map f4 and the feature map f5 and outputs them.

[0018] A further technical solution is that performing an oversampling operation on the initial sample set includes:

[0019] Determine the number of samples corresponding to each cloud map label value and the maximum number of samples respectively. The number of samples corresponding to any cloud map label value is the number of sample full-sky cloud maps with the cloud map label value in the initial sample set. The maximum number of samples is the maximum number of sample full-sky cloud maps with the same cloud map label value in the initial sample set.

[0020] Each cloud map label value is traversed in turn, and the number of extractions is determined according to the degree of difference between the number of samples corresponding to the traversed cloud map label value and the maximum number of samples. According to the number of extractions, sample full-sky cloud maps are extracted from the sample full-sky cloud maps with the currently traversed cloud map label value contained in the initial sample set and added to the initial sample set; after all non-repeated cloud map label values ​​are traversed, it is determined that the oversampling operation is completed, and a balanced sample set is obtained after adding the sample full-sky cloud map to the initial sample set.

[0021] A further technical solution is to determine the number of samples to be extracted based on the difference between the number of samples corresponding to the traversed cloud map label value and the maximum number of samples, including:

[0022] The number of samples is determined to be [(Nn) / β], where N is the maximum number of samples, n is the number of samples corresponding to the currently traversed cloud map label value, β is the coefficient, and [(Nn) / β] means rounding (Nn) / β.

[0023] Its further technical solution is that the cloud map label value of each sample full-sky cloud map is the sum of the label values ​​of various non-repetitive cloud labels contained in the sample full-sky cloud map, and the cloud labels contained in any two sample full-sky cloud maps with the same cloud map label value are the same or different.

[0024] A further technical solution is to use a balanced sample set to train a cloud recognition network, including:

[0025] The loss weight of each cloud shape label in the balanced sample set is calculated as M / (m*η). The loss function is calculated based on the loss weight of each cloud shape label and the cloud recognition network is trained according to the loss function; where M is the total number of sample full-sky cloud images included in the balanced sample set, m is the number of sample full-sky cloud images containing cloud shape labels in the balanced sample set, and η is a constant factor.

[0026] The beneficial technical effects of this application are:

[0027] The present application discloses a cloud recognition method for ground-based clouds based on a lightweight neural network. The method aims at the phenomenon of uneven distribution of sample data of different cloud categories, and combines the characteristics of multi-label recognition scenarios in which a single sample full-sky cloud map contains multiple cloud labels with correlation. A method is proposed to calculate the cloud label value of the entire sample full-sky cloud map and then realize oversampling operation to balance the number of samples of different cloud label combinations, which effectively alleviates the overfitting of the model and significantly improves the training accuracy of the model. The cloud recognition network constructed by the present application is a lightweight network. The cloud recognition network introduces the Inception module to effectively capture the multi-scale features of the cloud structure through parallel multi-scale convolution kernels, improves the recognition ability of cloud structures of different forms and scales, and uses the SRM module and residual convolution structure to capture remote spatial interactions with precise location information, effectively enhances the feature extraction ability of the cloud recognition network for the edge and texture of the cloud, and can improve the recognition accuracy of complex cloud structures. The preprocessing operation of the data set used for model training is combined with the optimization of the network structure, so that the method can improve the cloud recognition accuracy, while ensuring the operation speed, and can run on devices with limited resources.

[0028] The residual module introduced in the cloud recognition network constructed in this method also combines the advantages of GhostNet and the CA attention mechanism, which not only helps to solve the impact of redundant feature maps in the network on the increase of network time consumption and parameters, but also makes full use of location information, rather than being limited to capturing local information. This design improves the accuracy and adaptability of the cloud recognition network in multi-label recognition scenarios, while alleviating the gradient vanishing problem of deep networks, so that this method can well realize the recognition of 29 standard cloud categories specified by the China Meteorological Administration, as well as 31 types of cloud labels in cloudless scenes and visibility obstruction scenes.

[0029] In response to the uneven distribution of the number of different cloud-like labels, this method also introduces a new method for calculating the loss weight of cloud-like labels during the model training process. The calculated loss weight is used to solve the loss function, which can make the model training process better biased towards a small number of samples, further improving the recognition and classification accuracy and precision of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a method flow chart of a ground cloud shape recognition method according to an embodiment of the present application.

[0031] Figure 2 The diagram is a distribution diagram of cloud map label values ​​of a sample full sky cloud map in an initial sample set in an instance, and a distribution diagram of cloud map label values ​​after an oversampling operation is performed on the initial sample set.

[0032] Figure 3It is a network structure diagram of a cloud-like recognition network in one embodiment of the present application.

[0033] Figure 4 It is a structural diagram of each residual module in one embodiment of the present application.

[0034] Figure 5 It is a structural diagram of the Inception module in one embodiment of the present application. DETAILED DESCRIPTION

[0035] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0036] This application discloses a cloud-like recognition method for ground-based clouds based on a lightweight neural network. Figure 1 The method comprises:

[0037] Step 1: obtain an initial sample set containing several sample full-sky cloud images. The sample full-sky cloud images in the initial sample set can be obtained by taking pictures with a fisheye camera, or directly obtain the full-sky cloud images provided by various meteorological bureaus or meteorological detection centers. In order to expand the data set, the sample full-sky cloud images directly obtained are generally subjected to preprocessing operations including random cropping, random flipping, normalization, and channel transformation to obtain the initial sample set.

[0038] Each sample all-sky cloud image in the initial sample set has been labeled with a cloud shape label, and the cloud shape label of each sample all-sky cloud image indicates the cloud shape recognition result of the sample all-sky cloud image. The ground-based clouds in a sample all-sky cloud image may belong to one or more different cloud shape categories. Since the number of sample all-sky cloud images in the initial sample set is relatively large, for example, it generally reaches 20,000 sample all-sky cloud images or even more. Therefore, all sample all-sky cloud images usually cover various cloud shape categories specified by the China Meteorological Administration in the ground meteorological observation specification clouds, but the cloud shape categories of common clouds have a higher probability of appearing, while the cloud shape categories of rare clouds have a lower probability of appearing. In addition, there will be some sample all-sky cloud images that do not actually include ground-based clouds, and due to external influences such as the acquisition environment, some sample all-sky cloud images contain ground-based clouds, but due to visibility obstacles, the cloud shape categories of the ground-based clouds contained in the sample all-sky cloud images cannot be accurately distinguished.

[0039] Based on the above considerations, in one embodiment, the sample full-sky cloud image in the initial sample set contains a total of 31 different cloud shape labels, including a cloudless label (No), a visibility barrier label (ZangYun), and 29 different cloud shape classification labels. Among them, the cloudless label indicates that the sample full-sky cloud image does not contain ground-based clouds, the visibility barrier label indicates that the cloud shape category of the ground-based clouds contained in the sample full-sky cloud image cannot be determined due to visibility barriers, and each cloud shape classification label corresponds to and indicates a cloud shape category of the ground-based clouds.

[0040] According to the specifications formulated by the China Meteorological Administration in the standard clouds for ground meteorological observation, the cloud classification labels corresponding to all 29 cloud categories included in the cloud family from low clouds to high clouds include: light cumulus (Cuhum), broken cumulus (Fc), dense cumulus (Cucong), bald cumulonimbus (Cbcalv), bristle cumulonimbus (Cbcap), translucent stratocumulus (Sctra), opaque stratocumulus (Scop), cumulus stratus (Sccug), castellated stratocumulus (Sccast), lenticular stratocumulus (Sclent), stratus (St), broken stratus (Fs), Nimbostratus (Ns), broken nimbus (Fn), translucent altostratus (Astra), opaque altostratus (Asop), translucent altocumulus (Actra), opaque altocumulus (Acop), lenticular altocumulus (Aclent), cumulus altocumulus (Accug), flocculent altocumulus (Acflo), altocumulus castellate (Accast), cirrus (Cifil), dense cirrus (Cidens), pseudo cirrus (Cinot), cirrus hook (Ciunc), cirrostratus cirrostratus (Csfil), thin veil cirrostratus (Csnebu), and cirrocumulus (Cc).

[0041] Each sample full sky cloud map contains only cloudless labels, or only visibility obstruction labels, or at least one cloud classification label. In order to facilitate model training, the sample full sky cloud maps are usually annotated in a unified format. The annotation result of any sample full sky cloud map is a 31-bit binary sequence. Each data bit of the binary sequence corresponds to a cloud label. When any data bit in the binary sequence is 1, it means that the sample full sky cloud map contains the cloud label corresponding to the data bit. When any data bit in the binary sequence is 0, it means that the sample full sky cloud map does not contain the cloud label corresponding to the data bit. The arrangement order of different cloud labels can be customized.

[0042] For example, suppose that 31 different cloud labels are arranged according to the cloudless label (No), the visibility obstruction label (ZangYun), and the 29 cloud classification labels in the order shown above. Then, when a sample full-sky cloud image contains cumulonimbus, lenticular stratocumulus, and obscuring altostratus, the annotation result of the sample full-sky cloud image is 00000010000100000100000000000000. For another example, when a sample full-sky cloud image does not contain ground-based clouds, the annotation result of the sample full-sky cloud image is 1000000000000000000000000000000000, and the same is true for other cases.

[0043] Step 2: Determine the cloud image label value of the sample full sky cloud image according to the label value of the cloud shape label contained in each sample full sky cloud image in the initial sample set.

[0044] Different cloud categories appear at different frequencies in actual meteorological environments. The number of sample full-sky cloud images containing rare cloud categories is often small. For example, they only appear a dozen times in 20,000 sample full-sky cloud image samples. As a result, the number of different cloud labels in the initial sample set directly obtained varies greatly. If the initial sample set is used directly for model training, the model will not be able to accurately learn the characteristics of these rare cloud categories, and the recognition and classification effects of these cloud categories will be poor. Therefore, it is necessary to oversample the initial sample set to balance the number of different cloud labels.

[0045] However, as mentioned above, a sample full-sky cloud image will contain multiple cloud labels, so the annotation result of a sample full-sky cloud image is often a cloud label group. In addition, multiple cloud labels in a sample full-sky cloud image often have complex correlations. For example, according to the characteristics of ground-based clouds in different meteorological environments, some cloud categories often appear at the same time, but some cloud categories basically do not appear at the same time. Considering the annotation results in the form of cloud label groups and the correlation between cloud labels, in order to facilitate the subsequent step 2 to perform oversampling operations, the cloud label value of the sample full-sky cloud image is first determined, so that it is convenient to use a cloud label value to distinguish the sample full-sky cloud image, rather than directly using a cloud label group to distinguish the sample full-sky cloud image, and the obtained cloud label value can also well represent the cloud label combination contained in the sample full-sky cloud image.

[0046] The label value of each type of cloud label is different and unique. The label value of each type of cloud label can be customized. The following provides several possible label value definition methods through different embodiments: (1) In one embodiment, according to the predetermined arrangement order of each cloud label in the annotation results of the sample full-sky cloud map, the label values ​​of the 31 types of cloud labels are taken as 1 to 31 in sequence, and the label value of the ith cloud label in the arrangement order is defined as i. For example, based on the arrangement order of cloud labels in the above example, the label value of the cloudless label is 1, the label value of the visibility obstruction label is 2, the label value of the light cumulus is 3, the label value of the broken cumulus is 4, and so on. (2) Or in another embodiment, according to the predetermined arrangement order of each cloud label in the annotation results of the sample full-sky cloud map, the label value of the ith cloud label in the arrangement order is defined as a i . A common base a can be taken as 2. Taking this as an example, based on the arrangement order of the cloud-shaped labels in the above example, the label value of the cloudless label is 2, the label value of the visibility obstruction label is 4, the label value of the light cumulus is 8, the label value of the broken cumulus is 16, and so on. At this time, the label value of each cloud-shaped label is an even number. (3) Or in another embodiment, according to the predetermined arrangement order of each cloud-shaped label in the annotation result of the sample full-sky cloud map, the label value of the i-th cloud-shaped label in the arrangement order is defined as a i -1. The common base a can be 2. Taking this as an example, based on the order of arrangement of the cloud-shaped labels in the above example, the label value of the cloudless label is 1, the label value of the visibility obstruction label is 3, the label value of the light cumulus is 7, the label value of the broken cumulus is 15, and so on. At this time, the label value of each cloud-shaped label is an odd number. (4) Or further in another embodiment, further deformation can be performed based on the above several common methods, such as setting the label value of the ith cloud-shaped label in the arrangement order to a i Multiplication by a constant argument.

[0047] After presetting the label values ​​of each cloud-shaped label, the cloud map label value of the sample full-sky cloud map can be obtained according to the label value of the cloud-shaped label contained in each sample full-sky cloud map. In one embodiment, the label values ​​of the cloud-shaped labels contained in a sample full-sky cloud map are added to obtain the cloud map label value of each sample full-sky cloud map, which is the sum of the label values ​​of various non-repetitive cloud-shaped labels contained in the sample full-sky cloud map. Other methods can also introduce some mathematical operations to perform mathematical operations on the label values ​​of the cloud-shaped labels contained in a sample full-sky cloud map to obtain the cloud map label value, which is not limited in this embodiment. For example, when the label value of the i-th cloud-shaped label in the arrangement order is defined as i, and the annotation result of a sample full-sky cloud map is 0000001000010000010000000000000, the cloud map label value of the sample full-sky cloud map is 37.

[0048] On the basis of this embodiment, when the tag value of the ith cloud tag in the arrangement order is defined as i, the sum of different tag value combinations may be the same, so theoretically, the cloud tag values ​​of two sample full-sky cloud images containing different cloud tags may be the same or different, which results in the cloud tag value not being unique. However, in fact, as described above, the characteristics of the meteorological environment determine that there will be correlations between different cloud tags, so in fact, different cloud tags will not be able to be combined and appear in the same sample full-sky cloud image as in theory. According to the actual evaluation of the sample full-sky cloud image, even if this constant form method is used to define the tag value of the cloud tag, it is relatively rare for the cloud tag values ​​of two sample full-sky cloud images containing different cloud tags to be the same. Compared with the method of directly defining the cloud tag in constant form, when the exponential form is used to define the tag value of each cloud tag, the cloud tag values ​​of the sample full-sky cloud images containing different cloud tags are less likely to have the same value.

[0049] Therefore, it can be seen from the above introduction that no matter how the label value of the cloud label is defined, although the cloud label value of the sample full sky cloud image cannot be strictly guaranteed to correspond to the cloud label combination, the cloud label value of a sample full sky cloud image can still be well associated with the cloud label combination contained in the sample full sky cloud image, and in most cases can characterize the cloud label combination contained in the sample full sky cloud image to a certain extent. Therefore, the distribution of the cloud label value of the sample full sky cloud image can well represent the distribution of the cloud label combination, thereby facilitating the subsequent oversampling operation.

[0050] Step 3: Based on the distribution of the cloud image label values ​​of each sample full sky cloud image, an oversampling operation is performed on the initial sample set to balance the number of different cloud shape label combinations and obtain a balanced sample set, including:

[0051] First, determine the number of samples corresponding to each cloud map label value and determine the maximum number of samples. The number of samples corresponding to any cloud map label value is the number of sample full-sky cloud maps with the cloud map label value in the initial sample set, and the maximum number of samples is the maximum number of sample full-sky cloud maps with the same cloud map label value in the initial sample set. For example, in one example, the initial sample set contains a total of 19,200 sample full-sky cloud maps. The distribution diagram of the number of samples corresponding to different cloud map label values ​​is as follows: Figure 2 As shown in (a) in the figure, the range of different cloud map label values ​​is 0~108, the maximum number of samples is 3705, and the minimum number of samples is only 1. It should be noted that due to the limited resolution of the vertical axis in the figure, some cloud map label values ​​cannot be used when the number of samples is small. Figure 2This is clearly shown in (a) of Figure 2 It can be seen that due to the large difference in the number of cloud labels in the initial sample set, the number of samples corresponding to different cloud map label values ​​is also quite different.

[0052] In addition, as mentioned above, especially when the label value of each cloud label is defined in the form of a constant, it is impossible to completely guarantee the one-to-one correspondence between different cloud label combinations and cloud map label values. Therefore, a cloud map label value may not be able to uniquely identify a cloud label combination, so that any two sample full-sky cloud maps with the same cloud map label value may contain the same or different cloud label combinations.

[0053] Then, each cloud map label value is traversed in turn, and the number of extractions is determined according to the difference between the number of samples corresponding to the traversed cloud map label value and the maximum number of samples, and the sample full sky cloud map is extracted from the sample full sky cloud map with the currently traversed cloud map label value contained in the initial sample set according to the number of extractions and added to the initial sample set. After all non-repeated cloud map label values ​​are traversed, the oversampling operation is determined to be completed, and a balanced sample set is obtained after adding the sample full sky cloud map to the initial sample set.

[0054] In one embodiment, for each cloud map label value traversed, the number of extractions is determined to be [(Nn) / β], where N is the maximum number of samples, n is the number of samples corresponding to the currently traversed cloud map label value, β is a coefficient, and [(Nn) / β] represents rounding (Nn) / β. The value of the coefficient β is customized and adjusted according to the number of samples in the initial sample set and experience. Taking the initial sample set containing 19,200 sample full-sky cloud maps as an example, it is determined by debugging to take β=20, then for example, the maximum number of samples is 3705, assuming that the number of samples corresponding to the currently traversed cloud map label value of 43 is 23, and the number of extractions is determined to be 184, then 184 sample full-sky cloud maps with a cloud map label value of 43 are extracted from the initial sample set and added to the initial sample set, and the cloud labels contained in any two sample full-sky cloud maps extracted and added to the initial sample set are the same or different.

[0055] After traversing all non-repeated cloud map label values ​​and completing the oversampling operation, the distribution diagram of the number of samples corresponding to different cloud map label values ​​is as follows: Figure 2 (a) in the figure becomes (b), and the comparison Figure 2 It can be seen from (a) and (b) in Figure 1 that after the oversampling operation, the distribution of the cloud map label values ​​is more balanced, indicating that the balance of different cloud-shaped label combinations is also higher. In actual work, multiple rounds of oversampling operations can also be further performed.

[0056] When the label value of each cloud label is defined in exponential form, the cloud label values ​​of the sample full sky cloud images containing different cloud label combinations can basically be made non-repeated. In this case, the effect of the oversampling operation will be better, and it is also beneficial to improve the accuracy and universality of the model obtained by subsequent training. However, the introduction of exponential operations will increase the amount of calculation, resulting in increased computing power requirements and additional time consumption. When the label value of each cloud label is defined in constant form, although there may be repeated cloud label values ​​of the sample full sky cloud images containing different cloud label combinations, the amount of calculation is greatly reduced, which is beneficial to shorten the processing time. After actual measurement, the repetition of a small number of cloud label values ​​has little effect on the effect of the oversampling operation, and there can still be a good sample balancing effect to improve the quality of the subsequent model. Therefore, directly defining the label value of each cloud label in constant form has a good comprehensive effect in terms of both the amount of calculation and the model accuracy requirements.

[0057] Step 4: Use the balanced sample set to train the cloud recognition network.

[0058] The cloud recognition network constructed in this application is a lightweight model, including an Inception module, a residual module and an SRM module. Among them, the Inception module is used for multi-scale feature extraction, so that the cloud recognition network can extract richer cloud features. The SRM module can enhance the identification ability of CNN by adaptively recalibrating the intermediate feature map by using the style of the intermediate feature map, and the amount of parameters introduced is also very small.

[0059] Furthermore, in one embodiment, the idea of ​​repeating basic units to construct a network is borrowed from the existing network, and the network is constructed by repeatedly using multiple residual modules, so that the constructed cloud recognition network includes an Inception module and multiple residual modules cascaded in sequence, and multiple residual modules are used to realize the extraction of deep features, and also to avoid the degradation problem in deep network training. And each residual module is constructed based on the Ghost module and the CA attention mechanism, and the residual connection is used to realize feature reuse, which can alleviate the gradient disappearance problem in the deep network, improve the model training efficiency of the cloud recognition network and the expression ability of deep features. In addition, the introduction of the CA attention mechanism can also improve the recognition accuracy of the cloud recognition network for complex cloud structures. The introduction of these mechanisms enables the constructed cloud recognition network to effectively extract and integrate features of different scales and types, and better capture the correlation between different cloud labels through multi-level features and attention mechanisms, improve the adaptability to the shape recognition task of multiple cloud labels, and at the same time, the efficient and lightweight design is suitable for running on devices with limited resources.

[0060] In one embodiment, please refer to Figure 3The network structure diagram of the cloud recognition network, which includes several convolutional layers, Inception modules, multiple residual modules, flattening layers and linear layers cascaded in sequence. A maximum pooling layer is added after each convolutional layer and after the Inception module. An SRM module is inserted between the convolutional layers, and an SRM module is inserted before the Inception module. An SRM module is inserted before each residual module, and a maximum pooling layer is added after the inserted SRM module. An average pooling layer is added before the flattening layer. For example Figure 3 Taking the cloud recognition network as an example, which includes three convolutional layers using 3*3 convolutional kernels and three residual modules, it is not necessary to insert an SRM module between every two convolutional layers. For example Figure 3 In , the SRM module is only inserted between the second and third convolutional layers.

[0061] The structures of each residual module are the same. The structure of each residual module is as follows: Figure 4 As shown, each residual module includes a convolution layer, a Ghost module, a CA attention mechanism, a 1*1 convolution layer and an activation function connected in sequence. The feature map F1 input to the residual module is passed through the convolution layer to obtain a feature map F2 and input to the Ghost module. The feature map F2 is processed in parallel by identity mapping and linear transformation in the Ghost module to achieve channel splicing to obtain a feature map F3. The feature map F3 is sequentially passed through the CA attention mechanism and the 1*1 convolution layer to obtain a feature map F4. The feature map F4 is residually connected with the feature map F1 and then passes through the activation function and outputs the residual module. In one embodiment, the activation function used is relu, Figure 4 Take this as an example.

[0062] In one example, according to the scene requirements of cloud recognition, the structure of the Inception module is as follows: Figure 5 As shown in the figure, the feature map f1 input to the Inception module passes through the 3*3 convolution layer and the 1*1 convolution layer in sequence to obtain the feature map f2 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer and the 5*5 convolution layer in sequence to obtain the feature map f3 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer and the 3*3 convolution layer in sequence to obtain the feature map f4 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer to obtain the feature map f5 and output it to the concat module. The concat module performs channel splicing on the feature maps f2, f3, f4 and f5 and outputs them.

[0063] When using a balanced sample set to train the cloud recognition network, first normalize each sample full-sky cloud map in the balanced sample set to a size of 256×256 to retain more detail information, and then input it into the constructed cloud recognition network for feature extraction and cloud label recognition. The sigmoid function can also be used to calculate the confidence of each recognized cloud label, and the loss function is calculated based on the recognition result of the cloud label and the cloud label annotated in the sample full-sky cloud map. According to the loss function, the network parameters of the cloud recognition network are adjusted to achieve multi-epoch training until the training converges.

[0064] In order to further solve the problem of imbalance between samples, the loss weight of each cloud-like label is added to the loss function. In one embodiment, the loss weight of each cloud-like label in the balanced sample set is calculated as M / (m*η), where M is the total number of sample full-sky cloud images contained in the balanced sample set, m is the number of sample full-sky cloud images containing cloud-like labels in the balanced sample set, and η is a constant factor, which is set to 2.3 according to the actual data set used in one example. Then, the loss function is calculated based on the loss weight of each cloud-like label and the cloud recognition network is trained according to the loss function. In actual operation, the calculated loss weight of each cloud-like label is passed as a parameter into the loss function to adjust pos_weight. In one example, paddle.nn.functiona.binary_cross_entropy_with_logits is called to calculate the loss function, and the calculated loss weight of each cloud-like label is directly passed into the function. Other loss functions can also be used in practice. By defining the loss weight of each cloud-like label by the above method, the cloud-like recognition network can be biased towards the learning of a small number of samples in model training, so as to better learn the characteristics of rare cloud-like categories.

[0065] In a specific example, the batch_size used in training is 32, the learning rate is 0.001, and the Cosine training strategy is used to optimize the training process of the deep learning model. During the training process, this strategy decays the learning rate in the form of a cosine function as the number of training steps increases, so as to achieve a smoother learning rate change and better convergence effect. At the same time, conventional data enhancement methods such as RandCropImage and RandFlipImage are used in the training process. During each epoch of training, the performance of the test set is evaluated using an evaluation method. The evaluation methods are HammingDistance and Accuracy.

[0066] Step 5, using the trained cloud shape recognition network to perform ground-based cloud shape recognition on the all-sky cloud image to be identified to output the cloud shape recognition result of the all-sky cloud image to be identified, including performing preprocessing processes such as image scaling on the all-sky cloud image to be identified and then inputting the trained cloud shape recognition network, outputting the cloud shape label of the all-sky cloud image to obtain the cloud shape recognition result.

[0067] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A cloud-like recognition method for ground-based clouds based on a lightweight neural network, characterized in that: The ground-based cloud shape recognition method comprises: Obtain an initial sample set including several sample full-sky cloud images; Determine the cloud image label value of the sample full sky cloud image according to the label value of the cloud shape label contained in each sample full sky cloud image in the initial sample set, wherein the label value of each type of cloud shape label is different and unique, and the cloud image label value of each sample full sky cloud image is the sum of the label values ​​of each type of non-repetitive cloud shape label contained in the sample full sky cloud image; According to the distribution of the cloud image label values ​​of each sample full sky cloud image, an oversampling operation is performed on the initial sample set to balance the number of different cloud shape label combinations and obtain a balanced sample set; The cloud recognition network is trained by using the balanced sample set. The constructed cloud recognition network includes an Inception module, multiple residual modules and an SRM module. The Inception module is used for multi-scale feature extraction. The SRM module adaptively recalibrates the intermediate feature map by using the style of the intermediate feature map. The multiple residual modules are used to realize the extraction of deep features. Each residual module is constructed based on the Ghost module and the CA attention mechanism, and feature reuse is realized by using residual connection. Each residual module includes a convolution layer, a Ghost module, a CA attention mechanism, a 1*1 convolution layer and an activation function connected in sequence. The feature map F1 input into the residual module is passed through the convolution layer to obtain a feature map F2 and input into the Ghost module. The feature map F2 is respectively processed in parallel by identity mapping and linear transformation in the Ghost module, and channel splicing is realized to obtain a feature map F3. The feature map F3 is sequentially passed through the CA attention mechanism and the 1*1 convolution layer to obtain a feature map F4. The feature map F4 is residually connected with the feature map F1, and then passes through the activation function and outputs the residual module. The trained cloud shape recognition network is used to perform ground-based cloud shape recognition on the all-sky cloud image to be recognized, so as to output a cloud shape recognition result of the all-sky cloud image to be recognized.

2. The ground cloud shape recognition method according to claim 1, characterized in that: The sample full-sky cloud image in the initial sample set contains a total of 31 different cloud shape labels, including cloud-free labels, visibility obstruction labels, and 29 different cloud shape classification labels, each of which corresponds to a cloud shape category of ground-based clouds; Each sample all-sky cloud map contains only a cloud-free label, or only a visibility obstruction label, or at least one cloud shape classification label; wherein the cloud-free label indicates that the sample all-sky cloud map does not contain ground-based clouds, and the visibility obstruction label indicates that the cloud shape category of the ground-based clouds contained in the sample all-sky cloud map cannot be determined due to visibility obstruction.

3. The ground cloud shape recognition method according to claim 1, characterized in that: The cloud recognition network includes several convolutional layers, Inception modules, multiple residual modules, flattening layers and linear layers cascaded in sequence. A maximum pooling layer is added after each convolutional layer and after the Inception module, an SRM module is inserted between the convolutional layers, an SRM module is inserted before the Inception module, an SRM module is inserted before each residual module, a maximum pooling layer is added after the inserted SRM module, and an average pooling layer is added before the flattening layer.

4. The ground cloud shape recognition method according to claim 1, characterized in that: The feature map f1 input to the Inception module passes through the 3*3 convolution layer and the 1*1 convolution layer in sequence to obtain the feature map f2 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer and the 5*5 convolution layer in sequence to obtain the feature map f3 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer and the 3*3 convolution layer in sequence to obtain the feature map f4 and output it to the concat module. The feature map f1 also passes through the 1*1 convolution layer to obtain the feature map f5 and output it to the concat module; the concat module performs channel splicing on the feature maps f2, f3, f4 and f5 and outputs them.

5. The ground cloud shape recognition method according to claim 1, characterized in that: The performing an oversampling operation on the initial sample set comprises: Determine the number of samples corresponding to each cloud map label value and determine the maximum number of samples respectively, the number of samples corresponding to any cloud map label value is the number of sample full-sky cloud maps with the cloud map label value in the initial sample set, and the maximum number of samples is the maximum number of sample full-sky cloud maps with the same cloud map label value in the initial sample set; Each cloud map label value is traversed in turn, and the extraction number is determined according to the degree of difference between the number of samples corresponding to the traversed cloud map label value and the maximum number of samples, and according to the extraction number, sample full-sky cloud maps are extracted from the sample full-sky cloud maps with the currently traversed cloud map label value contained in the initial sample set and added to the initial sample set; after all non-repetitive cloud map label values ​​are traversed, it is determined that the oversampling operation is completed, and a balanced sample set is obtained after adding the sample full-sky cloud map to the initial sample set.

6. The ground cloud shape recognition method according to claim 5, characterized in that: The step of determining the number of samples to be extracted according to the difference between the number of samples corresponding to the traversed cloud map label values ​​and the maximum number of samples includes: The number of samples is determined to be [(Nn) / β], where N is the maximum number of samples, n is the number of samples corresponding to the currently traversed cloud map label value, β is the coefficient, and [(Nn) / β] means rounding (Nn) / β.

7. The ground cloud shape recognition method according to claim 5, characterized in that: Any two sample full-sky cloud images with the same cloud image label value may contain the same or different cloud shape label combinations.

8. The ground cloud shape recognition method according to claim 1, characterized in that: The using the balanced sample set to train the cloud recognition network comprises: The loss weight of each cloud shape label in the balanced sample set is calculated as M / (m*η), a loss function is calculated based on the loss weights of each cloud shape label, and the cloud recognition network is trained according to the loss function; wherein M is the total number of sample full-sky cloud images included in the balanced sample set, m is the number of sample full-sky cloud images containing the cloud shape label in the balanced sample set, and η is a constant factor.

Images