A method for identifying and predicting lightweight multimodal ground cloud images

Abstract

The present invention discloses a method for recognizing and predicting a lightweight multimodal ground-based cloud image, and relates to the field of computer vision technology. The method comprises: importing an input image into a pre-trained model, and performing model training iterations according to an optimization strategy to obtain a trained lightweight ground-based cloud image recognition and prediction model; the optimization strategy comprises using a loss function and a truth map for supervised learning; the input image comprises multimodal feature data of the ground-based cloud image; and the lightweight ground-based cloud image recognition and prediction model is used to predict the image to be tested, and a predicted image is determined as the final output. The present invention can achieve recognition and prediction of ground-based cloud images by constructing a lightweight network framework when computing power resources are limited.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for recognizing and predicting lightweight multimodal ground-based cloud images. Background Art

[0002] With the rapid development of renewable energy, wind power projects have become a vital component of the global energy industry. Ground-based cloud image saliency recognition and prediction technology plays a crucial role in wind power forecasting, closely tied to the widespread adoption of wind power generation technology, the need for power system stability, and the development of surveying and mapping technologies and artificial intelligence. In wind power forecasting, ground-based cloud image saliency detection technology is primarily used for cloud layer identification and classification. By performing saliency detection on ground-based cloud images, different cloud types, such as cumulus, stratus, and cirrus, can be accurately identified. Different cloud types have varying impacts on wind energy generation and transmission, making accurate cloud type identification crucial for improving the accuracy of wind power forecasts.

[0003] Saliency recognition and prediction techniques for ground-based cloud images can be categorized into two main approaches: traditional methods and deep learning. Traditional methods primarily leverage heuristic priors or intuitive human perception to detect and segment salient target regions, generating saliency maps that depict attributes such as bright color, texture, strong contrast, and compactness. However, these saliency maps are not ideal for complex or multi-target images and are not applicable to novel scenes or complex targets, resulting in significant limitations. Existing deep learning methods typically require significant computational resources to improve detection performance, which presents new challenges when applied to devices with limited computing power.

[0004] Current mainstream methods have the following shortcomings: First, due to the large differences in structural morphology and boundary details of salient objects, it is difficult to identify and depict target details in complex scenes; second, the information flow provided by semantic information and target details in traditional architectures is limited, which cannot cover the entire salient object and easily leads to incomplete predictions; and traditional architectures have difficulty capturing temporal target details and robust salient information that are feature invariant under certain transformations, which reduces the recognition and prediction capabilities of targets in complex noisy scenes; finally, low contrast between targets and backgrounds is common in ground-based cloud images, and misjudgment of feature categories will increase the difficulty of identifying and locating target boundaries in low contrast, thereby affecting the accuracy of predictions.

[0005] Therefore, current mainstream methods are unable to ensure the accuracy of target detection when computing resources are limited. Summary of the Invention

[0006] The purpose of the present invention is to provide a lightweight multimodal ground-based cloud image recognition and prediction method, which can realize the recognition and prediction of ground-based cloud images by constructing a lightweight network framework under the condition of limited computing resources.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A lightweight multimodal ground-based cloud image recognition and prediction method, comprising:

[0009] Acquire an input image; the input image includes multimodal feature data of a ground-based cloud image;

[0010] Constructing a pre-training model; the pre-training model includes a U-shaped connected encoding part and a decoding part; the decoding part includes an edge guidance module, a semantic fusion module and a spectrum conversion module; the input ends of the edge guidance module and the semantic fusion module are both connected to the encoding part, and the output ends are both connected to the spectrum conversion module;

[0011] Importing the input image into the encoding part for feature encoding, inputting the encoded feature data into the decoding part for feature fusion and decoding, and performing model training iterations according to an optimization strategy to obtain a trained lightweight ground-based cloud image recognition and prediction model; the optimization strategy includes using a loss function and a truth map for supervised learning;

[0012] The lightweight ground-based cloud image recognition and prediction model is used to predict the image to be tested, and the predicted image to be finally output is determined.

[0013] Optionally, the process of importing the input image into the encoding part for feature encoding specifically includes:

[0014] The input image is encoded using MobileNet as a baseline to obtain encoded first-stage features, second-stage features, third-stage features, fourth-stage features, and fifth-stage features; wherein the first-stage features and the second-stage features are used to describe position information and target details; and the third-stage features, the fourth-stage features, and the fifth-stage features are used to describe abstract semantic information.

[0015] Optionally, the process of inputting the encoded feature data into the decoding part for feature fusion and decoding specifically includes:

[0016] Inputting the first-stage features and the second-stage features into the edge-guided module for operation to obtain spatial features with different receptive fields, and performing element-by-element addition / subtraction, Hadamard product operation and channel cascade strategy on the spatial features to obtain fine-grained target detail features;

[0017] Performing semantic compression and dilated pyramid pooling on the fifth-stage features to generate convolution kernels with different expansion rates for the semantic fusion module;

[0018] Based on the convolution kernels with different expansion rates, the third stage features and the fourth stage features are input into the semantic fusion module for operation to obtain multi-scale features of semantic information, and a depth-wise separable convolution operation and a channel cascade strategy are performed on the multi-scale features to obtain rich semantic information features;

[0019] The fine-grained target detail features and the rich semantic information features are input into the spectrum transformation module, the spatial domain features are combined with the frequency domain features, and multi-scale feature fusion and decoding are performed through channel cascade and step-by-step fusion transformation operations.

[0020] Optionally, the first-stage features and the second-stage features are input into the edge guidance module for calculation, and the specific process includes:

[0021] Multi-scale convolution operation is performed on the first-stage features and the second-stage features to obtain spatial features with different receptive fields; the formula for the multi-scale convolution operation is:

[0022]

[0023] in, and represent the first stage characteristics and the second stage characteristics respectively, f i 1 and f i 2 They represent the generated spatial features, Ψ represents the bilinear interpolation operation, C 2i-1 (i=1,…,4) represents the multi-scale convolution kernel, DSConv(*,*) represents the depth-wise separable convolution operation;

[0024] Based on the spatial feature f1 1 、 f1 2 and Through convolution operation, average pooling operation and element-by-element subtraction operation, fusion features and prediction features are obtained. The mathematical formula is expressed as:

[0025]

[0026] Among them, g1 and g2 represent the generated fusion features, and Represent the generated prediction features, Conv(*,ε i ) represents the convolution calculation with a kernel size of 1×1, represents the average pooling operation; ρ represents the PReLU function;

[0027] In the spatial feature f3 1 、 f3 2 、 The channel cascade strategy, Hadamard product operation and element-by-element addition operation are used to integrate the spatial detail feature information on the fusion features g1 and g2 to obtain the fine-grained target detail features. The mathematical formula is expressed as:

[0028]

[0029] D out =CCS(a1,a2)

[0030] Among them, × represents the Hadamard product operation, σ represents the Sigmoid function, represents two-dimensional batch normalization, D out represents fine-grained target detail features, CCS represents the channel cascade strategy, h1 and h2 represent the corresponding intermediate layer features, and a1 and a2 represent the corresponding dense spatial features.

[0031] Optionally, based on the convolution kernels with different expansion rates, the third stage features and the fourth stage features are input into the semantic fusion module for operation, and the specific process includes:

[0032] Performing a dynamic depth convolution operation with an expansion rate on the third stage features and the fourth stage features to obtain multi-scale features and The calculation formula is:

[0033]

[0034] Among them, DConv represents the dynamic depth convolution operation, ω and υ represent the convolution kernel matrix, r i (r i =1,2,3) represents the expansion ratio, and They represent the third stage characteristics and the fourth stage characteristics respectively;

[0035] Multi-scale features are processed using element-wise addition and point-wise convolution operations. and Perform integration to generate features y1 and y2. The calculation formula is:

[0036]

[0037] Among them, + represents element-by-element addition operation, Ψ represents bilinear interpolation operation, PConv is point-by-point convolution, DSConv represents depth-wise separable convolution, αi and β i Both represent convolution kernels (i=1,2);

[0038] Perform channel cascade strategy on features y1 and y2 to fuse feature information and obtain rich semantic information feature h out , the calculation formula is:

[0039] h out =CCS(y1,y2)

[0040] CCS stands for channel cascade strategy.

[0041] Optionally, the loss function in the optimization strategy is constructed based on a first loss function and a second loss function; the first loss function is composed of a binary cross entropy loss function and an intersection-over-union loss function; and the second loss function adopts an L2 loss function.

[0042] Optionally, the first loss function and the second loss function specifically include:

[0043] Define the binary cross entropy loss function and Intersection-over-Union loss function for:

[0044]

[0045] Among them, p(x,y)∈[0,1] is the predicted probability of the image pixel point (x,y), and g(x,y)∈[0,1] is the true value label of the image pixel point (x,y);

[0046] Construct the first loss function of stage i:

[0047] Construct the second loss function: Among them, L mse represents the mean square error loss, ψ is the parameter correction linear unit, and represent the prediction features generated in the edge guidance module respectively.

[0048] Optionally, the loss function in the optimization strategy is expressed as:

[0049]

[0050] Where α=1 and β=0.5.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] The present invention discloses a lightweight multimodal ground-based cloud image recognition and prediction method. The method comprises importing an input image into a pre-trained model and iterating model training according to an optimization strategy to obtain a trained lightweight ground-based cloud image recognition and prediction model. The optimization strategy comprises supervised learning using a loss function and a truth map. The input image includes multimodal feature data of the ground-based cloud image. The lightweight ground-based cloud image recognition and prediction model is used to predict the image to be tested, and a predicted image is ultimately output. The present invention can achieve ground-based cloud image recognition and prediction by constructing a lightweight network framework under limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 Schematic diagram of the flow of the lightweight multimodal ground-based cloud image recognition and prediction method of the present invention;

[0055] Figure 2 3 is a comparison chart of the significance prediction results using the method of the present invention and other methods in this embodiment. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The purpose of the present invention is to provide a lightweight multimodal ground-based cloud image recognition and prediction method, which can realize the recognition and prediction of ground-based cloud images by constructing a lightweight network framework under the condition of limited computing resources.

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] like Figure 1As shown, the present invention provides a lightweight multimodal ground-based cloud image recognition and prediction method. Considering the differences in multi-level feature spaces, a multi-feature fusion network is first designed as a baseline for saliency inference. To effectively aggregate fine-grained visual features and abstract semantic information while ensuring consistency in the feature space, an edge guidance module and a semantic fusion module are proposed and integrated into the multi-feature fusion network. To enhance the robustness of saliency information, a spectral transformation module combining time-domain and frequency-domain features is proposed to highlight salient target details and encourage saliency prediction. In terms of optimization settings, a saliency prediction result optimization strategy is proposed to remove interference information and noise in complex images. This strategy uses binary cross-entropy loss and intersection-over-union loss to calculate the total pixel loss, aiming to optimize the global structure of the input image and improve the accuracy of saliency prediction. A mean square error loss is also introduced to further express enhanced target details. In terms of result comparison, experimental results of the proposed method on four ground-based cloud image datasets show that the proposed method outperforms other existing advanced methods. The specific steps are:

[0060] An input image is acquired; the input image includes multimodal feature data of a ground-based cloud image.

[0061] Constructing a pre-training model; the pre-training model includes a U-shaped connected encoding part and a decoding part; the decoding part includes an edge guidance module, a semantic fusion module and a spectrum conversion module; the input ends of the edge guidance module and the semantic fusion module are both connected to the encoding part, and the output ends are both connected to the spectrum conversion module;

[0062] Importing the input image into the encoding part for feature encoding, inputting the encoded feature data into the decoding part for feature fusion and decoding, and performing model training iterations according to an optimization strategy to obtain a trained lightweight ground-based cloud image recognition and prediction model; the optimization strategy includes using a loss function and a truth map for supervised learning;

[0063] The lightweight ground-based cloud image recognition and prediction model is used to predict the image to be tested, and the predicted image to be finally output is determined.

[0064] As a specific embodiment, the overall structure is as follows Figure 1 As shown in Figure 2, MobileNet is used as a baseline in the image encoding stage. Features are encoded in a top-down and bottom-up manner based on a U-shaped architecture, where E1-E5 represent low-level features to high-level features, respectively. E1 and E2 contain rich location information and object details, which are more conducive to obtaining saliency boundaries. As high-level features, E3, E4, and E5 contain abstract semantic information, which is more conducive to understanding object categories.

[0065] First, high-level features E5 are used to generate convolution kernels with different dilation rates for the semantic fusion module. Second, features E3 and E4 are used as input to this module to obtain multi-scale features with abstract semantic information. Depthwise separable convolution operations and a channel concatenation strategy are applied to these multi-scale features to obtain richer semantic information. Third, features E1 and E2 are used as input to the edge-guided module to obtain spatial features with different receptive fields. Element-wise addition / subtraction, Hadamard product operations, and a channel concatenation strategy are performed on these spatial features to generate output features with fine-grained object details. Fourth, these features, which contain rich semantic information and fine-grained object details, are used as input to the spectral transformation module, combining spatial and frequency domain features to obtain a more comprehensive object representation. Multi-scale feature fusion and decoding are then performed through channel concatenation and a step-by-step fusion transformation. Finally, a loss function is used to comprehensively supervise the saliency predictions at each level of the architecture's output, generating a final ground-truth saliency map, which is supervised in real time by the ground-truth saliency map.

[0066] The lightweight multimodal ground-based cloud image recognition and prediction framework proposed in this invention includes a semantic fusion module, an edge guidance module, and a spectrum transformation module. The specific calculation methods and formulas of each module are as follows.

[0067] 1. Semantic Fusion Module

[0068] Step 1: Input features and Perform dynamic depthwise convolution with dilation rate to obtain features and The detailed calculation formula is as follows:

[0069]

[0070] Among them, DConv represents the dynamic depth convolution operation, ω and υ represent the convolution kernel matrix, r i is the expansion ratio (r i =1,2,3), and They represent the third stage characteristics and the fourth stage characteristics respectively.

[0071] Step 2: Use element-wise addition and point-wise convolution operations to integrate multi-scale features and Thus generating features y1 and y2.

[0072]

[0073] Where + represents element-by-element addition, Ψ represents bilinear interpolation, PConv represents point-by-point convolution, DSConv represents depth-wise separable convolution, and α iand β i Both represent convolution kernels (i=1,2).

[0074] Step 3: Perform channel cascade strategy on features y1 and y2 to fuse feature information and finally output h out Expressed as:

[0075] h out =CCS(y1,y2) (3)

[0076] CCS stands for channel cascade strategy.

[0077] 2. Channel Cascading Strategy

[0078] Step 1: Perform scaled harmonic mapping and matrix multiplication operations on the input features x1 and x2∈{N,C,H,W} to obtain the activation feature tensor z.

[0079]

[0080] where · represents matrix multiplication, f i (i=1,2) represents the scale harmonic map, and τ represents the softmax function.

[0081] Step 2: Use inverse scale harmonic mapping on features y1, y2 and z to fuse and reconstruct spatial features, thereby obtaining features u1 and u2. The mathematical expression is shown in Equation (5).

[0082] u1=f1 -1 (y1·z),u2=f2 -1 (z·y2) (5)

[0083] Among them, f i -1 (i=1,2) represents the inverse scale harmonic mapping.

[0084] Step 3: Element-wise addition, channel-wise convolution, and depth-wise separable convolution operations are performed to generate the output features g out .

[0085]

[0086] Where DSConv(*,*) represents the depthwise separable convolution operation, and ω, υ, and μ represent the convolution kernel matrices.

[0087] 3. Edge guidance module

[0088] Step 1: Multi-scale convolution operations are performed on the input features and Thus generating spatial features f with different receptive fields i 1and f i 2 .

[0089]

[0090] Where Ψ is the bilinear interpolation algorithm, C 2i-1 (i=1,…,4) represents the multi-scale convolution kernel.

[0091] Step 2: Obtain fused features g1 and g2 and predicted features through convolution operation, average pooling operation and element-by-element subtraction operation and Its mathematical formula is as follows:

[0092]

[0093] Among them, g1 and g2 represent the generated fusion features, and Represent the generated prediction features, Conv(*,ε i ) represents the convolution calculation with a kernel size of 1×1, represents the average pooling operation; ρ represents the PReLU function.

[0094] Step 3: In the spatial feature f3 1 , f3 2 , The channel cascade strategy, Hadamard product operation and element-by-element addition operation are used to integrate the spatial detail feature information on the fusion features g1 and g2 to obtain the spatial aggregation feature D out Expressed as:

[0095]

[0096] Among them, × represents the Hadamard product operation, σ represents the Sigmoid function, represents two-dimensional batch normalization, D out represents fine-grained target detail features, CCS represents the channel cascade strategy, h1 and h2 represent the corresponding intermediate layer features, and a1 and a2 represent the corresponding dense spatial features.

[0097] 4. Spectrum transformation module

[0098] Step 1: Input feature f in Perform convolution operation to obtain hierarchical features X G and X L .

[0099] X G =DSConv(f in ,ω),XL =DSConv(f in ,υ) (10)

[0100] Where ω and υ represent the convolution kernel of size 3×3.

[0101] Step 2: Spectral transformation and convolution operation are applied to feature X G , to obtain the transformed feature X GG and X GL .

[0102] X GG =ST(X G ), X GL =DSConv(X G ,θ) (11)

[0103] Where ST represents the Fourier transform module, θ∈R 1×1×C .

[0104] Step 3: Multi-region convolution operations are performed on the level feature X L On the top, obtain the spatial domain feature X LG and X LL as follows:

[0105] X LG =MR.C(X L ,m),X LL =MR.C(X L ,n) (12)

[0106] Among them, MR.C represents the multi-region convolution operation module, and m and n both represent convolution kernels.

[0107] Step 4: Multi-channel output features are integrated through element-by-element addition and nonlinear transformation to obtain the fused feature X OG and X OL It is expressed as shown in formula (13).

[0108] X OG =δ(X GG +X LG ), X OL =δ(X GL +X LL ) (13)

[0109] Where δ represents the ReLU function.

[0110] Step 5: Multi-channel features X OG and X OL It is cascaded and compressed to enrich the pixel-level data representation, which is more conducive to the positioning and detection of the target, and finally outputs the feature f out It is expressed as the following mathematical formula.

[0111] f out =DSConv(τ(Concate([X OG ,X OL ],dim=1)),θ) (14)

[0112] Where τ represents channel compression and θ is the convolution kernel of size 1×1.

[0113] 5. Multi-region convolution operation module

[0114] Step 1: For the input feature x in Perform scale-harmonic tensor partitioning to generate small-scale features U L , U R , L L and L R .

[0115] U L ,U R ,L L ,L R =ψ(x in ) (15)

[0116] Where ψ represents the scale adjustment and tensor partition, x in ∈{N,C,H,W},

[0117] Step 2: In Feature U L , U R , L L and L R Perform convolution operation on it to obtain multi-region features x i (i=1,…,4).

[0118]

[0119] where ω i (i=1,…,4) represents the convolution kernel matrix.

[0120] Step 3: Use feature concatenation and convolution operations to perform multi-region feature i Reconstruct and fuse, output feature x out It is expressed as follows:

[0121]

[0122] Concate(*, dim=3) means concatenation at the width of the tensor, and Concate(*, dim=2) means concatenation at the height of the tensor. is the convolution kernel matrix, and x out ∈{N,C,H,W}.

[0123] 6. Fourier transform module

[0124] Step 1: Input feature y in Perform Fourier transform and feature reshaping to obtain frequency domain features and

[0125]

[0126] Where FFT stands for Fourier transform, ρ is the feature reshaping, θ i (i=1,2) represents a convolution kernel of size 1×1.

[0127] Step 2: Frequency Domain Features and are merged and reshaped to create fused frequency domain features, and an inverse Fourier transform is applied to obtain features z in the time domain.

[0128]

[0129] Here, · represents matrix multiplication, κ represents structural transformation, and IFFT represents inverse Fourier transform.

[0130] Step 3: Element-wise addition and depth-wise separable convolution operations are used to integrate the time domain features to obtain the output feature y out It can be expressed as follows:

[0131] y out =DSConv(y1+y2+z,θ) (20)

[0132] Where DSConv(*,θ) represents a function with parameters θ∈R 1×1×C Depthwise separable convolution.

[0133] Optimizing strategies for results

[0134] In order to further optimize the saliency prediction results and remove the interference information and noise in the image, this scheme proposes a new loss function, which consists of two parts: binary cross entropy loss Sum intersection loss Ability to calculate the loss for each pixel, It aims to optimize the global structure of the input image. These two components complement each other, thereby improving the accuracy of saliency prediction. and The definition at stage i is as follows:

[0135]

[0136]

[0137] Among them, p(x,y)∈[0,1] is the predicted probability of pixel point (x,y), and g(x,y)∈[0,1] is the true value label of pixel point (x,y).

[0138] Therefore, the total loss function of stage i is As shown in formula (23):

[0139]

[0140] Then, L2 loss is used to further express the enhanced target details, and the function L2 is expressed as:

[0141]

[0142] Among them L mse represents the mean square error loss, ψ is the parameter correction linear unit, and represent the prediction features generated in the edge guidance module respectively.

[0143] Finally, the total loss L total It is expressed by mathematical formula (25):

[0144]

[0145] Where α=1 and β=0.5.

[0146] 3.2.2 Results of the technical solution of the present invention (or utility model)

[0147] In this example, model training was performed on the additionally annotated SWINYSEG dataset, and model evaluation was performed on the HBMSEG, SWIMSEG, SWINSEG, and SHWIMSEG datasets. Information about the training and test datasets is as follows: the SWINYSEG dataset contains 6,768 sky and cloud images with 10 cloud image categories, all of which were used for training; the HBMSEG dataset contains 11,000 complex images, which were used for testing; the SWIMSEG dataset contains 1,013 test images of sky and cloud patches; the SHWIMSEG dataset contains 156 complex test images, most of which have low, medium, and high exposure characteristics; and the SWINSEG dataset contains 115 nighttime images of the sky and clouds.

[0148] In this example, the F-measure, MAE, and S-measure comparison results of the prediction technology of the present invention and other technologies on the SWIMSEG, SWINSEG, SHWIMSEG, and HBMSEG datasets are shown in Table 1. From the results in Table 1, we can see that the prediction technology of the present invention shows good performance on almost all ground cloud image databases in terms of the three scoring indicators, which also demonstrates the effectiveness and usability of the method of the present invention. Specifically, the prediction technology of the present invention achieves scores of 0.900, 0.931, 0.711, and 0.539 respectively when using the S-measure metric to calculate scores on the four datasets. In addition, in most cases, the method proposed in this solution outperforms other existing lightweight state-of-the-art methods, such as HVPNet and SAMNet, which also proves the reliability of the technology of this solution. In particular, when the S-measure metric (F-measure metric) is used to calculate the scores on the SWIMSEG, SWINSEG, SHWIMSEG and HBMSEG datasets, the performance of our method is better than SAMNet, with improvements of 7.6% (6.1%), 11.1% (6.4%), 4.9% (4.3%) and 19.8% (17.1%), respectively, which greatly reduces the error.

[0149] Table 1 Comparison of F-measure, S-measure and MAE between the proposed method and other methods on four data sets

[0150]

[0151] This example also provides a comparison of visual results of saliency prediction of ground cloud images using the technology of the present invention and other existing state-of-the-art methods on four datasets using some challenging scenes (including high exposure (1st and 2nd rows), dark environment (3rd and 4th rows), clumping clouds (5th and 7th rows) and reticular clouds (6th and 8th rows). Figure 2As shown, it can be concluded that the proposed method exhibits more continuous and smoother target detection and segmentation performance, providing a clear description of target morphology. Specifically, row 1 provides a cloud image in a high-exposure scene. The proposed method clearly identifies and describes the details of the cloud clusters in the high-exposure area, achieving satisfactory results. In row 7, despite the presence of a large cloud layer, most methods are unable to clearly locate and segment the entire prominent cloud region. Compared to row 5, the reticular cloud image in row 8 has a target cloud color that closely resembles the sky background, making locating and segmenting reticular clouds more challenging, primarily due to their more dispersed and complex distribution. As shown in the visualization results, the proposed method accurately locates the clouds and segments their shapes, while detecting fewer false pixels and noise. Furthermore, the proposed method demonstrates effectiveness in suppressing background noise in dark environments and accurately locating and segmenting prominent cloud targets. As shown in rows 3 and 4, the image depicts the distribution of reticular clouds in a dark environment (row 4). Even when the distribution is more dispersed and complex, the proposed method is able to accurately identify the location of cloud targets.

[0152] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0153] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A lightweight multimodal ground-based cloud image recognition and prediction method, characterized by: include: Acquire an input image; the input image includes multimodal feature data of a ground-based cloud image; Constructing a pre-training model; the pre-training model includes a U-shaped connected encoding part and a decoding part; the decoding part includes an edge guidance module, a semantic fusion module and a spectrum conversion module; the input ends of the edge guidance module and the semantic fusion module are both connected to the encoding part, and the output ends are both connected to the spectrum conversion module; the edge guidance module is used to obtain spatial features of different receptive fields; the semantic fusion module is used to obtain multi-scale features of semantic information; the spectrum conversion module is used to combine spatial domain features with frequency domain features; Importing the input image into the encoding part for feature encoding, inputting the encoded feature data into the decoding part for feature fusion and decoding, and performing model training iterations according to an optimization strategy to obtain a trained lightweight ground-based cloud image recognition and prediction model; the optimization strategy includes using a loss function and a truth map for supervised learning; The lightweight ground-based cloud image recognition and prediction model is used to predict the image to be tested, and the predicted image to be finally output is determined.

2. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 1, characterized in that: The process of importing the input image into the encoding part for feature encoding specifically includes: The input image is encoded using MobileNet as a baseline to obtain encoded first-stage features, second-stage features, third-stage features, fourth-stage features, and fifth-stage features; wherein the first-stage features and the second-stage features are used to describe position information and target details; and the third-stage features, the fourth-stage features, and the fifth-stage features are used to describe abstract semantic information.

3. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 2, characterized in that: The process of inputting the encoded feature data into the decoding part for feature fusion and decoding specifically includes: Inputting the first-stage features and the second-stage features into the edge-guided module for operation to obtain spatial features with different receptive fields, and performing element-by-element addition / subtraction, Hadamard product operation and channel cascade strategy on the spatial features to obtain fine-grained target detail features; Performing semantic compression and dilated pyramid pooling on the fifth-stage features to generate convolution kernels with different expansion rates for the semantic fusion module; Based on the convolution kernels with different expansion rates, the third stage features and the fourth stage features are input into the semantic fusion module for operation to obtain multi-scale features of semantic information, and a depth-wise separable convolution operation and a channel cascade strategy are performed on the multi-scale features to obtain rich semantic information features; The fine-grained target detail features and the rich semantic information features are input into the spectrum transformation module, the spatial domain features are combined with the frequency domain features, and multi-scale feature fusion and decoding are performed through channel cascade and step-by-step fusion transformation operations.

4. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 3 is characterized in that: The first-stage features and the second-stage features are input into the edge guidance module for operation. The specific process includes: Multi-scale convolution operations are performed on the first-stage features and the second-stage features to obtain spatial features with different receptive fields. The formula for the multi-scale convolution operation is: in, and represent the first stage characteristics and the second stage characteristics respectively, f i 1 and f i 2 They represent the generated spatial features, Ψ represents the bilinear interpolation operation, C 2i-1 (i=1,…,4) represents the multi-scale convolution kernel, DSConv(*,*) represents the depth-wise separable convolution operation; Based on spatial feature f1 1 、 f1 2 and Through convolution operation, average pooling operation and element-by-element subtraction operation, fusion features and prediction features are obtained. The mathematical formula is expressed as: Among them, g1 and g2 represent the generated fusion features, and Represent the generated prediction features, Conv(*,ε i ) represents the convolution calculation with a convolution kernel size of 1×1. represents the average pooling operation; ρ represents the PReLU function; Features in space The channel cascade strategy, Hadamard product operation and element-by-element addition operation are used to integrate the spatial detail feature information on the fusion features g1 and g2 to obtain the fine-grained target detail features. The mathematical formula is expressed as: <h2 style=";text-align:left;direction:ltr">D<h2 style=";text-align:left;direction:ltr"> out <h2 style=";text-align:left;direction:ltr"> =CCS(a1,a2) Among them, × represents the Hadamard product operation, σ represents the Sigmoid function, represents two-dimensional batch normalization, D out represents fine-grained target detail features, CCS represents the channel cascade strategy, h1 and h2 represent the corresponding intermediate layer features, and a1 and a2 represent the corresponding dense spatial features.

5. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 3 is characterized in that: Based on the convolution kernels with different expansion rates, the third stage features and the fourth stage features are input into the semantic fusion module for operation. The specific process includes: Performing a dynamic depth convolution operation with an expansion rate on the third stage features and the fourth stage features to obtain multi-scale features and The calculation formula is: Among them, DConv represents the dynamic depth convolution operation, ω and υ represent the convolution kernel matrix, r i (r i =1,2,3) represents the expansion ratio, and They represent the third stage characteristics and the fourth stage characteristics respectively; Multi-scale features are processed using element-wise addition and point-wise convolution operations. and Perform integration to generate features y1 and y2. The calculation formula is: Among them, + represents element-by-element addition operation, Ψ represents bilinear interpolation operation, PConv is point-by-point convolution, DSConv represents depth-wise separable convolution, α i and β i Both represent convolution kernels (i=1,2); Perform channel cascade strategy on features y1 and y2 to fuse feature information and obtain rich semantic information feature h out , the calculation formula is: h out =CCS(y1,y2) CCS stands for channel cascade strategy.

6. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 1, characterized in that: The loss function in the optimization strategy is constructed based on a first loss function and a second loss function; the first loss function is composed of a binary cross entropy loss function and an intersection-over-union loss function; the second loss function adopts an L2 loss function.

7. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 6, characterized in that: The first loss function and the second loss function specifically include: Define the binary cross entropy loss function and Intersection-over-Union loss function for: Among them, p(x,y)∈[0,1] is the predicted probability of the image pixel point (x,y), and g(x,y)∈[0,1] is the true value label of the image pixel point (x,y); Construct the first loss function of stage i: Construct the second loss function: Among them, L mse represents the mean square error loss, ψ is the parameter correction linear unit, and represent the prediction features generated in the edge guidance module respectively.

8. The method for recognizing and predicting lightweight multimodal ground-based cloud images according to claim 7, characterized in that: The loss function in the optimization strategy is expressed as: Where α=1 and β=0.5.

Images