A multi-modal cloud classification method based on a visual language model

Abstract

The embodiment of the application discloses a kind of multi-modal cloud classification methods based on visual language model, the method comprises: constructing text generation module, generates the cloud description text containing high-level semantic information;Cloud multi-modal feature extraction and fusion module is constructed, cloud image, cloud description text and millimeter wave cloud radar observation data are carried out feature coding and adaptive fusion, obtain multi-modal feature representation;Classifier module is constructed, and the cloud shape classification result of training cloud image is obtained;Total loss function is constructed, and the multi-modal cloud shape classification model consisting of cloud multi-modal feature extraction and fusion module and classifier module is optimized;Test cloud image, and the cloud description text and millimeter wave cloud radar observation data corresponding to test cloud image are input into multi-modal cloud shape classification model and carry out cloud shape classification.The application is realized by introducing high-level semantic cloud description and designing cloud multi-modal adapter, effective fusion of image, radar and text information, improve the accuracy and robustness of cloud shape classification.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a multimodal cloud classification method based on a visual language model. Background Technology

[0002] In recent years, cloud classification has received widespread attention in meteorological research and related application fields due to its important role in improving the level of automated meteorological monitoring and enhancing disaster early warning capabilities. Existing cloud classification methods are mainly based on cloud image data or millimeter-wave cloud radar observation data.

[0003] Cloud image-based classification methods typically employ deep learning techniques. For example, DeepCloud was the first to introduce deep convolutional neural networks into the field of cloud classification, using cloud images as input to achieve automatic classification. CloudNet constructed a convolutional neural network model containing five convolutional layers and two fully connected layers to extract cloud image features and perform classification. Subsequently, CloudSwinNet combined convolutional neural networks with a Transformer structure to simultaneously acquire local detail features and global contextual information.

[0004] In classification methods based on millimeter-wave cloud radar observation data, researchers typically utilize physical indicators such as radar reflectivity and linear depolarization ratio for classification. For example, Kodilkar et al. constructed a fuzzy logic system to achieve cloud shape recognition; Nizar et al.'s CloudSense method combines a rule-based discrimination mechanism with a lightweight gradient booster to achieve cloud shape classification. Furthermore, considering the complementary relationship between cloud images and millimeter-wave cloud radar observation data, Liu et al. proposed a multimodal fusion Transformer structure to jointly model the features of both modalities to improve classification performance.

[0005] However, the above methods mainly rely on low-level semantic information such as image texture, edge features and radar physical parameters for feature extraction and fusion, lacking modeling of high-level semantic information such as cloud morphology, making it difficult to conform to human meteorological cognition and limiting further improvement in classification performance.

[0006] Visual language models possess cross-modal semantic understanding and alignment capabilities, enabling joint modeling using both visual and linguistic information to achieve high-level semantic expression and cross-modal feature learning. Therefore, introducing visual language models into cloud classification tasks helps integrate multimodal information such as cloud images, millimeter-wave cloud radar observation data, and cloud description text, providing a new technical path for cloud classification. In recent years, to adapt to downstream task requirements, researchers have typically introduced adapter mechanisms into visual language models for efficient parameter fine-tuning. For example, Gao et al. used a bottleneck structure as an adapter to improve knowledge transfer capabilities; Zhang et al. proposed Tip-Adapter, achieving training-free adaptation by constructing a key-value cache; and Yang et al. proposed MMA, enhancing semantic alignment capabilities by projecting different modalities onto a shared feature space.

[0007] However, existing fine-tuning methods typically only insert independent adapter structures after each Transformer module for different modality encoding branches, lacking effective interaction and fusion mechanisms between modalities. These methods focus more on parallel feature adjustments for different modalities without establishing cross-modal semantic associations, making it difficult to fully leverage the advantages of visual language models in multimodal fusion and limiting further improvements in classification performance.

[0008] Therefore, in view of the limitations of existing cloud-like classification methods, it is necessary to propose a cloud-like classification method that can incorporate high-level semantic information and achieve multimodal deep fusion in order to improve classification accuracy and generalization ability. Summary of the Invention

[0009] The purpose of this invention is to address the problem that existing cloud classification methods only utilize low-level semantic information and lack high-level semantic information, thus failing to adapt to multimodal cloud classification tasks. To this end, this invention provides a multimodal cloud classification method based on a visual language model.

[0010] To achieve the aforementioned objective, the present invention proposes a multimodal cloud classification method based on a visual language model, comprising the following steps:

[0011] Step S1: Construct a text generation module. In this module, a large language model is used to generate corresponding cloud description text containing preset cloud features based on the cloud image. The cloud images include training cloud images and test cloud images;

[0012] Step S2: Construct a cloud multimodal feature extraction and fusion module based on a visual language model. In the cloud multimodal feature extraction and fusion module, a text encoder and a radar encoder are constructed to encode the cloud description text corresponding to the training cloud image and the millimeter-wave cloud radar observation data corresponding to the training cloud image, respectively, to obtain text features and radar features. An image encoder is constructed to encode the features of the training cloud image and adaptively fuse it with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features.

[0013] Step S3: Construct a classifier module by inputting the image features into the classifier module and outputting the cloud classification result of the training cloud image;

[0014] Step S4: Construct a total loss function based on the text features, radar features, and image features, and optimize the multimodal cloud classification model composed of the cloud multimodal feature extraction and fusion module and the classifier module to obtain the final multimodal cloud classification model;

[0015] Step S5: Input the test cloud image, the cloud description text corresponding to the test cloud image, and the millimeter-wave cloud radar observation data into the multimodal cloud classification model to obtain the cloud classification result of the test cloud image.

[0016] Optionally, the radar encoder in the cloud multimodal feature extraction and fusion module has the same structure as the image encoder in the visual language model.

[0017] Optionally, the step S2 of constructing a text encoder to encode the features of the cloud description text corresponding to the training cloud image to obtain text features includes the following steps:

[0018] Step S211, construct a text encoder, wherein the text encoder includes Layer Transformer module;

[0019] Step S212: Introduce a bottleneck adapter after each Transformer module of the text encoder and initialize the parameters of the constructed text encoder.

[0020] Step S213: The cloud description text is converted into a word identifier sequence through byte pair encoding, then mapped through the text embedding layer in the visual language model and superimposed with the first preset position encoding to obtain the initial text features. ;

[0021] Step S214, the initial text features The text encoder is input for feature encoding, and the EOS token is extracted as a text feature. .

[0022] Optionally, in step S214, the text encoder... The output of the layer Transformer module is :

[0023]

[0024] in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module;

[0025] No. Output of the bottleneck adapter after the Transformer module Represented as:

[0026]

[0027] in, For a dimension reduction matrix, For an upgraded matrix, This is the ReLU activation function.

[0028] Optionally, the step S2 of constructing a radar encoder to encode the features of millimeter-wave cloud radar observation data corresponding to the training cloud image to obtain radar features includes the following steps:

[0029] Step S221, construct a radar encoder, wherein the radar encoder includes Layer Transformer module;

[0030] Step S222: Introduce a bottleneck adapter after each Transformer module of the radar encoder and initialize the parameters of the constructed radar encoder.

[0031] Step S223: The millimeter-wave cloud radar observation data is converted into cloud-related indicators. The cloud-related indicators are processed by the radar embedding layer and superimposed with [CLS] token embedding and the second preset position code to obtain the initial radar features. The radar embedding layer is built on a fully connected layer, and its structure is similar to that of the text embedding layer in the visual language model.

[0032] Step S224, the initial radar features The input to the radar encoder is used for feature encoding, and the [CLS] token is extracted and embedded as a radar feature. .

[0033] Optionally, in step S224, the radar encoder... The output of the layer Transformer module is :

[0034]

[0035] in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module;

[0036] No. Output of the bottleneck adapter after the Transformer module Represented as:

[0037]

[0038] in, For a dimension reduction matrix, For an upgraded matrix, This is the ReLU activation function.

[0039] Optionally, the step S2, which involves constructing an image encoder to encode features of the training cloud image and adaptively fusing them with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features, includes the following steps:

[0040] Step S231: Construct an image encoder and a cloud multimodal adapter, wherein the image encoder includes... Layer Transformer module;

[0041] Step S232: Introduce a cloud multimodal adapter after each Transformer module of the image encoder, and initialize the parameters of the constructed image encoder.

[0042] Step S233: Divide the training cloud image into several image blocks, process the image blocks through the image embedding layer in the visual language model, and superimpose them with [CLS] token embedding and third preset position encoding to obtain initial image features. ;

[0043] Step S234, the initial image features The image encoder is input for feature encoding, and during the encoding process, it is adaptively fused with the corresponding intermediate layer output features of the text encoder and radar encoder. After feature encoding is completed, the [CLS] token embedding is extracted as the image feature. .

[0044] Optionally, in step S234, the image encoder... The output of the layer Transformer module is :

[0045]

[0046] in, For the first The output of the stratus cloud multimodal adapter, Indicates the first Layer Transformer module;

[0047] No. Output of the cloud multimodal adapter after the Transformer module Represented as:

[0048]

[0049]

[0050] in, For a dimension reduction matrix, For an upgraded matrix, It is the ReLU activation function. , and It is a learnable parameter that can automatically determine the optimal contribution of each mode. and They are defined as follows:

[0051]

[0052]

[0053] in, Scaling factor These represent the first and second parts of the text encoder and radar encoder, respectively. The output of the layer Transformer module, and This is a dimension-reduced matrix.

[0054] Optionally, the total loss function in step S4 includes a three-modal contrastive learning loss function and a cross-entropy loss function, wherein the three-modal contrastive learning loss function is established based on the text features, radar features, and image features.

[0055] Optionally, the establishment of the three-modal contrastive learning loss function includes the following steps:

[0056] Step S41: Construct intra-modal components based on the text features, radar features, and image features:

[0057]

[0058] in, Representing modes The Middle Anchor features of each sample and These represent the positive samples corresponding to the anchor point features. Corresponding mode With negative samples Corresponding mode , This represents a modal index, corresponding to cloud images, cloud description text, and cloud-related metrics, respectively. ,but Image features representing sample i ,like ,but Text features representing sample i ,like ,but Radar features representing sample i , For temperature parameters, Indicates the similarity between features;

[0059] Step S42, construct intermodal components:

[0060]

[0061] in, This indicates two different modal indices;

[0062] Step S43: Construct a three-modal contrastive loss function based on the intra-modal components and inter-modal components:

[0063]

[0064] in, These are the weighting parameters used to balance the intramodal and intermodal components.

[0065] The beneficial effects of this invention are as follows: This invention generates high-level semantic cloud description text by introducing a large language model, using text information as a new semantic modality, and introducing high-level semantic information to establish a correlation between the original cloud observation data and human knowledge cognition, thereby improving semantic understanding ability; it jointly models cloud description text, cloud images, and millimeter-wave cloud radar observation data, constructs an overall architecture based on a visual language model, and designs a cloud multimodal adapter to achieve cross-modal adaptive fusion during visual encoding, enhancing information interaction and semantic alignment between different modalities, while also transferring knowledge embedded in the visual language model and fine-tuning the visual language model to better adapt to cloud classification tasks; at the same time, it constructs a three-modal contrastive learning mechanism to strengthen intramodal discrimination ability and intermodal consistency, thereby improving the representation ability of fused features and enhancing the accuracy, stability, and robustness of cloud classification. Attached Figure Description

[0066] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0067] Figure 1 This is a flowchart of a multimodal cloud classification method based on a visual language model according to an embodiment of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0069] Figure 1 This is a flowchart of a multimodal cloud classification method based on a visual language model according to an embodiment of the present invention. The following is an example... Figure 1 To illustrate some specific implementation processes of the present invention, we will use examples, such as... Figure 1 As shown, the multimodal cloud classification method based on a visual language model includes the following steps:

[0070] Step S1: Construct a text generation module. In this module, a large language model is used to generate corresponding cloud description text based on cloud images, which includes preset cloud features such as cloud color, whether the cloud boundaries are clear, and cloud shape. That is, each cloud image corresponds to a cloud description text, wherein the cloud images include training cloud images and test cloud images, and the cloud description text... Length not exceeding The large language model can be GPT;

[0071] In this invention, considering that the text encoder in the visual language model used below has an upper limit constraint on the length of the input sequence, and that excessively long texts are prone to introducing redundant semantic information and increasing computational complexity, in one embodiment of the invention, the maximum number of words in the generated cloud description text is limited. To impose restrictions, such as setting... This ensures that the generated cloud description text covers key semantic features while improving coding efficiency.

[0072] Step S2: Construct a cloud multimodal feature extraction and fusion module based on a visual language model. In this module, a text encoder and a radar encoder are constructed to encode the cloud description text corresponding to the training cloud image and the millimeter-wave cloud radar observation data corresponding to the training cloud image, respectively, to obtain text features and radar features. An image encoder is also constructed to encode the training cloud image and adaptively fuse it with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features with enhanced representation capabilities, thereby enabling information interaction between different modalities.

[0073] In one embodiment of the present invention, the visual language model may be CLIP. The text encoder in the cloud multimodal feature extraction and fusion module adopts the text encoder in the visual language model, and the image encoder in the cloud multimodal feature extraction and fusion module adopts the image encoder in the visual language model. A radar encoder is added to the cloud multimodal feature extraction and fusion module to achieve feature space alignment and cross-modal information interaction. The radar encoder has the same structure as the image encoder in the visual language model. The text encoder and image encoder are initialized based on pre-trained weights, and the original backbone parameters are frozen during fine-tuning. Only the introduced bottleneck adapter and cloud multimodal adapter parameters, and other custom trainable parameters, are trained to achieve efficient transfer and adaptation to multimodal cloud classification tasks while retaining the original cross-modal semantic alignment capabilities. The introduced bottleneck adapter and cloud multimodal adapter will be described in detail below.

[0074] To extract high-level semantic information about clouds, in one embodiment of the present invention, textual features of the cloud description text are extracted. Further, step S2, which involves constructing a text encoder to encode the features of the cloud description text corresponding to the training cloud image to obtain the textual features, includes the following steps:

[0075] Step S211, construct a text encoder, wherein the text encoder includes Layer Transformer module;

[0076] Step S212: Introduce a bottleneck adapter after each Transformer module of the text encoder and initialize the parameters of the constructed text encoder.

[0077] Step S213: The cloud description text is converted into a word identifier sequence through byte pair encoding, then mapped through the text embedding layer in the visual language model and superimposed with the first preset position encoding to obtain the initial text features. ;

[0078] Specifically, in this step, for the cloud description text, a byte-pair encoding method is first used to convert the cloud description text into a token identifier sequence. Then, the initial text features are obtained by mapping through the text embedding layer in the visual language model and superimposing the first preset position encoding. :

[0079]

[0080] in, Represents a text embedding layer. This indicates the first preset position code.

[0081] Step S214, the initial text features The text encoder is input for feature encoding, and the EOS token is extracted as a text feature. The text features It contains high-level semantic information about the cloud, thus enabling the provision of explicit semantic priors.

[0082] In one embodiment of the present invention, the text encoder processes the initial text features. Feature encoding specifically includes: The text encoder of the layer Transformer module processes the initial text features. As mentioned above, the text encoder introduces a bottleneck adapter after each Transformer module to achieve efficient parameter fine-tuning. Therefore, if we assume the text encoder's first... The output of the layer Transformer module is ,but It can be represented as:

[0083]

[0084] in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module, The value can be selected according to the actual application needs; for example, it can be set to 12.

[0085] The bottleneck adapter adopts a "dimensionality reduction-nonlinear mapping-dimensionality increase" structure and fuses with the original features through residual connections. Therefore, the first... Output of the bottleneck adapter after the Transformer module It can be represented as:

[0086]

[0087] in, For a dimension reduction matrix, For an upgraded matrix, The ReLU activation function is used, and the feature dimension after dimensionality reduction is smaller than the original feature dimension, while the feature dimension after dimensionality increase is equal to the original feature dimension. In one embodiment of the present invention, The dimension is 512, after dimensionality reduction matrix The dimension reduction results in a dimension of 256, which is achieved through the dimension-upgrading matrix. The dimension was restored to 512 after dimensionality increase, and the final text features were obtained after feature encoding by the text encoder. The dimension is 512.

[0088] Furthermore, the step S2, which involves constructing a radar encoder to encode the features of millimeter-wave cloud radar observation data corresponding to the training cloud image, and obtaining radar features, includes the following steps:

[0089] Step S221, construct a radar encoder, wherein the radar encoder includes Layer Transformer module;

[0090] Step S222: Introduce a bottleneck adapter after each Transformer module of the radar encoder and initialize the parameters of the constructed radar encoder.

[0091] Step S223: The millimeter-wave cloud radar observation data is converted into cloud-related indicators. The cloud-related indicators are processed by the radar embedding layer and superimposed with [CLS] token embedding and the second preset position code to obtain the initial radar features. The radar embedding layer is built on a fully connected layer, and its structure is similar to that of the text embedding layer in the visual language model.

[0092] In one embodiment of the present invention, the millimeter-wave cloud radar observation data includes three physical quantities: radar reflectivity factor, radial velocity, and radar velocity spectral width. The observation range of these three physical quantities in the height direction is 0 to 15 kilometers. To enhance the ability to express the physical structure characteristics of clouds, the millimeter-wave cloud radar observation data is first converted into a data structure containing... Individual cloud-related metrics and coverage A matrix of indicators for consecutive minutes In one embodiment of the present invention, , Then, after processing by the radar embedding layer and superimposed with the [CLS] token embedding and the second preset position code, the initial radar feature is obtained. :

[0093]

[0094] in, Indicates radar embedding layer, and These represent the learnable [CLS] token embedding and the second preset position encoding, respectively.

[0095] Step S224, the initial radar features The input to the radar encoder is used for feature encoding, and the [CLS] token is extracted and embedded as a radar feature. .

[0096] In one embodiment of the present invention, the radar encoder performs a test on the initial radar feature. Feature encoding specifically involves: a radar encoder comprising n Transformer modules encoding the initial radar features. As mentioned above, a bottleneck adapter is introduced after each Transformer module in the radar encoder to achieve efficient parameter fine-tuning. Therefore, if we assume the radar encoder's first... The output of the layer Transformer module is ,but It can be represented as:

[0097]

[0098] in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module, The value can be selected according to the actual application needs; for example, it can be set to 12.

[0099] The bottleneck adapter adopts a "dimensionality reduction-nonlinear mapping-dimensionality increase" structure and fuses with the original features through residual connections. Therefore, the first... Output of the bottleneck adapter after the Transformer module It can be represented as:

[0100]

[0101] in, For a dimension reduction matrix, For an upgraded matrix, The ReLU activation function is used, and the feature dimension after dimensionality reduction is smaller than the original feature dimension, while the feature dimension after dimensionality increase is equal to the original feature dimension. In one embodiment of the present invention, The dimension is 768, after dimensionality reduction matrix The dimension reduction results in a dimension of 256, which is achieved through the dimension-upgrading matrix. The dimension is restored to 768 after dimensionality increase, and the radar features are finally obtained after feature encoding by the radar encoder. The dimension is 768, but in order to match the text features The dimensions are consistent, the radar features It is also projected onto the shared feature space through a mapping matrix, and its dimension changes from 768 to 512.

[0102] Further, the step S2, which involves constructing an image encoder to encode features of the training cloud image and adaptively fusing them with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features, includes the following steps:

[0103] Step S231: Construct an image encoder and a cloud multimodal adapter, wherein the image encoder includes... Layer Transformer module;

[0104] Step S232: Introduce a cloud multimodal adapter after each Transformer module of the image encoder, and initialize the parameters of the constructed image encoder.

[0105] Step S233: Divide the training cloud image into several image blocks, process the image blocks through the image embedding layer in the visual language model, and superimpose them with [CLS] token embedding and third preset position encoding to obtain initial image features. ;

[0106] In one embodiment of the present invention, the cloud image is an RGB cloud image, and the original RGB cloud image size is [size missing]. Before being divided into image blocks, the images are preprocessed to reduce their size. .

[0107] In step S233, it is assumed that the training cloud image is divided into several image patches. The initial image features are obtained by processing the image block embedding layer and overlaying it with the [CLS] token embedding and the third preset position encoding. :

[0108]

[0109] in, Represents the image patch embedding layer. and These represent the learnable [CLS] token embedding and the third preset position encoding, respectively. In one embodiment of the invention, the number of image blocks is 196, and the size of the image blocks is... .

[0110] Step S234, the initial image features The image encoder is input for feature encoding, and during the encoding process, it is adaptively fused with the corresponding intermediate layer output features of the text encoder and radar encoder. After feature encoding is completed, the [CLS] token embedding is extracted as the image feature. .

[0111] In one embodiment of the present invention, the image encoder processes the initial image features. The feature encoding process, and its adaptive fusion with the corresponding intermediate layer output features of the text encoder and radar encoder, specifically includes: The image encoder of the layer Transformer module processes the initial image features. As mentioned above, layer-by-layer feature extraction is performed. Each Transformer module in the image encoder incorporates a cloud-based multimodal adapter to enable efficient parameter fine-tuning. Therefore, if we assume the image encoder's first... The output of the layer Transformer module is ,but It can be represented as:

[0112]

[0113] in, For the first The output of the stratus cloud multimodal adapter, Indicates the first Layer Transformer module, The value can be selected according to the actual application needs; for example, it can be set to 12.

[0114] To transfer embedded knowledge from visual language models and achieve full fusion of features from three modalities, this application proposes a cloud multimodal adapter for cross-modal interactive fusion. The cloud multimodal adapter adopts a "dimensionality reduction-nonlinear mapping-dimensionality increase" structure, and fuses with the original features through residual connections, while simultaneously achieving adaptive fusion with the corresponding intermediate layer output features of the text encoder and radar encoder.

[0115] With the first Taking the layered cloud multimodal adapter as an example, in the first... In the Stratocloud Multimodal Adapter:

[0116] First, for the image encoder... Output of the Layer Transformer module Using dimensionality reduction matrices Dimensionality reduction is performed to obtain .

[0117] Then, to The corresponding intermediate layer output features of the text encoder and radar encoder are adaptively fused to obtain the fused features. :

[0118]

[0119] in, , and It is a learnable parameter that can automatically determine the optimal contribution of each mode; and They are defined as follows:

[0120]

[0121]

[0122] in, This is the scaling factor, which can be set to 32. These represent the first and second parts of the text encoder and radar encoder, respectively. The output of the layer Transformer module, and As a dimensionality reduction matrix, it is used to reduce the features and Mapped to the same dimensional feature space.

[0123] Finally, regarding the fusion features After performing dimensionality upgrading, we obtain the first... Output of the Stratocloud Multimodal Adapter :

[0124]

[0125] in, For a dimension reduction matrix, For an upgraded matrix, This is the ReLU activation function.

[0126] In one embodiment of the present invention, The dimension is 768, after dimensionality reduction matrix The dimension reduction is 256, after dimension increase matrix The dimension is restored to 768 after dimensionality upscaling, and the final image features are obtained after feature encoding by the image encoder. The dimension is 768, but in order to match the text features The dimensions are consistent, the image features It is also projected onto the shared feature space through a mapping matrix, and its dimension changes from 768 to 512.

[0127] Step S3: Construct a classifier module by inputting the image features extracted by the cloud multimodal feature extraction and fusion module into the classifier module and outputting the cloud classification result of the training cloud image.

[0128] In one embodiment of the present invention, the classifier module is a multilayer perceptron structure, including a hidden layer and an output layer. The hidden layer is used for non-linear mapping of the input features. The hidden layer employs the GELU activation function and introduces a Dropout regularization mechanism after activation, with a Dropout ratio of 0.5 to reduce the risk of overfitting. Its input dimension is 512, and its output dimension is 256. The dimension of the output layer is equal to the number of cloud type categories. The output of the classifier module... It can be represented as:

[0129]

[0130]

[0131]

[0132] in, and It is a linear transformation matrix. and For bias terms, () indicates the normalization layer. Represents image features.

[0133] Furthermore, the parameters of the hidden layer and output layer are initialized using a Xavier uniform distribution, and the bias is initialized to 0 to ensure the stability of model training. Finally, the output of the classifier module... By converting the cloud type into a probability distribution using the Softmax function, the corresponding cloud classification results can be obtained.

[0134] Step S4: Construct a total loss function based on the text features, radar features, and image features, and optimize the multimodal cloud classification model composed of the cloud multimodal feature extraction and fusion module and the classifier module to obtain the final multimodal cloud classification model;

[0135] In one embodiment of the present invention, the total loss function includes a three-modal contrastive learning loss function and a cross-entropy loss function, wherein the three-modal contrastive learning loss function is established based on the text features, radar features and image features.

[0136] The trimodal contrastive learning loss function includes an intramodal component to enhance the consistency of feature representations within the same modality, and an intermodal component to narrow the feature distance between similar cloud samples in different modalities, thereby achieving joint alignment of text features, image features, and radar features.

[0137] In one embodiment of the present invention, the intramodal component is used to constrain the feature distribution structure within the cloud description text, cloud image, and millimeter-wave cloud radar observation data, respectively. By optimizing the similarity relationship between features in their respective modal feature spaces, the feature expression of cloud samples of the same category is made more concentrated, and the feature distribution of cloud samples of different categories is more separated, thereby enhancing the discriminative ability and structural consistency of each single modality. The intermodal component is used to constrain the feature correspondence between different modalities. By optimizing the relative distance relationship between text features, image features, and radar features in the shared feature space, the three modalities are kept consistent at the semantic level, thereby achieving cross-modal feature alignment and improving the stability and consistency of multimodal fusion representation.

[0138] Furthermore, the establishment of the three-modal contrastive learning loss function includes the following steps:

[0139] Step S41: Construct intra-modal components based on the text features, radar features, and image features to integrate features with the same cloud type, enhance the discriminative ability within each modality, and enable each modality to learn a more structured and consistent representation.

[0140] In one embodiment of the present invention, the intra-modal component treats features of the same class within the same modality as positive sample pairs and features of different classes as negative sample pairs to enhance the discriminative power of single-modal features. The intra-modal component can be represented as follows:

[0141]

[0142] in, Representing modes The Middle Anchor features of each sample and These represent the positive samples corresponding to the anchor point features. Corresponding mode With negative samples Corresponding mode , This represents a modal index, corresponding to cloud images, cloud description text, and cloud-related metrics, respectively. For example, if... ,but Image features representing sample i ,like ,but Text features representing sample i ,like ,but Radar features representing sample i ; Temperature is a parameter used to adjust the similarity scale; Indicates the similarity between features.

[0143] In one embodiment of the present invention, The dimensions of the three modal features are all Before calculating feature similarity, L2 normalization is performed on each modality feature to stabilize the training process and avoid the impact of feature scale differences.

[0144] Step S42: Construct intermodal components to encourage features with the same cloud type but from different modalities to approach each other, thereby achieving alignment of features between different modalities and promoting effective multimodal fusion;

[0145] In one embodiment of the present invention, the intermodal components treat features of the same class in different modalities as positive sample pairs and features of different classes as negative sample pairs to achieve cross-modal feature alignment. The intermodal components can be represented as follows:

[0146]

[0147] in, This indicates two different modal indices. Representing modes The Middle Anchor features of each sample and These represent the positive samples corresponding to the anchor point features. Corresponding mode With negative samples Corresponding mode .

[0148] In one embodiment of the present invention, The dimensions of the three modal features are all Before calculating feature similarity, L2 normalization is performed on each modality feature to stabilize the training process and avoid the impact of feature scale differences.

[0149] Step S43: Construct a three-modal contrast loss function based on the intra-modal components and inter-modal components to jointly optimize the intra-modal components and inter-modal components;

[0150] In one embodiment of the present invention, the three-modal contrast loss function can be expressed as:

[0151]

[0152] in, These are the weighting parameters used to balance the intramodal and intermodal components.

[0153] The three-modal contrastive loss function proposed in this application utilizes both intramodal and intermodal relationships, which improves intramodal consistency and promotes cross-modal alignment, thereby enhancing the performance of the model.

[0154] In one embodiment of the present invention, When the model reaches its highest classification accuracy, it indicates that intramodal consistency and cross-modal alignment are equally important. Overemphasizing either one may adversely affect the model's ability to effectively learn information from the three modalities.

[0155] Step S5: Input the test cloud image, the cloud description text corresponding to the test cloud image, and the millimeter-wave cloud radar observation data into the multimodal cloud classification model to obtain the cloud classification result of the test cloud image.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal cloud classification method based on a visual language model, characterized in that, The method includes the following steps: Step S1: Construct a text generation module. In this module, a large language model is used to generate corresponding cloud description text containing preset cloud features based on the cloud image. The cloud images include training cloud images and test cloud images; Step S2: Construct a cloud multimodal feature extraction and fusion module based on a visual language model. In the cloud multimodal feature extraction and fusion module, a text encoder and a radar encoder are constructed to encode the cloud description text corresponding to the training cloud image and the millimeter-wave cloud radar observation data corresponding to the training cloud image, respectively, to obtain text features and radar features. An image encoder is constructed to encode the features of the training cloud image and adaptively fuse it with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features. Step S3: Construct a classifier module by inputting the image features into the classifier module and outputting the cloud classification result of the training cloud image; Step S4: Construct a total loss function based on the text features, radar features, and image features, and optimize the multimodal cloud classification model composed of the cloud multimodal feature extraction and fusion module and the classifier module to obtain the final multimodal cloud classification model; Step S5: Input the test cloud image, the cloud description text corresponding to the test cloud image, and the millimeter-wave cloud radar observation data into the multimodal cloud classification model to obtain the cloud classification result of the test cloud image.

2. The method according to claim 1, characterized in that, The radar encoder in the cloud multimodal feature extraction and fusion module has the same structure as the image encoder in the visual language model.

3. The method according to claim 1, characterized in that, The step S2, which involves constructing a text encoder to encode the features of the cloud description text corresponding to the training cloud images and obtain the text features, includes the following steps: Step S211, construct a text encoder, wherein the text encoder includes Layer Transformer module; Step S212: Introduce a bottleneck adapter after each Transformer module of the text encoder and initialize the parameters of the constructed text encoder. Step S213: The cloud description text is converted into a word identifier sequence through byte pair encoding, then mapped through the text embedding layer in the visual language model and superimposed with the first preset position encoding to obtain the initial text features. ; Step S214, the initial text features The text encoder is input for feature encoding, and the EOS token is extracted as a text feature. .

4. The method according to claim 3, characterized in that, In step S214, the text encoder... The output of the layer Transformer module is : in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module; No. Output of the bottleneck adapter after the Transformer module Represented as: in, For a dimension reduction matrix, For an upgraded matrix, This is the ReLU activation function.

5. The method according to claim 2, characterized in that, The step S2, which involves constructing a radar encoder to encode the features of millimeter-wave cloud radar observation data corresponding to the training cloud image, and obtaining radar features, includes the following steps: Step S221, construct a radar encoder, wherein the radar encoder includes Layer Transformer module; Step S222: Introduce a bottleneck adapter after each Transformer module of the radar encoder and initialize the parameters of the constructed radar encoder. Step S223: The millimeter-wave cloud radar observation data is converted into cloud-related indicators. The cloud-related indicators are processed by the radar embedding layer and superimposed with [CLS] token embedding and the second preset position code to obtain the initial radar features. The radar embedding layer is built on a fully connected layer, and its structure is similar to that of the text embedding layer in the visual language model. Step S224, the initial radar features The input to the radar encoder is used for feature encoding, and the [CLS] token is extracted and embedded as a radar feature. .

6. The method according to claim 5, characterized in that, In step S224, the radar encoder... The output of the layer Transformer module is : in, For the first The output of the layer bottleneck adapter, Indicates the first Layer Transformer module; No. Output of the bottleneck adapter after the Transformer module Represented as: in, For a dimension reduction matrix, For an upgraded matrix, This is the ReLU activation function.

7. The method according to claim 2, characterized in that, The step S2, which involves constructing an image encoder to encode features of the training cloud image and adaptively fusing them with the corresponding intermediate layer output features of the text encoder and radar encoder to obtain image features, includes the following steps: Step S231: Construct an image encoder and a cloud multimodal adapter, wherein the image encoder includes... Layer Transformer module; Step S232: Introduce a cloud multimodal adapter after each Transformer module of the image encoder, and initialize the parameters of the constructed image encoder. Step S233: Divide the training cloud image into several image blocks, process the image blocks through the image embedding layer in the visual language model, and superimpose them with [CLS] token embedding and third preset position encoding to obtain initial image features. ; Step S234, the initial image features The image encoder is input for feature encoding, and during the encoding process, it is adaptively fused with the corresponding intermediate layer output features of the text encoder and radar encoder. After feature encoding is completed, the [CLS] token embedding is extracted as the image feature. .

8. The method according to claim 7, characterized in that, In step S234, the image encoder... The output of the layer Transformer module is : in, For the first The output of the stratus cloud multimodal adapter, Indicates the first Layer Transformer module; No. The output of the cloud multimodal adapter after the Transformer module Represented as: in, For a dimension reduction matrix, For an upgraded matrix, It is the ReLU activation function. , and It is a learnable parameter that can automatically determine the optimal contribution of each mode. and They are defined as follows: in, Scaling factor These represent the first and second parts of the text encoder and radar encoder, respectively. The output of the layer Transformer module, and This is a dimension-reduced matrix.

9. The method according to claim 1, characterized in that, The total loss function in step S4 includes a three-modal contrastive learning loss function and a cross-entropy loss function, wherein the three-modal contrastive learning loss function is established based on the text features, radar features, and image features.

10. The method according to claim 9, characterized in that, The establishment of the three-modal contrastive learning loss function includes the following steps: Step S41: Construct intra-modal components based on the text features, radar features, and image features: in, Representing modes The Middle Anchor features of each sample and These represent the positive samples corresponding to the anchor point features. Corresponding mode With negative samples Corresponding mode , This represents a modal index, corresponding to cloud images, cloud description text, and cloud-related metrics, respectively. ,but Image features representing sample i ,like ,but Text features representing sample i ,like ,but Radar features representing sample i , For temperature parameters, Indicates the similarity between features; Step S42, construct intermodal components: in, This indicates two different modal indices; Step S43: Construct a three-modal contrastive loss function based on the intra-modal components and inter-modal components: in, These are the weighting parameters used to balance the intramodal and intermodal components.

Images