Weather classification method and system based on cross-modal information

Abstract

The invention discloses a weather classification method and system based on cross-modal information, and belongs to the field of computer vision. Comprising the following steps of analyzing the defects of an existing weather classification method, establishing association of two types of data based on a randomization algorithm by collecting a weather classification image data set and a text data set, and constructing a cross-modal weather classification data set; the method comprises the following steps: constructing a weather classification model ArossMod-WC based on lightweight and efficient network models such as a CoordGate basic network structure, multi-scale linear attention, a UnetTSF structure and the like; for the designed model, a cross-modal loss function based on cross entropy loss is constructed, and an initialized cross-modal weather classification data set is utilized and training work of ArossMod-WC is completed, so that a computer system can classify weather types in combination with text information and image information, the accuracy of weather classification is improved, the application range of weather classification is widened, and the method has the advantages of being high in practicability and the like. And meanwhile, the robustness of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a weather classification method and system based on cross-modal information. Background Art

[0002] Weather classification is a classic problem in the field of computer vision. The goal of this problem is to find the mapping relationship between input information and weather categories. The visual system extracts and classifies features of a given image or text data to calculate the weather type corresponding to the data. With the popularization of artificial intelligence in various industries, weather classification problems are widely used in target detection, drones, autonomous driving and other fields. At the same time, weather classification problems are often published in top journals and conferences in the fields of computer vision and artificial intelligence, such as CVPR, TPAMI, IJCV, etc., which shows that weather classification problems have important research significance both in practical applications and theoretical research.

[0003] Existing weather classification methods can be divided into two categories according to the different processing objects: weather classification methods based on text information and weather classification methods based on outdoor images. Wang Pikun and Wu Ling realized in the article "Image-Based Self-attentive Multi-label Weather Classification Network[C].Proceedings ofInternational Conference on Image,Vision and Intelligent Systems,2022:497-504." that in practical applications, multiple weather conditions often coexist, so multi-label classification networks are more suitable for weather classification tasks than traditional single-label convolutional neural networks. Therefore, an image-based self-attention multi-label weather classification network is proposed, which combines convolutional neural networks (CNN) and Transformer structures to achieve multi-label recognition of weather images. Mohamad Safia and Rodi Abbas proposed a weather classification method based on supervised learning using the Weatherstack dataset in the article "Classification of Weather Conditions Based on Supervised Learning for Swedish Cities[J]. Atmosphere, 2023, 14(7): 477-496." This method constructs an artificial neural network to mine text data and classify the weather conditions of Swedish cities. Rudresh Pillai and Neha Sharma used Fine-Tuned EfficientNetB4 and Adam optimizer to construct and train a deep learning model in the article "Fine-Tuned EfficientNetB4 Transfer Learning Model for Weather Classification[C]. 2023 3rd Asian Conference on Innovation in Technology, 2023: 1-6." They used 6862 pairs of images with labeled information as a dataset and achieved 92% of the outdoor image weather classification results.Afxentiou Viktoria and Vladimirova Tanya proposed a multi-label weather classification framework (MLWC) for embedded computing platforms in the article "Multi-Label Weather Image Classification for Embedded Computing Platforms[C].9th IEEE Smart World Congress, 2023:190-197." The framework consists of five convolutional neural network (CNN) models, each of which is trained for weather conditions such as sunny, cloudy, rainy, foggy and snowy, thereby distinguishing multiple weather conditions in outdoor images.

[0004] However, the existing weather classification methods based on image information mainly utilize spatial information and channel information but cannot utilize contextual information. Meanwhile, the existing weather classification methods based on text information utilize contextual information but cannot utilize spatial information and channel information. This results in that the accuracy, universality and scope of application of weather classification methods based on both image information and text information need to be improved.

[0005] Therefore, it is necessary to provide an improved technical solution to address the above-mentioned deficiencies in the prior art. Summary of the invention

[0006] The purpose of the present invention is to provide a weather classification method and system based on cross-modal information to solve the shortcomings of existing weather classification methods in terms of accuracy, scope of application, and robustness. Specifically, the present invention studies how to fully mine the spatial and channel information contained in the image and the contextual semantic information contained in the text based on image-text cross-modal information, and use the above information in combination, thereby realizing the joint application of the two existing methods, thereby improving the problems existing in the existing weather classification methods.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] In a first aspect, the present application provides a weather classification method based on cross-modal information, comprising:

[0009] Step S101: Initialize a cross-modal data set: the cross-modal data set includes a weather classification image data set and a text data set; associate the above two types of data sets using a randomization algorithm to construct a cross-modal weather classification data set; divide the constructed cross-modal weather classification data set into a training data set, a verification data set, and a test data set;

[0010] Step S102: construct a weather classification model (abbreviated as AcrossMod-WC): analyze the optical properties and imaging characteristics of various images in the weather classification image dataset, and build an image feature extraction module based on multi-scale linear attention and CoordGate basic network structure; build a text information feature extraction module based on the UnetTSF structure according to the contextual semantic information and time series information of the text information in the cross-modal weather classification dataset; construct a cross-modal feature fusion module based on deep learning for the acquired image features and text features; integrate the image feature extraction module, the text feature extraction module and the cross-modal feature fusion module to construct a weather classification model;

[0011] Step S103: construct a cross-modal loss function based on cross entropy loss: in combination with the training data set, the constructed loss function is used to complete the training of the weather classification model, and the trained weather classification model can be used to achieve the cross-modal weather classification task.

[0012] In combination with the first aspect, in some possible implementations, step S101: initializing a cross-modal dataset, specifically:

[0013] Step S111: selecting four types of weather including sunny, rainy, snowy and foggy in the weather classification image data set as the weather classification targets corresponding to the weather classification task;

[0014] Step S121: using a randomization algorithm to determine a weather image from the weather classification image dataset and a text message from the text dataset, forming the weather image and the text message into a pair of associated data, combining all the associated data, and constructing a cross-modal weather classification dataset;

[0015] Step S131: Divide the constructed cross-modal weather classification data into a training data set, a validation data set, and a test data set.

[0016] In conjunction with the first aspect, in some possible implementations, the randomization algorithm is expressed as:

[0017]

[0018] Where i_n is the number of data corresponding to the i-th weather category in the weather classification image dataset, mod(·) is the remainder operation, and d i_n-1 is the seed of the random sequence, b i_n-1 and c i_n-1 is a constant of the random sequence, a i_n is a random number.

[0019] In combination with the first aspect, in a possible implementation, in step S102, the image feature extraction module is constructed according to the following steps:

[0020] Step S112: construct an image feature extraction unit based on a sequentially connected CoordGate basic network structure, multi-scale linear attention, and a 3×3 convolution;

[0021] Step S122: superimpose the four constructed image feature extraction units to construct the image feature extraction module.

[0022] In combination with the first aspect, in a possible implementation, the working process of the image feature extraction module is:

[0023]

[0024] Where I is the image to be classified, Conv1(·) is a 1×1 convolution, Conv3(·) is a 3×3 convolution, Conv5(·) is a 5×5 convolution, CNN(·) is a convolutional neural network function, MLP(·) is a multi-layer perceptron function, Lin(·) is a linear layer function, ReLU(·) is ReLU linear attention, is the channel fusion function, ⊙ is the matrix multiplication function, Ifeature i It is the feature output by the i-th image feature extraction unit, and Ifeature is the final extracted feature.

[0025] In combination with the first aspect, in some possible implementations, in step S102, the text information feature extraction module is constructed according to the following steps:

[0026] Step S132: constructing a text feature extraction unit with the UnetTSF structure;

[0027] Step S142: superimpose the five constructed text feature extraction units to construct the text feature extraction module.

[0028] In combination with the first aspect, in some possible implementations, the working process of the text feature extraction module is as follows:

[0029]

[0030] Among them, T is the text data to be classified, Avg(·) is the average pooling layer, and Tfeature i is the feature output by the i-th text feature extraction unit, and Tfeature is the final extracted feature.

[0031] In combination with the first aspect, in some possible implementations, in step S102, the cross-modal feature fusion module is constructed according to the following steps:

[0032] Step S152: Fusing the multi-layer perceptron, channel fusion, 3×3 convolution, and multi-scale linear attention;

[0033] Step S162: input the image features and text features into the cross-modal feature fusion module respectively, and the module output is the fused features.

[0034] In combination with the first aspect, in some possible implementations, the working process of the cross-modal feature fusion module is as follows:

[0035]

[0036] Among them, feature_fusion is the output of the cross-modal feature fusion module, that is, the cross-modal feature.

[0037] In a second aspect, the present invention provides a weather classification method system based on cross-modal information, comprising:

[0038] Initialize the cross-modal dataset unit to randomly combine image and text data to obtain a cross-modal weather classification dataset;

[0039] The weather classification model operation unit is used to extract and fuse the features of the input image and text to obtain the weather type based on cross-modal information;

[0040] The loss function calculation unit is used to count and analyze the weather types obtained to obtain the final model parameters and weather classification results.

[0041] The advantages of the present invention over the prior art are:

[0042] (1) Innovatively randomly combine image data and text data to construct a cross-modal dataset.

[0043] (2) Integrate the CoordGate basic network structure, multi-scale linear attention, UnetTSF structure, and cross-modal feature fusion.

[0044] (3) Construct a new weather classification model AcrossMod-WC to effectively handle weather classification problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic flowchart of a weather classification method based on cross-modal information according to some embodiments of the present application.

[0046] Figure 2 A schematic diagram of the structure of an image feature extraction module provided according to some embodiments of the present application.

[0047] Figure 3 A schematic diagram of the structure of a text feature extraction module provided according to some embodiments of the present application.

[0048] Figure 4 A schematic diagram of the structure of a cross-modal feature fusion module provided according to some embodiments of the present application.

[0049] Figure 5 A schematic diagram of the AcrossMod-WC structure provided according to some embodiments of the present application.

[0050] Figure 6 A schematic diagram of the structure of a weather classification system based on cross-modal information according to some embodiments of the present application. DETAILED DESCRIPTION

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0052] Embodiment 1:

[0053] This embodiment provides a weather classification method based on cross-modal information, such as Figure 1 As shown, the method includes: Steps S101 to S103, which are specifically as follows:

[0054] Step S101: Initialize a cross-modal dataset: The cross-modal dataset includes a weather classification image dataset and a text dataset; use a randomization algorithm to associate the above two types of datasets to construct a cross-modal weather classification dataset; divide the constructed cross-modal weather classification data into a training dataset, a verification dataset, and a test dataset.

[0055] In this embodiment, the weather classification image dataset refers to a series of image collections classified and marked according to different weather conditions (such as sunny, cloudy, rainy, snowy, etc.), which are used to train the weather classification model constructed in this embodiment to realize automatic recognition of weather conditions in images. The text dataset refers to a weather classification text dataset, which is usually composed of a large number of texts with weather type labels. These texts may come from various channels such as observation records of weather stations, news reports, and social media posts. Therefore, this type of data has the characteristics of strong timeliness and can reflect the latest weather conditions. Each sample in the text dataset usually includes two parts: one part is the text describing the weather (such as "Today is sunny and the temperature is moderate"), and the other part is the corresponding weather type label (such as "sunny").

[0056] It should be noted that the existing weather classification methods can only be implemented for single modes such as images or text information. Among them, the weather classification method based on image information mainly uses spatial information and channel information and cannot use context information. At the same time, the existing weather classification method based on text information uses context information and cannot use spatial information and channel information. This leads to the fact that both the weather classification methods based on image information and text information need to be improved in accuracy, universality and scope of application. Therefore, this embodiment designs a cross-modal weather classification method based on image and text information. For the cross-modal weather classification method, initializing the cross-modal data set is the first task to implement the weather classification method.

[0057] In some optional embodiments, in step S101, the steps of initializing the cross-modal dataset are as follows:

[0058] Step S111: selecting four types of weather including sunny, rainy, snowy and foggy in the weather classification image data set as the weather classification targets corresponding to the weather classification task;

[0059] Step S121: using a randomization algorithm to determine a weather image from the weather classification image dataset and a text message from the text dataset, forming the weather image and the text message into a pair of associated data, combining all the associated data, and constructing a cross-modal weather classification dataset;

[0060] Step S131: Divide the constructed cross-modal weather classification data into a training data set, a validation data set, and a test data set.

[0061] As an example, the above steps can be specifically described as follows: First, obtain the Multi-class WeatherImage Dataset and dataset1 in the article "Classification of Weather Conditions Based on Supervised Learning for Swedish Cities", and use the above two datasets as the weather classification dataset and text dataset respectively; then, select the Multi-class Weather Image Dataset image dataset containing four types of weather, such as sunny, rainy, snowy, and foggy, as the target of weather distinction for this weather classification task; thirdly, use the randomization algorithm to associate the above weather classification image data and text dataset, that is, use the randomization algorithm to determine a weather image from the weather classification image dataset, and determine a text information from the text dataset, form the weather image and the text information into a pair of associated data, and combine all the associated data to construct a cross-modal weather classification dataset.

[0062] Exemplarily, the expression of the randomization algorithm is:

[0063]

[0064] Among them, i_n is the number of data corresponding to the i-th type of weather in the weather classification image dataset, mod(·) is the remainder operation, and d i_n-1 is the seed of the random sequence, b i_n-1 and c i_n-1 is a constant of the random sequence, a i_n is a random number.

[0065] The method of obtaining random numbers for text data is consistent with that for image data, and will not be described in detail. Since the number of images and text data corresponding to each type of weather is not necessarily the same, the data corresponding to the random numbers of the image and text data generated each time according to the above formula are combined to form paired cross-modal data.

[0066] Finally, 60% of the constructed cross-modal weather classification data is divided into training data, 20% as validation data set, and 20% as test data set.

[0067] It is understandable that the training data, the validation data set, and the test data set may be divided in other ratios, such as 7:2:1. The division ratio of each data set in this embodiment is not limited.

[0068] Step S102: Construct a weather classification model AcrossMod-WC: Analyze the optical properties and imaging characteristics of various images in the weather classification image dataset, and build an image feature extraction module based on multi-scale linear attention and CoordGate basic network structure; According to the contextual semantic information and timing information of the text information in the cross-modal weather classification dataset, build a text information feature extraction module based on the UnetTSF structure; For the acquired image features and text features, construct a cross-modal feature fusion module based on deep learning; Integrate the image feature extraction module, text feature extraction module and cross-modal feature fusion module to construct a weather classification model.

[0069] The cross-modal information weather classification method designed by the present invention aims to establish a mapping relationship between images, texts and weather types. Since two types of information, image and text, need to be processed, the amount of data is greater than the existing weather classification method, and the type is more complex. In view of this, the present invention constructs a weather classification model AcrossMod-WC based on lightweight and efficient network models such as CoordGate basic network structure, multi-scale linear attention, and UnetTSF structure. The constructed AcrossMod-WC is divided into four parts, namely, image feature extraction module, text feature extraction module, cross-modal feature fusion module and classifier. Among them, the classifier structure is relatively simple, so the structure and working principle of the image feature extraction module, text feature extraction module and cross-modal feature fusion module will be introduced in detail.

[0070] The following is a detailed description of how the image feature extraction module works.

[0071] Since two different modal data need to be processed, a simple and efficient feature extraction module needs to be selected in the basic feature extraction unit. At the same time, the CoordGate basic network structure has the characteristics of low parameter amount and high efficiency, and has achieved good results in the fields of image segmentation and image deblurring. Therefore, the CoordGate basic network structure is selected as the image feature extraction structure. In addition, the image data in the constructed cross-modal weather classification dataset has multi-scale characteristics, so the multi-scale linear attention is combined with the CoordGate basic network structure to extract image features. Therefore, in some optional embodiments, in step S102, the image feature extraction module is constructed as follows:

[0072] Step S112: construct an image feature extraction unit based on a sequentially connected CoordGate basic network structure, multi-scale linear attention, and a 3×3 convolution;

[0073] Step S122: superimpose the four constructed image feature extraction units to construct an image feature extraction module.

[0074] Figure 2 This is a schematic diagram of the structure of the image feature extraction module provided in this embodiment. Figure 2As shown in the figure, the four image feature extraction units are all constructed based on the CoordGate basic network structure, multi-scale linear attention and a 3×3 convolution, and the CoordGate, multi-scale linear attention and 3×3 convolution are connected in sequence. Among them, the CoordGate basic network further includes: a convolutional neural network CNN and a multi-layer perceptron MLP, which are set in parallel to process the input data (including the input image to be classified or the input feature map) in parallel, and perform matrix multiplication on the processing results, and then provide them to the multi-scale linear attention in the same image feature extraction unit. The multi-scale linear attention includes a linear layer at the front end, three parallel linear attentions of different scales (i.e., different convolution kernel sizes), a channel fusion and a linear layer at the back end. Among them, the three parallel linear attentions of different scales further include upper, middle and lower branches. The upper branch is a ReLU linear attention, the middle branch includes a 3×3 convolution, a 1×1 convolution, a ReLU linear attention, and the lower branch includes a 5×5 convolution, a 1×1 convolution and a ReLU linear attention. The outputs of the upper, middle and lower branches are channel-fused and then processed by the linear layer to obtain the output results of multi-scale linear attention.

[0075] Based on the above image feature extraction module structure, the process of using the image feature extraction module to extract features from an image is as follows:

[0076] First, the image to be classified is input into the first image feature extraction unit;

[0077] Then, the output of the first image feature extraction unit is input to the next image feature extraction unit; finally, the feature output by the fourth image feature extraction unit is the extracted weather classification image feature; the image feature extraction process can be expressed as:

[0078]

[0079] Where I is the image to be classified, Conv1(·) is a 1×1 convolution, Conv3(·) is a 3×3 convolution, Conv5(·) is a 5×5 convolution, CNN(·) is a convolutional neural network function, MLP(·) is a multi-layer perceptron function, Lin(·) is a linear layer function, ReLU(·) is ReLU linear attention, is the channel fusion function, ⊙ is the matrix multiplication function, Ifeature i It is the feature output by the i-th image feature extraction unit, and Ifeature is the final extracted feature.

[0080] The above is an introduction to the image feature extraction module. The following is a detailed explanation of the working principle of the text feature extraction module.

[0081] Text information has strong context relevance, and in order to ensure the efficiency of the constructed method, in this embodiment, a text information feature extraction module is built based on the UnetTSF structure.

[0082] Specifically, the text information feature extraction module is constructed according to the following steps:

[0083] Step S132: constructing a text feature extraction unit with the UnetTSF structure;

[0084] Step S142: superimpose the five constructed text feature extraction units to construct a text feature extraction module. Figure 3 shown.

[0085] Based on the structure of the above text feature extraction module, the working process of this module is as follows:

[0086] First, the text to be classified is input into the first text feature extraction unit; then, the output of the first text feature extraction unit is input into the next text feature extraction unit; finally, the features output by the fifth text feature extraction unit are the extracted weather classification text features.

[0087] Furthermore, the text feature extraction process can be expressed as:

[0088]

[0089] Among them, T is the text data to be classified, Avg(·) is the average pooling layer, and Tfeature i is the feature output by the i-th text feature extraction unit, and Tfeature is the final extracted feature.

[0090] The above is an introduction to the text feature extraction module. The following is a detailed explanation of the working principle of the cross-modal feature fusion module.

[0091] After obtaining the image features and text features, the traditional feature fusion method is not applicable because the image features and text features belong to different modalities. Inspired by the idea of ​​Transformer in dealing with cross-modal problems, this embodiment designs the following cross-modal feature fusion solution: first, the multi-layer perceptron, channel fusion, 3×3 convolution, and multi-scale linear attention are fused; secondly, the image feature Ifeature and the text feature Tfeature are input into the cross-modal feature fusion module respectively, and the module output is the fused feature.

[0092] Figure 4 This is a schematic diagram of the structure of the cross-modal feature fusion module provided in this embodiment. Figure 4As shown in the figure, the cross-modal feature fusion module includes two parallel multi-layer perceptrons MLP, which are used to receive image features Ifeature and text features Tfeature respectively. The image features Ifeature and text features Tfeature are processed by channel fusion after passing through MLP, and then the results of channel fusion processing are sequentially subjected to MLP, multi-scale linear attention, and finally 3×3 convolution to obtain the output of the cross-modal feature fusion module, namely, the cross-modal feature.

[0093] Furthermore, based on the structure of the above cross-modal feature fusion module, the working process of the cross-modal feature fusion module is as follows: first, the image feature Ifeature and the text feature Tfeature are input into different multi-layer perceptrons; second, the output results of the two multi-layer perceptrons are integrated using the channel fusion function; finally, the results of the channel fusion function integration are input into the multi-layer perceptron, multi-scale linear attention and 3×3 convolution in turn, and the output result of the 3×3 convolution is the final cross-modal feature. The above working process can be expressed as:

[0094]

[0095] Among them, feature_fusion is the output of the cross-modal feature fusion module, that is, the cross-modal feature; Ifeature is the feature finally extracted by the image feature extraction module; Tfeature is the feature finally extracted by the text feature extraction module; Conv1(·) is a 1×1 convolution, Conv3(·) is a 3×3 convolution, and Conv5(·) is a 5×5 convolution; MLP(·) is a multi-layer perceptron function, Lin(·) is a linear layer function, is the channel fusion function.

[0096] After constructing the image feature extraction module, text feature extraction module, and cross-modal feature fusion module, the above modules are then fused, and based on the Softmax classifier, the weather classification model AcrossMod-WC is constructed. The specific method is as follows: First, the image feature extraction module and the text feature extraction module are connected in parallel; second, the cross-modal feature fusion module is placed after the image feature extraction module and the text feature extraction module; finally, the Softmax classifier is placed after the cross-modal feature fusion module to complete the construction of AcrossMod-WC. The constructed AcrossMod-WC model structure is as follows: Figure 5 shown.

[0097] like Figure 5As shown in the figure, the workflow of the constructed AcrossMod-WC model is as follows: first, the image and text are input into the image feature extraction module and the text feature extraction module in the AcrossMod-WC model in parallel to obtain image features and text features; secondly, the image features and text are input into the cross-modal feature fusion module to obtain cross-modal features; finally, the cross-modal features are input into the Softmax classifier to obtain the weather classification results.

[0098] Step S103: construct a cross-modal loss function based on cross entropy loss: combine the training data set, use the constructed loss function to complete the training of the weather classification model, and use the trained weather classification model to achieve the cross-modal weather classification task.

[0099] After the AcrossMod-WC model is constructed in step S102, in order to obtain the mapping relationship between images, texts and weather types, the AcrossMod-WC model needs to be trained using the initialization data set in step S103. Based on the idea of ​​machine learning, it is necessary to establish a loss function between the image and text information in the data set and the ground truth to achieve the training of AcrossMod-WC.

[0100] The constructed cross-modal loss function expression is:

[0101]

[0102] Where L is the loss function, M is the number of training samples, i is the index of the training sample, and S i is the real weather classification result (ie, true value, label value), It is the classification result given by the AcrossMod-WC model.

[0103] In summary, the weather classification method based on cross-modal information provided in this embodiment initializes a cross-modal data set, constructs a weather classification model, constructs a cross-modal loss function based on cross entropy loss and trains the model, and applies the trained weather classification model to the weather classification of real scenes to achieve cross-modal weather classification tasks. In the above steps, the cross-modal data set is obtained by randomly associating weather classification image data and text data. Fusion of the features of the two types of data can provide a richer feature expression for the weather classification model. In the weather classification model, combined with multi-scale linear attention and CoordGate basic network structure, the image feature extraction module can simultaneously extract rich image features from multiple scales and spatial positions, enhancing feature expression capabilities and robustness. Building a text information feature extraction module based on the UnetTSF structure can gradually extract the deep semantic information of the text from local to global, thereby capturing text information at different scales. The visual features in the weather classification image are combined with the text description information provided by the text information through the cross-modal feature fusion module to achieve the complementarity of multi-modal information, reduce the uncertainty of single modal information, and improve the accuracy of classification.

[0104] Embodiment 2:

[0105] This embodiment provides a weather classification system based on cross-modal information, such as Figure 6 As shown, the system includes: a cross-modal data set initialization unit 601, a weather classification model operation unit 602 and a loss function calculation unit 603. Among them:

[0106] Initializing a cross-modal data set unit 601, for randomly combining image and text data to obtain a cross-modal weather classification data set;

[0107] The weather classification model operation unit 602 is used to extract and fuse features of input images and texts to obtain weather types based on cross-modal information;

[0108] The loss function calculation unit 603 is used to perform statistics and analysis on the obtained weather types to obtain the final model parameters and weather classification results.

[0109] The weather classification system based on cross-modal information provided in this embodiment can implement the steps and processes of the weather classification method based on cross-modal information provided in any of the above embodiments, and achieve the same technical effects, which will not be described one by one here.

[0110] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A weather classification method based on cross-modal information, characterized in that: The steps include: Step S101: Initializing a cross-modal dataset: the cross-modal dataset includes a weather classification image dataset and a text dataset; the two datasets are associated using a randomization algorithm to construct a cross-modal weather classification dataset; Divide the constructed cross-modal weather classification data into a training data set, a validation data set, and a test data set; Step S102: constructing a weather classification model: analyzing the optical properties and imaging characteristics of various types of images in the weather classification image dataset, and building an image feature extraction module based on multi-scale linear attention and CoordGate basic network structure; According to the contextual semantic information and time series information of the text information in the cross-modal weather classification dataset, a text information feature extraction module is built based on the UnetTSF structure; for the acquired image features and text features, a cross-modal feature fusion module is constructed based on deep learning; the image feature extraction module, the text feature extraction module and the cross-modal feature fusion module are integrated to construct a weather classification model; Step S103: construct a cross-modal loss function based on cross entropy loss: in combination with the training data set, the constructed loss function is used to complete the training of the weather classification model, and the trained weather classification model can be used to achieve the cross-modal weather classification task.

2. The weather classification method based on cross-modal information according to claim 1, characterized in that: Step S101: Initialize the cross-modal dataset, specifically: Step S111: selecting four types of weather including sunny, rainy, snowy and foggy in the weather classification image data set as the weather classification targets corresponding to the weather classification task; Step S121: using a randomization algorithm to determine a weather image from the weather classification image dataset and a text message from the text dataset, forming the weather image and the text message into a pair of associated data, combining all the associated data, and constructing a cross-modal weather classification dataset; Step S131: Divide the constructed cross-modal weather classification data into a training data set, a validation data set, and a test data set.

3. The weather classification method based on cross-modal information according to claim 1 or 2, characterized in that: The expression of the randomization algorithm is: Where i_n is the number of data corresponding to the i-th weather category in the weather classification image dataset, mod(·) is the remainder operation, and d i_n-1 is the seed of the random sequence, b i_n-1 and c i_n-1 is a constant of the random sequence, a i_n is a random number.

4. The weather classification method based on cross-modal information according to claim 1, characterized in that: In step S102, the image feature extraction module is constructed according to the following steps: Step S112: construct an image feature extraction unit based on a sequentially connected CoordGate basic network structure, multi-scale linear attention, and a 3×3 convolution; Step S122: superimpose the four constructed image feature extraction units to construct the image feature extraction module.

5. The weather classification method based on cross-modal information according to claim 1 or 4, characterized in that: The working process of the image feature extraction module is as follows: Where I is the image to be classified, Conv1(·) is a 1×1 convolution, Conv3(·) is a 3×3 convolution, Conv5(·) is a 5×5 convolution, CNN(·) is a convolutional neural network function, MLP(·) is a multi-layer perceptron function, Lin(·) is a linear layer function, ReLU(·) is ReLU linear attention, is the channel fusion function, ⊙ is the matrix multiplication function, Ifeature i It is the feature output by the i-th image feature extraction unit, and Ifeature is the feature finally extracted by the image feature extraction module.

6. The weather classification method based on cross-modal information according to claim 1, characterized in that: In step S102, the text information feature extraction module is constructed according to the following steps: Step S132: constructing a text feature extraction unit with the UnetTSF structure; Step S142: superimpose the five constructed text feature extraction units to construct the text feature extraction module.

7. The weather classification method based on cross-modal information according to claim 1 or 6, characterized in that: The working process of the text feature extraction module is as follows: Among them, T is the text data to be classified, Lin(·) is the linear layer function; Avg(·) is the average pooling layer, Tfeature i is the feature output by the i-th text feature extraction unit, and Tfeature is the feature finally extracted by the text feature extraction module.

8. The weather classification method based on cross-modal information according to claim 1, characterized in that: In step S102, the cross-modal feature fusion module is constructed according to the following steps: Step S152: Fusing the multi-layer perceptron, channel fusion, 3×3 convolution, and multi-scale linear attention; Step S162: input the image features and text features into the cross-modal feature fusion module respectively, and the module output is the fused features.

9. The weather classification method based on cross-modal information according to claim 1 or 8, characterized in that: The working process of the cross-modal feature fusion module is as follows: Among them, feature_fusion is the output of the cross-modal feature fusion module, that is, the cross-modal feature; Ifeature is the feature finally extracted by the image feature extraction module; Tfeature is the feature finally extracted by the text feature extraction module; Conv1(·) is a 1×1 convolution, Conv3(·) is a 3×3 convolution, and Conv5(·) is a 5×5 convolution; MLP(·) is a multi-layer perceptron function, Lin(·) is a linear layer function, is the channel fusion function.

10. A weather classification system based on cross-modal information, characterized in that: include: Initialize the cross-modal dataset unit to randomly combine image and text data to obtain a cross-modal weather classification dataset; The weather classification model operation unit is used to extract and fuse the features of the input image and text to obtain the weather type based on cross-modal information; The loss function calculation unit is used to count and analyze the weather types obtained to obtain the final model parameters and weather classification results.

Images