Weather phenomenon identification method based on improved Transform
By improving the Transformer model, combining local self-attention mechanism and offset block serialization operation, the interference and scene transformation adaptability problems of weather phenomenon recognition in outdoor image data are solved, and higher recognition accuracy and robustness are achieved , especially under the conditions of data set imbalance and background interference.
Patent Information
- Application Number
- CN202411386276.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-14
AI Technical Summary
The prior art recognizes weather phenomena in outdoor image and video data, and has poor adaptability to scene transformation, resulting in unsatisfactory recognition of weather phenomena and unsatisfactory effects.
Using the improved Transformer model, combining local self-attention mechanism (LSA) and offset block serialization operations, a weather phenomenon recognition model is constructed, image features are extracted through multiple Swin-Transformer modules and merged modules, and dynamic weight adjustment and comparison are introduced. Learn loss functions, enhance the classification ability of the model and the ability to combat confusing categories.
In scenarios where the data set is small, the distribution is unbalanced, background interference and redundant information are large, the accuracy and recognition performance of weather phenomena are improved, and the categories that are prone to confusion can be better distinguished, which enhances the robustness of the model sex.
Smart Images
Figure CN119314159A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a weather phenomenon recognition method based on an improved Transformer. Background Art
[0002] From outdoor activities to industrial production, weather has a very important impact on human activities. For example, in bad weather such as fog, rainstorm, snow, etc., visibility will be greatly reduced and the road surface will be slippery, leading to traffic jams and even traffic accidents. With the continuous development of social and economic levels and science and technology, there is an extremely urgent need for the detection and identification of various weather phenomena in outdoor environments, especially bad weather phenomena. Traditionally, the observation of weather phenomena mainly relies on manual observation, and the collection of weather phenomenon information consumes a lot of manpower and material resources. With the development of image processing technology and artificial intelligence technology, the research on automatic recognition of weather phenomena based on outdoor image and video data has received more extensive attention and has gradually developed in recent years.
[0003] However, weather phenomenon recognition based on outdoor images and video data is affected by various factors in the image data. The actual collected outdoor images and video data often contain scene light reflections and other objects unrelated to weather characteristics. These factors are interrelated and interfere with weather recognition. In addition, weather recognition itself has very high requirements for adapting to scene changes. The structure of the scene itself has a great influence on the entire recognition result. These factors have led to the current accuracy and effect of automatic weather phenomenon recognition based on outdoor images and video data being less than ideal. Summary of the invention
[0004] In response to the above problems and technical requirements, this application proposes a weather phenomenon recognition method based on an improved Transformer. The technical solution of this application is as follows:
[0005] A weather phenomenon recognition method based on an improved Transformer, the weather phenomenon recognition method comprising:
[0006] Obtain a weather image dataset, the weather image dataset including sample images taken under several different weather phenomena;
[0007] A weather phenomenon recognition model is constructed. The weather phenomenon recognition model includes a sequentially connected block module, multiple feature processing layers, a normalization layer, an average pooling layer, and a fully connected layer. The block module is used to perform an offset block serialization operation on the input image to divide the input image into multiple non-overlapping image blocks. The first feature processing layer includes a sequentially connected linear embedding module and multiple Swin-Transforme modules. Other feature processing layers include sequentially connected merging modules and multiple Swin-Transforme modules. The linear embedding module is used to perform a linear transformation on each image block. The merging module is used to achieve downsampling by merging image blocks. Multiple Swin-Transforme modules in each feature processing layer are used to perform hierarchical image feature extraction processing. Each Swin-Transforme module is constructed based on the local self-attention mechanism LSA.
[0008] The weather image dataset is used to train the constructed weather phenomenon recognition model, and the trained weather phenomenon recognition model is used to recognize weather phenomena.
[0009] Its further technical solution is that the blocking module and the merging modules in each feature processing layer are implemented by an offset blocking serialization module. Each offset blocking serialization module spatially shifts the input image in four diagonal directions and stacks it with the input image to obtain a stacked feature map. After the stacked feature map is block-serialized, patch flattening, layer normalization and linear projection are performed in sequence before output.
[0010] Its further technical solution is that each Swin-Transforme module includes a first normalization layer, a W-LSA layer, a second normalization layer, an MLP layer, a third normalization layer, a SW-LSA layer, a fourth normalization layer and an MLP layer connected in sequence, wherein the W-LSA layer is processed using a local self-attention mechanism LSA based on a fixed window, and the SW-LSA layer is processed using a local self-attention mechanism LSA based on a moving window;
[0011] The input image X of the Swin-Transforme module l-1 After passing through the first normalization layer and the W-LSA layer, the image is added to the input image X1 to obtain
[0012] image After the second normalization layer and MLP layer, the image Add to get image X l ;
[0013] ImageX l After the third normalization layer and SW-LSA layer, the image X l Add to get the image
[0014] image After the fourth normalization layer and MLP layer, the image Add together to get image X l+1 And output the Swin-Transforme module.
[0015] A further technical solution is to use the weather image data set to train the constructed weather phenomenon recognition model, including:
[0016] Using the weather image dataset according to the loss function Conduct model training on the constructed weather phenomenon recognition model;
[0017] Where N is the number of sample images in the weather image dataset, M is the number of weather phenomenon categories contained in the weather image dataset; f(x i ) is any i-th sample image x i Input the recognition result of the weather phenomenon recognition model, is the i-th sample image x i The weather phenomenon category y i The class center vector, W yi is the weather phenomenon category y i The weight of j is the class center vector of weather phenomenon category j, j≠y i , is the temperature coefficient.
[0018] A further technical solution is that any weather phenomenon category y i The weight W yi According to the weather phenomenon category i The intra-class distances and inter-class distances of different weather phenomenon categories are dynamically adjusted during the model training process.
[0019] Its further technical solution is:
[0020]
[0021] in, Represents the recognition result f(x i ) and weather phenomenon category y i The intra-class distance between the class center vectors, Indicates the weather phenomenon category y i The inter-class distance between the class center vector of and the class center vectors of other weather phenomenon categories j.
[0022] The beneficial technical effects of this application are:
[0023] The present application discloses a weather phenomenon recognition method based on an improved Transformer, which introduces a local self-attention mechanism LSA to modify the Swin-Transformer module, and combines the offset block serialization operation to construct a weather phenomenon recognition model for realizing weather phenomenon recognition. The offset block serialization operation spatially shifts the input image in several directions and connects them with the input image, which can embed more spatial information into the visual tag, thereby increasing the local inductive bias of the weather phenomenon recognition model, and the local self-attention mechanism LSA sharpens the distribution of attention scores by learning the temperature parameter of the softmax function, and removes the self-tag relationship by applying diagonal shielding, which relatively increases the attention scores between different tokens, making the distribution of attention scores clearer. The weather phenomenon recognition model with this structure can also have good recognition performance in scenarios such as weather phenomenon recognition where the data set is small in scale, unbalanced in distribution, image background interference, and excessive redundant information.
[0024] This method also provides a new loss function to replace the cross-entropy loss of the traditional classification network. The loss function provided in this application pays more attention to the relationship between categories through the combination of contrastive learning and dynamic weights, and can better distinguish those easily confused categories. It not only enhances the classification ability of the model, but also can effectively deal with data imbalance and the relationship between categories, further improving the recognition performance of the weather phenomenon recognition model on non-large-scale weather image data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a network structure diagram of a weather phenomenon recognition model constructed in one embodiment of the present application.
[0026] Figure 2 It is a structural diagram of each Swin-Transforme module in the weather phenomenon recognition model constructed in one embodiment of the present application.
[0027] Figure 3 It is a processing flow chart of the offset block serialization module in the weather phenomenon recognition model constructed in one embodiment of the present application. DETAILED DESCRIPTION
[0028] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.
[0029] The present application discloses a weather phenomenon recognition method based on an improved Transformer, the weather phenomenon recognition method comprising the following steps:
[0030] Step 1: Obtain weather image dataset.
[0031] The acquired weather image dataset includes sample images taken under several different weather phenomena, and each sample image is labeled with the weather phenomenon category to which it belongs. The weather phenomenon categories included in the weather image dataset include at least sunny, rainy, snowy, foggy, haze, dew, frost, and sandstorm.
[0032] In order to expand the data set, after the sample images are captured, various data enhancement techniques are used to perform data enhancement operations on the sample images to expand the data set, and the image size of all sample images is unified. The data enhancement operation includes at least one of the following: CutMix operation, Mixup operation, RandAugment operation, random flipping, and random cropping. In one example, the sample images in the weather image dataset are in the ImageNet format and the image size is unified to 224*224.
[0033] Step 2: Build a weather phenomenon recognition model.
[0034] The weather phenomenon recognition model is built based on the neural network model. The traditional approach is to build it based on the CNN model. However, the receptive field of the CNN model depends on the convolution kernel size, step size, and number of network layers, which is prone to information loss. It is limited by local perception characteristics and lacks the ability to represent long-distance features, resulting in unsatisfactory recognition results. Weather phenomenon recognition is affected by various factors in the image data, such as the presence of scene light reflections and other objects unrelated to weather characteristics in the image. These factors are interrelated and weather recognition itself has high requirements for adapting to scene changes. The structure of the scene itself has a great impact on the entire recognition result. For extracting weather information from visible light images, it is difficult to obtain an accurate description of the overall features simply through local information.
[0035] Based on this consideration, this application builds a weather phenomenon recognition model based on the ViT (Vision Transformer) model. ViT is not limited by local interactions, and multi-head self-attention operations can capture long-distance information more flexibly and learn high-level semantic information, so the recognition surface exceeds the CNN model. However, the traditional ViT model requires a large amount of data for pre-training to achieve the expected effect. It is difficult to directly apply it in scenarios such as weather phenomenon recognition where the data set is small in scale, unbalanced in distribution, and has too much redundant information from image background interference. Therefore, this application optimizes the traditional ViT model to construct the weather phenomenon recognition model in this application.
[0036] Please refer to Figure 1 As shown in the network structure diagram, the weather phenomenon recognition model includes sequentially connected block modules, multiple feature processing layers, a normalization layer, an average pooling layer and a fully connected layer.
[0037] The blocking module is used to perform an offset blocking serialization operation on an input image to divide the input image into multiple non-overlapping image blocks.
[0038] Each feature processing layer will reduce the resolution of the input feature map, expanding the receptive field layer by layer like CNN. The first feature processing layer includes a linear embedding module and multiple Swin-Transforme modules connected in sequence. The linear embedding module is used to perform linear transformation on each image block to generate feature representation. Other feature processing layers include a merging module and multiple Swin-Transforme modules connected in sequence. The merging module is used to achieve downsampling by merging image blocks, so that the width and height of the input feature map are reduced by half, the number of channels is doubled and output. In practical applications, it generally includes four feature processing layers. Figure 1 Taking this as an example, the first feature processing layer includes two Swin-Transforme modules, the second feature processing layer includes two Swin-Transforme modules, the third feature processing layer includes six Swin-Transforme modules, and the fourth feature processing layer includes two Swin-Transforme modules.
[0039] After passing through multiple feature processing layers, the feature sequence obtained is then processed by the normalization layer and the average pooling layer in turn to prevent gradient disappearance and gradient explosion. Finally, the recognition probability of each weather phenomenon category is obtained through the fully connected layer, and then the one with the highest recognition probability can be taken as the recognition result.
[0040] Multiple Swin-Transforme modules in each feature processing layer are used to perform hierarchical image feature extraction and processing. Each Swin-Transforme module is built based on the local self-attention mechanism LSA (Locality Self-Attention). Please refer to Figure 2 The module structure diagram of each Swin-Transforme module shown in FIG. 1 includes a first normalization layer, a W-LSA layer, a second normalization layer, an MLP layer, a third normalization layer, a SW-LSA layer, a fourth normalization layer, and an MLP layer connected in sequence. The input image X of the Swin-Transforme module l-1 After passing through the first normalization layer and the W-LSA layer, the image is added to the input image X1 to obtain image After the second normalization layer and MLP layer, the image Add together to get image X l Image X l After the third normalization layer and SW-LSA layer, the image X l Add to get the image image After the fourth normalization layer and MLP layer, the image Add to get image X l+1 And output the Swin-Transforme module.
[0041] Among them, the W-LSA layer uses the local self-attention mechanism LSA based on a fixed window for processing. It uses a local window of fixed size to divide the image into non-overlapping areas and performs LSA calculations on different local areas of the image. The SW-LSA layer uses the local self-attention mechanism LSA of a moving window for processing. It exchanges information between windows through cyclic displacement, solves the problem that the W-LSA layer cannot transfer information between different windows, and greatly reduces the computational complexity. The MLP layer uses a two-layer fully connected network to further process the features and output serialized information.
[0042] LSA obtains the Query, Key, and Value by applying a learnable linear projection to each token. Next, the similarity matrix representing the relationship between tokens is calculated by the dot product operation of the Query and Key. After that, a diagonal mask is applied to the diagonal components of the similarity matrix. After the diagonal mask, a learnable temperature scaling is applied to allow the model to determine the softmax temperature by itself during the learning phase. In general, the softmax function can control the smoothness of the output distribution through temperature scaling. LSA mainly sharpens the distribution of attention scores by learning the temperature parameter of the softmax function. In addition, self-label relationships are removed by applying diagonal masking, which forces the diagonal components of the similarity matrix calculated by the Query and Key to be suppressed. This masking relatively increases the attention scores between different tokens, making the distribution of attention scores clearer.
[0043] The block module and the merging modules in each feature processing layer are implemented using the offset block serialization module. Please refer to Figure 3 As shown in the network structure diagram, each offset block serialization module spatially shifts the input image in four diagonal directions and stacks it with the input image to obtain a stacked feature map. After the stacked feature map is block serialized, it is patch flattened, layer normalized, and linearly projected in sequence before output. The offset block serialization module can embed more spatial information into the visual markup by spatially shifting the input image in several directions and connecting them with the input image. The resulting visual feature sequence will contain more spatial information and increase the local inductive bias of the Transformer model.
[0044] Step 3: Use the weather image data set to train the constructed weather phenomenon recognition model, and then use the trained weather phenomenon recognition model to recognize weather phenomena.
[0045] When training the model, the sample image is input into the weather phenomenon recognition model to obtain the recognition result, and the loss function is calculated based on the weather phenomenon category to which the sample image belongs labeled. Then, the network parameters can be adjusted using back propagation according to the loss function to achieve model training.
[0046] Compared with the approach of directly using the conventional cross entropy loss function, the loss function used in one embodiment is:
[0047]
[0048] Wherein, N is the number of sample images in the weather image dataset, and M is the number of weather phenomenon categories contained in the weather image dataset. i ) is any i-th sample image x i Input the recognition result obtained by the weather phenomenon recognition model, is the i-th sample image x i The weather phenomenon category y i The class center vector, W yi is the weather phenomenon category y i The weight of j is the class center vector of weather phenomenon category j, j≠y i , τ is the temperature coefficient.
[0049] This loss function introduces the idea of contrastive learning into the traditional classification task, so that the model not only learns to distinguish between positive and negative samples, but also enhances the perception of classification boundaries through the similarities between positive samples and the differences between negative samples. i The weight W yi According to the weather phenomenon category i The intra-class distance and the inter-class distance of different weather phenomenon categories are dynamically adjusted during the model training process, so that any weather phenomenon category y i The weight W yi It not only depends on the number of sample images, but also on the similarity between weather phenomenon categories. This method is particularly effective when the intra-class distance is large and the inter-class distance is small. The class weights are dynamically adjusted according to the intra-class and inter-class distances, so that the model pays more attention to those categories that are easy to be confused, thereby achieving better results in unbalanced datasets.
[0050] Any weather phenomenon category y i The weight W yi The calculation formula is:
[0051]
[0052] in, Represents the recognition result f(x i ) and weather phenomenon category y i The intra-class distance between the class center vectors of the weather phenomenon category y i The intra-class distance The smaller the value, the higher the intra-class density. i The weight W yi The bigger.
[0053] Indicates the weather phenomenon category y i The inter-class distance between the class center vector of and the class center vectors of other weather phenomenon categories j, The larger the value, the more it indicates the weather phenomenon category y i The higher the degree of differentiation between weather phenomenon category j, the higher the i The weight W yi The smaller.
[0054] This loss function adaptively weights difficult-to-classify samples through contrast loss and dynamic weights, so that the model pays more attention to these samples. It not only enhances the classification ability of the model, but also effectively handles the problem of data imbalance and can better distinguish those easily confused categories. Therefore, it has greater potential on complex and unbalanced data sets.
[0055] Before model training, set appropriate hyperparameters and initialize them: set the number of iterations to 300, the number of images input to the network model each time to 16, select the Adam optimizer, set the initial learning rate to 0.001, and use the cosine annealing strategy during training to prevent the model from falling into the local optimal solution.
[0056] After the training is completed, the test set can also be used to test and evaluate the performance of the trained weather phenomenon recognition model, including the use of indicators such as accuracy, precision, and recall.
[0057] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.
Claims
1. A weather phenomenon recognition method based on improved Transformer, characterized in that: The weather phenomenon identification method comprises: Acquire a weather image dataset, wherein the weather image dataset includes sample images taken under several different weather phenomena; Construct a weather phenomenon recognition model, the weather phenomenon recognition model includes a block module, multiple feature processing layers, a normalization layer, an average pooling layer and a fully connected layer connected in sequence; the block module is used to perform an offset block serialization operation on an input image to divide the input image into multiple non-overlapping image blocks, the first feature processing layer includes a linear embedding module and multiple Swin-Transforme modules connected in sequence, and each other feature processing layer includes a merging module and multiple Swin-Transforme modules connected in sequence, the linear embedding module is used to perform a linear transformation on each image block, the merging module is used to achieve downsampling by merging image blocks, and the multiple Swin-Transforme modules in each feature processing layer are used to perform image feature extraction processing in a hierarchical manner, and each Swin-Transforme module is constructed based on a local self-attention mechanism LSA; The weather image dataset is used to train the constructed weather phenomenon recognition model, and the trained weather phenomenon recognition model is used to recognize weather phenomena.
2. The weather phenomenon recognition method according to claim 1, characterized in that: The block module and the merging modules in each feature processing layer are all implemented by an offset block serialization module. Each offset block serialization module spatially shifts the input image in four diagonal directions and stacks it with the input image to obtain a stacked feature map. After the stacked feature map is block serialized, patch flattening, layer normalization and linear projection processing are performed in sequence before output.
3. The weather phenomenon identification method according to claim 1, characterized in that: Each Swin-Transforme module includes a first normalization layer, a W-LSA layer, a second normalization layer, an MLP layer, a third normalization layer, a SW-LSA layer, a fourth normalization layer and an MLP layer connected in sequence, wherein the W-LSA layer is processed using a local self-attention mechanism LSA based on a fixed window, and the SW-LSA layer is processed using a local self-attention mechanism LSA based on a moving window; The input image X of the Swin-Transforme module l-1 After passing through the first normalization layer and the W-LSA layer in sequence, the image is added to the input image X1 to obtain the image image After the second normalization layer and the MLP layer, the image Add together to get image X l ; ImageX l After passing through the third normalization layer and the SW-LSA layer, the image X1 is added to obtain the image image After the fourth normalization layer and the MLP layer, the image Add together to get image X l+1 And output the Swin-Transforme module.
4. The weather phenomenon identification method according to claim 1, characterized in that: The method of using the weather image data set to perform model training on the constructed weather phenomenon recognition model includes: Using the weather image dataset according to the loss function Conduct model training on the constructed weather phenomenon recognition model; Wherein, N is the number of sample images in the weather image dataset, M is the number of weather phenomenon categories contained in the weather image dataset; f(x i ) is any i-th sample image x i Input the recognition result obtained by the weather phenomenon recognition model, is the i-th sample image x i The weather phenomenon category y i The class center vector, W yi is the weather phenomenon category y i The weight of j is the class center vector of weather phenomenon category j, j≠y i , τ is the temperature coefficient.
5. The weather phenomenon recognition method according to claim 4, characterized in that: Any weather phenomenon category y i The weight W yi According to the weather phenomenon category i The intra-class distances and inter-class distances of different weather phenomenon categories are dynamically adjusted during the model training process.
6. The weather phenomenon recognition method according to claim 5, characterized in that: in, Represents the recognition result f(x i ) and weather phenomenon category y i The intra-class distance between the class center vectors, Indicates the weather phenomenon category y i The inter-class distance between the class center vector of and the class center vectors of other weather phenomenon categories j.